You've hit on the logical fallacy that everyone seems to gloss over, namely, Men and Women are not different species. Any trait selected for would manifest in both genders. What this would lend credence to, is the cultural meme concept.