> There are other factors like development within the mother's body, levels of hormones / chemicals, and so on which could contribute to a different sexuality later on.

Those could still be influences if it was purely genetic. Biological sex is approximately fully genetic (not quite, but very close to it), but womb environment (I remember pH differences specifically being identified as an influence) play a role in which sperm, with which sex chromosomes, are most likely to survive and reach the ova.