You actually do need social knowledge to know that those differences between sexes are attributed to different sexes. How else would you know what those differences mean with respect to sex if you don't already have some knowledge from other humans regarding what sex is? I don't think that whoever you talked with was saying that people can decide to ignore their existent chromosomes so much as they were trying to point out that the existence of an XY pair of chromosomes does not create the state of "Being a Male" as who's to say what "Male" means outside of social/cultural knowledge.
If you say that male = having an XY pair of chromosomes, then nothing traditionally considered masculine is related at all to "Being a Male" because choice of toys, choice of books, choice of hobbies, etc, are not related to having an XY pair.