You are basically correct: https://www.psychologytoday.com/us/blog/sex-dawn/201004/why-do-breasts-mesmerize

But evolution can make animals sexually aroused by body parts that aren't explicitly sexual or related to sex as well. Female birds are aroused by male birds with bright colors on their wings.

But lets look at the alternative: that society has conditioned men to be aroused by breasts. I remember not thinking twice about a woman's breasts until I hit puberty, and then i couldn't help but get turned on. That alone should indicate that it's genetics and not society that makes men attracted to the breasts