I know this will be hated and downvoted to hell, but I agree with your boyfriend. I think he expressed a reality that many falsely call victim-blaming and misogyny, but... it's simply a fact of life. He DID say it doesn't justify the man's behaviour - what he did was 100% wrong and he should be punished, there is no question about that in my mind nor in your man's, I think. However, as I have found out from multiple men, it is really difficult to keep from impure thoughts and suggestions when they are constantly visually exposed to poorly covered, beautified female bodies. They know it's wrong, they don't want to objectify women or think anything sexual about them, but the visual stimulus is extremely potent, so that they have great trouble pushing away thoughts that they do not wish to have. My husband sometimes describes his torment to me, when despite his best intentions to keep from any inappropriate desire for other women, he has to constantly struggle within his own mind to control his thoughts, which flood and overwhelm him. A Christian friend of mine described the same problem, both my husband and him saying summer is the worst for this. Now these are men who hold themselves to a very high standard, and do not wish to even fantasize inappropriately, which many would reason does no harm to anybody so why not? But how many men have given up on this struggle, or never bothered in the first place, letting their thoughts run wild? They are undoubtedly responsible for any wrong actions they take as a result of these thoughts; but I unpopularly believe that if there were less constant temptation, more consideration on the part of women for the peculiar weaknesses and urges of men, more men would be able to keep themselves in check. The struggle would be easier, and fewer would lose.
I don't wish to minimize the trauma you experienced. I have had a similar incident, and other violations that have left me emotionally scarred inside. It really, really hurts to be disrespected like that, and to feel vulnerable out in public. I think your boyfriend should have been more sensitive to your pain, and been more careful about what views he expresses in our corner of society. This was probably the wrong time to test the waters. Also, I think many men just don't get how deeply such a violation touches a woman, especially when it's "superficial" and there's no actual rape or violence. It seems like not a big deal to some men maybe? Since you didn't get hurt. They just don't get it - that's more of a peculiarity of female psychology that they are unaware of, just like women are unaware of the temptations of men.
It's not really your fault either though. In modern society, modest dress is not really a thing; fashions are ignorant of this principle, and pretty much everyone accepts that dressing the way women do now is fine and normal. So how's anyone supposed to realize there could be a problem with it, especially since men don't really talk about it, probably fearing the wrath of feminism?
In short, in an ideal world, I think men would respect and protect women, since they are more physically vulnerable, and women would respect men by being aware of the difficulties they have inside and not making it worse.
PS. I don't really know where to stick this, but I try to relate to the way men must feel in the summer to my personal temptation (which I think many women share) - unhealthy food. I really want to eat a vegan whole food diet, I truly believe it's healthy, but when I got a job at a bakery, and at a pizza shop, my struggle became 100 times harder. Whereas before I would never go ahead and buy and consume these things, now I had trouble not doing so every day. You could say it's 100% my fault, and that would be right in a sense - but one can't argue that the environment made it much more difficult for me to make good decisions. These things were constantly in my face, in my hands, giving off their scent to my nose, and I was always seeing customers relishing them. I just broke.
TL;DR: Sincerely sorry for anyone I offend with this, but I believe your boyfriend was expressing a reality of life and of male psychology that no amount of feminism, shaming, and ignorance can remove: men are intensely visual, and vulnerable to sexual temptation. While it doesn't take away their responsibility to keep their imagination and actions in check, it does make it a lot harder for them to do so, and I believe it's disrespectful to ignore the burden we women place on them by dressing in revealing ways. Your boyfriend should have been more sensitive to the suffering you were going through, but perhaps it's something that in turn men find difficult to understand about women.