“you’re a lot bigger than her”

This should have nothing to do with how you feel about your body and your reaction has everything to do with your past issues and nothing to do with your bf. Unless you are the height of an 8 year old, you don't weigh very much.

People come in all different sizes and body types. Some people have smaller frames and bones. Larger framed people actually think they could ever weigh the same. It's impossible. It's just the effect of societies unrealistic views on body image.

If his first reaction was the answer he gave, he was just being honest and you should get over it. Even an extra 5kg is a big difference when you are putting someone on your shoulders. The last time I did that, was likely a 20 kg kid and even then it was a struggle. Just let it go.