Oh goodness, I completely understand where you are coming from. My 'baby!' part was, I suspect never installed (thank goodness). It is difficult, extremely difficult, and I suspect the guilt comes from not wanting to accidently hurt or upset everyone else who is excited.
Where I'm late 20's most of my friends and family, be they younger or older are starting to have children which for me is terrifying (cannot even be in the same room as a visibly pregnant woman as I have panic attacks!) and I felt extremely guilty about upsetting so many of my friends and family over this.
Thankfully most know of my fears and are ok but one couple did take serious offense I did not want to visit baby... I had said my congrats to them when they announced (truely I am happy for your happiness!), even bought them a little wooden merry go round music box, a few bits of clothing for various sizes and some simple books, I wanted to show I did care even if I was unable to in the conventional sense.
My partner has visited and was great, he told me not to worry and stay home. He explained to them my literal fear of pregnancy and babies. Never held one, never will kind of lass! That it scares me and I feel uncomfortable. They didn't understand until he explained to her that it was like her fear of animals (she would, I do not kid, scream and burry her face in her partners chest the moment she saw a dog, cat, bird, anything when out), I'm not as bad as her in that regards but it helped them understand I didn't hate them and we all have fears and things that make us uncomfortable.
Perhaps your partner could help you to explain similar to family in a way that explains why you might be more distant but you mean no disrespect by it. I also find it helps to keep a few things pre prepared to say so you avoid the staring at this baby (that looks like all the others) and looking uncomfortable/disinterested. I've learnt even asking about how they're coping helps as they'll talk a long while about that usually taking some pressure off to coo over baby. Other things like saying how happy you are for them helps (unless it's that couple I mentioned previously) as most people appreciate that genuine aspect that you can share in another's happyness without wanting that specifically for yourself.
Sorry for long post, I wish I could offer more useful advise. It is difficult when people can treat you do harshly over lack of interest in babies, so no wonder it causes anxiety and guilty feelings as you don't want to upset them and you're not trying too. I wish you luck sweetie!