I know exactly how you feel. No matter how confident you are in what you believe now, the feelings of guilt and shame still linger because they're not based on evidence and reason, but in emotion and primal fear. That's part of the power of Christianity, it appeals to our fear of being ostracized and blamed, and those emotions don't go away with a change of belief. For me it's gotten easier with time because I'm not attending church or spending time with really devout people, so I'm not hearing things to reinforce those old beliefs and emotions. Sometimes I'll walk myself through the reasons I deconverted, among them that I don't afford other religions the same authority as Christianity—basically, the arguments used to defend Christianity can be used to defend anything else, but I'm only affected by them in relation to Christianity because that's what I was raised with.

Another thing that has helped me is continuing to read and learn. The more I learn about history, philosophy, and science, the more I have available to reference when I start to feel uneasy.

As far as feeling guilt or shame because of other people, that's different altogether. Some people just don't care what other people think, but I've always cared too much what other people think, and it's hard for me to stand my ground when someone else says something confidently and passionately. I just don't have the brains or rhetorical skill for debates.