Gluten's fine. I don't live on breads, but whole grains are good sources of energy that slow releases throughout the day and hit me with fiber and moderate amounts of protein. Though if you're still eating beans, I think you can substitute like a half cup of beans for a grain serving and get what you need carb wise.

I'm a big fan of checking your body. And cutting out different things or trying out a diet are good ways to figure out if you're eating too much, too little, need to eat more of x, or less of x, so I'll give different diets a try just as a way to fine tune what my regular diet is. Sometimes those diets are "let's go vegan for a week", other times they're "I'm going to not feel bad for eating ALL OF THE CHEESE."

So the other side of my life experience with cutting out things is that I've been a vegetarian for 20 years now (not really by choice- stupid body). The nutrition that you don't get from cutting out a basic (like "carbs" or "fats" or "gluten" or "meat"), in my experience, needs to be really carefully weighed against the pay off. The rough time I've had with a no meat diet is why I'm more suspicious of those types of diets than the "let's only eat things that are yellow" diets.

If the pay off is worth it, you have to do your due diligence and compensate for what you're losing. As long as you're cognizant and healthy, you're fine. But I can't tell you how many brothers in arms I've met as a vegetarian who eat terrible things or just don't eat because they cut out this one massive thing and are either new to the fold or don't think about themselves inside of the diet. Always be aware of what the food you are eating is doing. Food should optimize you're performance, not hamstring it.

I'd just keep your routine and fine tune (assuming sturdy bowels). Break the diet, see how you feel, get back on it, see how you feel. You'll even out on something that works for you.

Sidenote: If the diet is worth it, you just have to start bringing your own stuff to events. If the holidays are full of cookies, there's a million paleo cookie recipes on pinterest. Have at it. Vegetarians at BBQ's have been pulling that move for decades and most of our families still invite us over.