I'm not going to argue that LBS do tend to push what they have available in stock. I've experienced this before too. When I was just a commuter wanting to buy my first bike that wasn't 20 years old, I asked about a entry-level race bike with 105. I told them my current frame was a 58 and felt too large. They ended up putting me on a 60 frame because it was in stock. It felt way too large and awkward. I left and went to another bike shop who was willing to listen to me.
Now, it's easy to see that we're being victims here, but there is one thing to consider on why this occurs so frequently. I don't necessarily blame the bike shops, but rather the industry as a whole. I don't work in a bikeshop so I don't know all of the details but from what I've gathered on reading various threads on ST, bike shops have to put their orders for bikes well in advance and pay up front. This creates an obvious cash flow problem when they're pre-ordering bikes and paying up front for something that won't leave the floor until months later. It's not surprising to me that they would try to fit you on a bike that is in stock, whether they realize it or not.
Having said that, I do think there are bike shops out there that are willing to listen and order you what you need if they don't have it in stock.
As far as OP's initial question - I do not think LBS are trying to rip you off. There are some products they simply cannot beat the online price because of how the industry is set-up. Manufacturers get discounts on groupos so this is why you see some online places selling groupos at such a cheap cost compared to an LBS. The online retailer has a "house brand" that qualifies them for manufacturer pricing, and they are ordering enough bulk to keep the price well below what a LBS's cost. This is just one example.
As far as when I decide to buy online vs in-store...it depends. I used to do a lot of my shopping online, I admit it. But over the years I've geared myself more towards the stance of buying everything I can from the LBS. Here's why - When I was a little kid I used to play various card games, everything from pokemon, to digimon and eventually settled on MTG for several years. When I was playing pokemon as a little kid I used to do tournaments in a local card shop. I can't tell you how devastated I was when that card shop/boardgame shop went out of business. Same thing happened to another MTG/board game shop when I was an MTG player just doing FNM. Retail is tough and it's unfortunate to see some of these shops close up because people are buying offline. Now I don't play MTG as much, but I still do boardgames. It's often cheaper to buy boardgames online vs in the shop, But I try to support the board game shop as much as possible. They add value to the experience by getting to know me personally and make me recommendations. They host events that I could go to if I wanted to (I don't anymore, but I like to think that I'm supporting younger version of me out there who loves going to the events at a young age).
How does this apply to a LBS? I support my LBS because I love bikes, arguably more than I ever loved MTG or board games. I remember how devastated i was to have my board game shops close on me and I don't want that to happen to a bike shop if I can help it.
The other thing that people don't consider here is that buying from a LBS can be seen as an investment. Yes, you can order stuff online for cheaper, but where is the value-add? I've gotten close with my LBS, I'm an ambassador for their brand and I'm friends with everyone in the shop. The obvious thing that people seem to care about are discounts, that's not really a value-add, though it is nice and definitely appreciated. By value-add I mean me being able to go in and have them look at something and sometimes just dropping what they're doing to take care of it right then and there. I've had times where they told me not to worry about the service cost. I've had them help me pick out my cassette ranges for my trainer (indoor), training (outdoor) and my race wheels. One of those cassettes wasn't widely available yet, but my LBS was able to get it in for me. Another huge value-add was recently my bikeshop helped show me how to pack my bike up for shipping. They had given me the bike box (which any bike shop will do), but I was not expecting them to help out with packing the bike. I suspect this was because they knew I was planning on ordering a new bike from them. Then when I did buy my new bike from them they gave me an additional discount on top of my normal discount. They didn't have to- my team has very clear guidelines for what our discount is. But they gave it to me and I can tell you that the money I saved from the additional discount far outweighs any savings I would've had from buying parts online throughout the year.
Those are just some examples of why purchasing from my LBS has been an investment. Not all LBS are great bike shops, I like to think mine is though and hope that other people are able to find an equally great LBS.