Most vegetarians and vegans I know who I've talked to about this basically conclude that although there may be good reason to believe that it isn't ethically wrong to kill bivalves, they really don't have any desire to eat them anyway. Outside of certain regions, oysters aren't particularly popular, so there's a good chance that it wasn't even something they ate before becoming vegetarian. After adjusting to a vegetarian or vegan diet, a lot of people don't even find the thought of popular meats to be particularly appetizing anymore, so telling them that it's totally ethical to eat this slimy little sea creature is usually just going to elicit a "but why would I want to?"