It is possible that miners will stop signalling support for SegWit some time in the future, and as SegWit is a UASF this is perfectly valid.

However, in the case that this does happen, all SegWit-transfers run the risk of being stolen, as essentially no one can prove they weren't sent to whomever the miners say they were. Previously the 51% attack only allowed miners to double-spend the same transaction, as the user when they sign the transaction also sign who it's going to with the witness data. While this may sound bad, the probability of this actually happening is very very low, unless miners decide to all come together and stop signalling for SegWit solely for the purpose of eating up all the SegWit transactions. This would hurt the value of bitcoin and the miners would lose money.

However.

What the miners can do is reduce support for SegWit to around 50%, right on the brink. The markets will react accordingly. Every time they mine a block, they are able to vote on yes-segwit or no-segwit, pushing the price of bitcoin up or down respectively. With this they can make more money playing the markets with their insider knowledge than they make in the block reward. For this reason, people are worried about what will actually happen when SegWit is activated, and how many of the miners showed support but don't actually support it. Only time will tell :/