Less variety when you always play against people at a similar skill level. You can't mess around as much if every game is like a ranked match. Even if you do well it won't feel satisfying because you'll just think you got an easy lobby by playing badly earlier. You'll never actually see what the players that are better than you do and try to learn from them because you won't end up in their lobbies. Playing with people of different skill levels will basically always mean the lower skilled players in the party will struggle with no oponents at the same level etc.

SBMM has almost no place in casual multiplayer games, let the bottom players have their own lobbies, but everyone else should be in the same pool of players.