To be fair it's quite difficult to come up with a solution that requires you use an algorithm to identify and separate human behavior (trolling vs not) to a very accurate level where there is little risk of a normal player being banned for picking unconventionally. Plus the solution would have to account for people abusing it to get out of unfavorable matchups, and identify that behavior as well.

The only solution that I can think of is this:

Implement a WoW-esque party matchmaking system. Players must identify at least three roles that they are willing to play. Champs that are not in those three roles are removed from their picks. Matchmaking will of course try to find a team in the same elo range with a viable 5 man lineup.

The probability that a matchup is unfavorable will be much lower, so problems here are likely to be as a result of trolling. Shove in a report button in champ select that is available for use once a day. If all four players on a team report a player, then the match is cancelled and the players are distributed into games where they will not see each other. If the same person is reported again, then that person receives a 1 day suspension. If it happens again, it'll be a week, then thirty days, then perma.

I'd say the likelihood of a player being maliciously reported twice for no reason or to restart a match are quite low, given that it requires all four players to have a ban available. Theoretically they would keep the daily report in case they need it when they run into an actual troll, and would be less inclined to use it wastefully. This won't get rid of the problem immediately since the number of available daily bans is limited, but over time with trolls getting increasingly larger suspensions it should reduce the number of incidents significantly.