what would be legally / morally / pragmatically wrong with this?
Polygamy is a canard. The current legal framework for marriage specifically set up for binary rights and obligations between two spouses. Once you introduce polygamy (and the rights of third, fourth, fifth, etc. spouses), then the system breaks down.
For example, A and B are married. If A wishes to marry C, must B also consent? If B does not consent, can the marriage still occur between A and C?
Furthermore, under current law, parental rights are automatically established when a child is born of the marriage. If B has a child with A, could C also claim parents rights? What if B never consented to marry C?
Pretend C dies and has retirement payments with survivor spouse benefits (like a pension or Social Security). Are both A and B eligible? If so, is it fair that B only paid into the retirement system for two people but would be receiving payments for three? What if C were also married to D, E, and F? How long before the retirement systems are bankrupted by multiple payments that they did not plan for? (Notice that this issue does not arise under same sex marriages because people pay into the system for two people irrespective of marital status or sex.)
As you can see, there is no slippery slope to polygamy. If people actually think that the government has no rational basis for prohibiting polygamy (problems arising from divorce, inheritance, alimony, custody, etc. when those obligations are not binary), then they should feel free to file a lawsuit. Before complaining that it will be too expensive, § 1983 has an attorney's fees provision, so such legal fees would be reimbursed if they are successful in court in legalizing polygamy. Of course, their legal arguments are tenuous and would not win, so they are not willing to risk their own money on their rhetoric.