A few possibilities spring to mind.

Firstly, build encounters so that, instead of one powerful enemy, have there be a number of enemies each roughly as powerful as a single PC, equal to the number of PCs. Remember that swarms of smaller creatures are far, far weaker in practice than their calculated CRs would indicate. Squishy spellcasters are much the same - the nature of the action economy means that unless you just fudge how much health they have, the PCs will just roast them in 2-3 rounds. "Squishy" bosses should be paired with a few big meaty characters (giants, trolls, ogres, etc.) who will interpose themselves between PCs and their leaders.

Also worth considering is staging multiple encounters in such a way that the PCs cannot take long rests between them - thus, when they get to the end, they will (if they haven't been careful) have expended some measure of their resources (spells/rages/smites/ etc.) and come into the final fight with much reduced firepower.

From a less mechanical perspective, think about employing smarter tactics - and even more unsettling for PCs, offensive tactics. Consider - the standard DnD paradigm is that the PCs go to a place, fight the things who live there, and then leave, right? The PCs are the ones with the initiative - they choose when they leave, when they arrive, and, to some extent, the order in which they get to fight things. But if the monsters come to them - attack them where they live while their guards are down, that gets flipped on it's head. If your antagonists have had multiple contacts with the PCs (or if the PCs are suitably famous) then they might find themselves the targets of attacks specifically tailored to them. Some hair stolen from a brush by a bribed servant would be enough to let a suitably powerful enemy spellcaster scry the party. Mooks that know the party will use AOE attacks deliberately fan out to limit their exposure to such effects. Flying enemies target terrestrial ones with ranged attacks and never allow themselves to be struck. Spellcasters throwing out silence spells to shut down magical PCs.

Also, remember - if all else fails, any sort of crazy, optimized build your PCs can come up with, you can throw back at them - but two at a time and with a dragon for backup :D. You're the DM - their resources are limited. Yours are not. Best of luck to you!