an old, coked-up Pernell Whitaker
Excuse me? When DLH fought Whitaker, Whitaker was 33, he hadn't lost in more than a decade and had been considered the best welterweight in the world for years. It was De La Hoya's first fight at welterweight, and Whitaker was absolutely the toughest fight De La Hoya could have chosen. Argue the decision if you want, but Whitaker was a monster when De La Hoya fought him.
one of Floyd's toughest fights along with vs. Cotto, Maidana I and Castillo I.
Hindsight is 20/20. You named the fights in which Floyd came closest to losing. Floyd's win over Oscar was famously a split decision, Cotto famously bloodied Floyd's nose, Maidana landed more shots on Floyd in the first fight than any fighter ever has, and I think there are probably still people arguing over whether Floyd beat Castillo the first time.
But closest does not equal toughest.
I think Ricky Hatton was pretty tough when he was a monster with 30+ wins in a row and an insane KO percentage. I think Canelo was and is pretty tough. He'd steamrolled through everyone up until Floyd, and has continued to since. But those weren't close victories, so no one considers them among Floyd's "toughest" fights. But they were among Floyd's toughest fights. They just weren't close.
People say Floyd is on a whole other level to guys like Pernell Whitaker and James Toney, and I don't really see why.
On a whole other level? That's what Floyd says about Floyd. I'm not sure I know of anyone else who would say that about Floyd, at least not without some serious qualifications and definitions.
Financially you could make the case that Floyd was on a whole other level. On a strictly defensive basis you could probably argue it. I think Floyd's defense was probably better than Whitaker's or Toney's, but not by so much, and wouldn't be better at all if Toney had discipline. I think it would probably be a completely bogus argument if you were to make it about offense or countering ability, though.
When you're talking about all time greats like this it's never as easy as saying "A is on a whole other level to B." It's a lot more subtle than that.