The defense had a lot less depth. Guys like Browner, Thurmond, Clemons, and others I'm forgetting got nice contracts elsewhere. So when the injuries mentioned below happened their roster wasn't so goddamn stacked they could just keep on

Injuries ... Sherman with one arm (still played well though), Thomas' torn shoulder labrum, Kam played on a partially torn MCL that he injured the night before (and he didn't play so well), a few D-lineman were out and then once Avril went out the pass-rush was kinda gone. The injuries to the above DB's led them to have Tharold Simon (depth outside CB) active instead of a back-up slot corner (Marcus Burley). Jeremy Lane got injured in the first half and from that point on Brady just picked on Simon over and over and over and over and over

Frankly, dominating the first super bowl like that was a bit fluky, and then the defense was a bit beat-up / less depth the next year.