I'm not sure if there even is a way to make Dubious Challange work. For your scenario:
If the opponent takes the Gryff, you end up with a 9/8 flyer with no draw backs. This is obviously bad for your opponent, so they're not going to take it.
If your opponent takes neither, you end up with 11 power in the air, no drawbacks. Another bad option for the opponent, so they're not likely to take it.
If they take the 9/8, you get the Gryff. Your opponent puts the Eater into play under their control, you put the Gryff into play under your control, and Dubious Challenge finishes resolving, then triggered abilities try to happen. Except the Eater can't trigger because you have a Gryff. It doesn't matter that the spell made the Eater ETB first, because it put the Gryff in there before the ability could trigger, the Gryff is going to prevent the trigger. Bad option for you.
Dubious Challange is made even worse, though, by being optional for the opponent, and not for you. You can't DC into two Levelers and give your opponent one. They just won't take it. They'll choose not to take the 10/10 that kills their library, and let you take both, and you don't have the option to say no to library eating death machines. Any bad option you hit with DC, you're stuck with.
WotC made this spell even worse than that, though, by making it a goddamn fucking sorcery piece of shit. You can't even DC on opponent's end step, and then untap and cast Brand. You have to DC on your main phase, and then pass turn to your opponent so that they get a turn to punish you and your jank card with the free beater you just handed to them. Unless you had the mana to DC and then use some control-magic-naturalizing effect all on the same turn. And at four mana for the DC, I don't think you did.
There were interesting ways to make this card:
Could have been an Instant. You're straight-up giving a creature to your opponent. Why do you have to give them a turn with it as well?
Could have been cheaper. Four mana to give your opponent a creature, and then maybe get one yourself? No thanks.
Could have been search. You shuffle your library anyway. Why restrict the finding clause to top X cards?
Any one of these changes would have made the card good enough for a modern brew, at the very least.