The smallest quantity of soy found to have any sort of estrogen-feminizing effects on a man was 12 servings of soy, and that was in a case study of a single subject. And it's believed this was a person who was particularly sensitive to it. And after cutting down on soy, the subject's effects reversed to normal. You won't find any other evidence of this. I've looked. If you have found evidence, please point it my way, because this topic interests me. I'm not sure how much red clover those sheep were eating (red clover tends to have more phytoestrogens than soybeans anyways, generally speaking), but 12 servings of soy is the equivalent of...
110 grams of soy protein, or 6 full cups of tofu, or 6 full cups of soybeans, or 6 cups of soy-based veggie meats, or 12 cups of soymilk
We are talking a TON of soy. Who eats that much? Other than the guy in the case study who was inhaling it, apparently.
Let's say you had a glass of soymilk and 1/4 cup of soy sausage in the morning. (1.25 servings) Snacked on half a cup of dried edamame at work or school. (1 serving) Had stir fry with half a cup of tofu in it. (1 serving) Then after the gym ate a protein bar with 20g of soy protein in it. (2 servings)
That is under 6 servings of soy. Again, the smallest amount of soy ever recorded to have feminizing effects on a man was 12 servings per day, and he ate that much consistently. So even if your whole day was loaded with soy like in the above meal plan, you wouldn't even come close. And a lot of things like veggie meats and veggie proteins are often blends of pea protein, rice protein and gluten protein, all of which have have almost no phytoestrogen. Throw in the fact your stir fry probably had cruciferous vegetables, which block estrogen (broccoli, cauliflower, kale, cabbage, and so on). You have nothing to worry about.
It's also worth mentioning that moderate amounts of soy intake have been shown to have a lot of benefits.
Plus, while this is a great thread with some good advice, literally every food mentioned here should not be eaten in excess. Not soy, not grapes, not anything. There is not a food on that planet that should be overeaten. The one I would avoid the most of the top-rated ideas is beef jerky. Yeah, it's got some iron and protein. But with all of its heterocyclic amines, carnitine, cholesterol, sodium, polycyclic aromatic hydrocarbons, saturated fat. Not to mention sugar. Whether you make it homemade or not, that's in your jerky. Then there's other things that might be added, liquid smoke, dyes and sodium nitrite. You'd think there would be more controversy there, but somehow soy causes a lot more discussion.