Here is the not stupid answer.

We always think about where life is, but never when life is. Think Star Wars' every single intro: Long ago in a galaxy far, far away

Then consider, (or assume, whatever) for there to be interstellar travel, necessarily requires civilization. Civilizations crash lotsa times. Human civilization has been developing for what? 10,000 years? We are not guaranteed to keep going. We could all/mostly die within 100 years. Let alone 1000 years, 2000 years, we won't live for sure. Shit, our SUN has an expiry date. Think of that! I personally believe that the motivation of greed, which helps foster successful life, is itself anathema to the survival of civilization, which to me seems both instrumental, and more significant than the Fermi paradox. But neverminding this, simply assume the reasonable truth that survival of civilization, even life, is not guaranteed. Assuming civilization is necessary for space travel, civilizations are more delicate than life is. Now if life develops 1000 light years away, who's to say it doesn't develop say... 10 million years ago? (a long time ago) Who's to say it doesn't develop 100 million light yeats away?? (...in a galaxy far, far away) Who's to say it succeeded at not dying for even a million years? Human civilization barely squeaks by and we've been around like 10,000 years. If we CAN live 20,000 years, that would be impressive, but it doesn't exactly help us find life anywhere from 21,000 years to 3 billion years in the past or future.

TL,DR: Space travel requires a lot of cooperation(read:civilization) and civilizations are much, much more fragile than life. If another one happened, but happened say 1 million years ago, we're too late. If it happened 1 million light years away, we missed our chance already. Time is a bigger frontier than space.