Not only is there the lack of nutrient problem, but there is a big radiation problem.
Some bacteria can go dormant for a long, long time. They don't need nutrients, water and can withstand radiation (assuming it's impossible for them to be protected by heavy elements that asteroids are typically composed of).
How many more millions of years would that take, and how much more unlikely is it that a meteor would escape another solar system and intercept with ours?
It's not impossible. We could have easily had millions of kgs of material from other solar systems intercept ours over billions of years.
The atmospheric entry would be more than impossible to survive.
Says who? How hot does the center of a big meteor get during reentry? If it's big enough, it easily be at a survivable temperature. If it breaks up in the atmosphere, the bacteria won't need to survive the collision, either.
The rest of your comment tries to rule out panspermia by saying abiogenesis itself is unlikely! Like evolution, panspermia is a concept that doesn't care about the truth of abiogenesis.
I think it is scientifically irresponsible to try and conclude something in this manner.
Science cares nothing for what you want the truth to be, or what you expect it to be. When Einstein came up with relativity, he wasn't scientifically irresponsible, was he? He was just drawing conclusions based on observed phenomena.
In this case, there isn't really evidence that suggests that the following conditions for lithopanspermia are impossible: That unsterilized material can be transported by collisions on planets, that bacteria can lay dormant on space rocks for (potentially) millions of years.