Panspermia also has a parsimony problem, it ends up falling to Occam's Razor.
In a sense, I think that, in the absence of evidence that particularly demands panspermia over more local theories of the emergence of life, this is a bigger problem than the "space is big" problem.
Well, to be fully accurate, panspermia doesn't even cover the origin or emergence of life anyway. It still had to start somewhere, panspermia just pushes that somewhere off of Earth (and perhaps to Mars, or perhaps off Mars and onto Earth). Panspermia is a distribution thing, and only an "origin" thing if you limit your focus to a particular planet.
So basically even if you go with "aliens made us", you've still got to have abiogenesis for those aliens, or the aliens that created those aliens, or the aliens that created....
Running to panspermia to avoid the hypothesis of life beginning on earth out of natural processes is just silly. Silly because Occams Razor slices it off, and silly because it doesn't even get you away from life being a natural phenomenon.
In a sense, I think that, in the absence of evidence that particularly demands panspermia over more local theories of the emergence of life, this is a bigger problem than the "space is big" problem.