I think it’s more that variance leads to small sample sizes being less representative of the population, but the sample mean still could randomly be close to the population mean. So you’re pretty much right but the law of large numbers doesn’t work backwards, like the sample mean doesn’t get closer to the higher or lower results in small samples like the sample mean gets closer to the population mean in large samples.

It’s been some time since I studied this so I could be wrong and have used some terms wrong certainly, and anyone that took more than two stats classes can correct me, that is just my understanding. But the essence of your and the OPs points stand. It is more likely than usual to have higher or lower results, but that doesn’t necessarily make it more likely to not be representative of the population, if that makes sense.