You always hear "correlation is not causation" which is certainly true, and is often misunderstood.

But even people who realize this may not realize that a correlation does not mean a significant correlation.

I realized my explanation got super long so:

TL;DR: Scientific journals usually only publish "significant" findings and news orgs only talk about results that will interest the public; this can make the weird/interesting correlations we see blasted across the news less likely to be truly statistically significant

For example, if I talk to 5 people and they all ate chocolate every day and they've all had cancer, I can say that there is a correlation between chocolate and cancer, but this is not significant.

If I talk to 10,000 people and find that all of the people who ate chocolate got cancer and none of the non-chocolate eaters got it, this would be more significant.

What do I mean by significant? Well, published papers have standards for how "significant" something has to be. Usually around the 10% "significance level."

If your result (e.g. eating chocolate is correlated with cancer) is significant at the 10% level that means that, statistically speaking, if your hypothesis were not true (i.e. chocolate does nothing), there is only a 10% chance you would see the data you saw in your study.

This still means that there is a 10% chance that you are saying there is a correlation where none exists. That's why the lower the level the better (e.g. it's better to have only a 1% chance the correlation is not there when you say it is).

However, imagine 100 scientists each independently conduct very similar studies about cancer and chocolate. Ninety-nine of them find no connection and throw out their useless data, because no journal would publish/care about finding no correlation.

Now one scientist does find a significant correlation, and at the 1% level! He sends it in to a journal, and wow, what interesting news, so it's published: "Chocolate linked to cancer!" and every news organization discusses the shocking findings.

Even though if you run the same study 100 times, you would expect that 1% of the time you would get a significant result even if it were not true.

However, since insignificant results never see the light of day, no one knows how significant a finding really is.

Note that this doesn't just apply to studies finding strange correlations (I'm just more familiar with those studies). It is a common problem in scientific fields called "publication bias."

This is why we need some kind of journal for "shit we tried that didn't work", that publishes high quality studies that had a null result.