Suppose there is a search for a cure (or relief from) a certain disease. Most of the time, cures are difficult (second law of thermodynamics at work here). So, the ratio of “stuff that works” to “stuff that doesn’t work” is pretty small. For our case, say it is 1 to 1000.
Now when a proposed “remedy” is tested in a clinical trial, there is always a possibility for two types of error: type I which is the “false positive” (e. g., the remedy appears to work beyond placebo but really doesn’t) and “false negative” (we miss a valid remedy).
Because there is so much variation in humans, setting the threshold for accepting the remedy too low means we’ll never get cures. Hence a standard threshold is .05, or “the chance that this is a false positive is 5 percent”.
So, suppose 1001 different remedies are tried and it turns out that only 1 of them is a real remedy (and we’ll assume that we don’t suffer a type II error). Well, we will have 1000 remedies that are not actually real remedies, but about 5 percent, or about 50 will show up as “positive” (e. g. brings relief beyond placebo). Let’s just say that there are 49 “false positives”.
Now saying “we tried X and it didn’t work” isn’t really exciting news for anyone other than the people searching for the remedy. So these results receive little publicity. But “positive” results ARE considered newsworthy. Hence the public sees 50 results being announced: 49 of these are false positive and 1 is true. So the public sees 50 “this remedy works! (we think; we still need replication)” announcements, and often the medial leaves off the “still needs replication” part..at least out of the headline.
And….of the 50 announcements …..only ONE (or 2 percent) pans out.
The vast majority of results you see announced are…wrong. 🙂
Now, I just made up these numbers for the sake of argument; but this shows how this works, even when the scientists are completely honest and competent.