My college physics professor once gave a lecture to a humanities class on the need for scientific literacy. At one point, he told us, “No matter what field you plan to go into, learn math. Math is how you know when you’re being lied to.”
I recently finished a book that makes the same point. How Not To Be Wrong: The Power of Mathematical Thinking was written by a math professor named Jordan Ellenberg who does a nice job of explaining mathematical concepts without causing the reader’s eyes to glaze over.
I liked the book, but before you run out and buy a copy, I should mention that much of the material it covers seems unrelated to the title. Yes, it’s interesting to learn how some MIT students crunched numbers and devised a plan to guarantee themselves payouts from the Massachusetts lottery under certain conditions, but the chapter won’t teach you how not to be wrong … unless you’re designing a lottery, that is.
That being said, there are several sections that are relevant for people interested in the health sciences. Rather than write one very long post about those sections, I figured I’d cover one or two topics in a short series of posts. So let’s start with a topic near and dear to my heart …
As I mentioned in my Science For Smart People speech, when most people say an event or a fact is significant, they mean it’s important or meaningful. But in the world of scientific studies, significant simply means that based on tried and true statistical formulas, the result is not likely due to chance. It’s important not to confuse the two meanings.
In science, significance is expressed as a p-value, which Ellenberg explains in the book. If the p-value is .10, there’s a 10% chance the results were due to chance. For the results of a study to be called statistically significant, the p-value must be .05 or smaller. But again, significant doesn’t necessarily mean important.
Given a large enough sample size and enough data to crunch, scientists could say, for example, that cigar smokers have a higher rate of mouth cancer and that the difference is significant. But if the “significant” difference is one additional case of mouth cancer for every 250,000 people, most of us wouldn’t consider that meaningful or important. The actual odds of developing mouth cancer have barely changed at all.
Ellenberg makes the same point about the meaning of significant, then tags on some additional warnings for readers who don’t want to be bamboozled by media reports on the latest something-will-kill-you or something-will-save-you study. One of those warnings falls into the scientists are freakin’ liars category:
And all this assumes the scientists in question are playing fair. But that doesn’t always happen…. If you run your analysis and get a p-value of .06, you’re supposed to conclude that your results are statistically insignificant. But it takes a lot of mental strength to stuff years of work in the file drawer. After all, don’t the numbers for that one subject look a little screwy? Probably an outlier, maybe try deleting that line of the spreadsheet. Did we control for age? Did we control for the weather? Did we control for age and the weather? Give yourself license to tweak and shade the statistical tests you carry out on your results, and you can often get that .06 down to a .04.
Now imagine the numbers you’re crunching are for what was supposed to be a breakthrough drug and there are millions of dollars at stake. You get the idea.
But here’s what I consider the most important (and significant) point Ellenberg makes in the chapter: If the p-value is .05, that means the odds are only 1-in-20 that those impressive results were due to chance, right? Right … which means given enough chances, I could end up with impressive results that are significant, but still due solely to chance.
Ellenberg asks us to imagine 20 scientists running independent experiments to determine if eating a particular color of jelly bean causes outbreaks of acne. In 19 of the experiments, the color of the jelly beans consumed makes no difference. But in one of the 20 experiments, the subjects who ate green jelly beans had more outbreaks of acne – and those results are significant, because the statistical odds of them being due to chance are just 5%.
The 19 scientists who found no difference grumble, light a cigar, toss back a scotch, stuff their papers in their desk drawers, and go write their next grant proposal. The one scientist who found a significant difference proudly publishes his results … and a day later, there are media headlines trumpeting the now-established “fact” that green jelly beans cause acne.
The significant result was due to chance. But as Ellenberg points out, given enough chances, chance happens. That’s why the significant results of many studies don’t hold up and can’t be replicated.
So (and this is me talking, not Ellenberg) … now let’s think about how science is conducted for Big Pharma. Drug companies aren’t required to publish all their results, so they don’t. They aren’t required to share the raw data with other scientists, so they don’t. A good friend of mine has a brother who worked in Big Pharma and admitted to my friend, “We just keep running studies until we get the results we want.”
Given enough chances, chance happens. And if that p-value of .06 is the best we could get after multiple chances, well, perhaps a little tweaking here and there …
That’s why I don’t trust the results of studies funded by drug companies. Ellenberg doesn’t come out and say as much directly, but he does mention that industry-funded studies often can’t be successfully replicated.
Math is how you know – or at least have reason to suspect – you’re being lied to.
More mathematical-thinking examples from the book in future posts.