My college physics professor once gave a lecture to a humanities class on the need for scientific literacy. At one point, he told us, “No matter what field you plan to go into, learn math. Math is how you know when you’re being lied to.”
I recently finished a book that makes the same point. How Not To Be Wrong: The Power of Mathematical Thinking was written by a math professor named Jordan Ellenberg who does a nice job of explaining mathematical concepts without causing the reader’s eyes to glaze over.
I liked the book, but before you run out and buy a copy, I should mention that much of the material it covers seems unrelated to the title. Yes, it’s interesting to learn how some MIT students crunched numbers and devised a plan to guarantee themselves payouts from the Massachusetts lottery under certain conditions, but the chapter won’t teach you how not to be wrong … unless you’re designing a lottery, that is.
That being said, there are several sections that are relevant for people interested in the health sciences. Rather than write one very long post about those sections, I figured I’d cover one or two topics in a short series of posts. So let’s start with a topic near and dear to my heart …
Statistically Significant
As I mentioned in my Science For Smart People speech, when most people say an event or a fact is significant, they mean it’s important or meaningful. But in the world of scientific studies, significant simply means that based on tried and true statistical formulas, the result is not likely due to chance. It’s important not to confuse the two meanings.
In science, significance is expressed as a p-value, which Ellenberg explains in the book. If the p-value is .10, there’s a 10% chance the results were due to chance. For the results of a study to be called statistically significant, the p-value must be .05 or smaller. But again, significant doesn’t necessarily mean important.
Given a large enough sample size and enough data to crunch, scientists could say, for example, that cigar smokers have a higher rate of mouth cancer and that the difference is significant. But if the “significant” difference is one additional case of mouth cancer for every 250,000 people, most of us wouldn’t consider that meaningful or important. The actual odds of developing mouth cancer have barely changed at all.
Ellenberg makes the same point about the meaning of significant, then tags on some additional warnings for readers who don’t want to be bamboozled by media reports on the latest something-will-kill-you or something-will-save-you study. One of those warnings falls into the scientists are freakin’ liars category:
And all this assumes the scientists in question are playing fair. But that doesn’t always happen…. If you run your analysis and get a p-value of .06, you’re supposed to conclude that your results are statistically insignificant. But it takes a lot of mental strength to stuff years of work in the file drawer. After all, don’t the numbers for that one subject look a little screwy? Probably an outlier, maybe try deleting that line of the spreadsheet. Did we control for age? Did we control for the weather? Did we control for age and the weather? Give yourself license to tweak and shade the statistical tests you carry out on your results, and you can often get that .06 down to a .04.
Now imagine the numbers you’re crunching are for what was supposed to be a breakthrough drug and there are millions of dollars at stake. You get the idea.
But here’s what I consider the most important (and significant) point Ellenberg makes in the chapter: If the p-value is .05, that means the odds are only 1-in-20 that those impressive results were due to chance, right? Right … which means given enough chances, I could end up with impressive results that are significant, but still due solely to chance.
Ellenberg asks us to imagine 20 scientists running independent experiments to determine if eating a particular color of jelly bean causes outbreaks of acne. In 19 of the experiments, the color of the jelly beans consumed makes no difference. But in one of the 20 experiments, the subjects who ate green jelly beans had more outbreaks of acne – and those results are significant, because the statistical odds of them being due to chance are just 5%.
The 19 scientists who found no difference grumble, light a cigar, toss back a scotch, stuff their papers in their desk drawers, and go write their next grant proposal. The one scientist who found a significant difference proudly publishes his results … and a day later, there are media headlines trumpeting the now-established “fact” that green jelly beans cause acne.
The significant result was due to chance. But as Ellenberg points out, given enough chances, chance happens. That’s why the significant results of many studies don’t hold up and can’t be replicated.
So (and this is me talking, not Ellenberg) … now let’s think about how science is conducted for Big Pharma. Drug companies aren’t required to publish all their results, so they don’t. They aren’t required to share the raw data with other scientists, so they don’t. A good friend of mine has a brother who worked in Big Pharma and admitted to my friend, “We just keep running studies until we get the results we want.”
Given enough chances, chance happens. And if that p-value of .06 is the best we could get after multiple chances, well, perhaps a little tweaking here and there …
That’s why I don’t trust the results of studies funded by drug companies. Ellenberg doesn’t come out and say as much directly, but he does mention that industry-funded studies often can’t be successfully replicated.
Math is how you know – or at least have reason to suspect – you’re being lied to.
More mathematical-thinking examples from the book in future posts.
If you enjoy my posts, please consider a small donation to the Fat Head Kids GoFundMe campaign.
Good blog post. When I was in college I took Statistics. My teacher told us that statistics was how you lie to people with math. Since then I’ve been pretty wary of statistical information given in news articles. It gave me the mental flexibility to see how they could bend the numbers to get the answer they want you to hear. Math is so interesting and very useful.
Your stat teacher’s comment is the flipside of my physics teacher’s comment. Math is how you lie, so understanding math is how you spot the lies.
Taking statistics in college was the only worthwhile class I took. It was so useful and enlightening. I feel sad for normal people who have never understood it because reading any study is like reading a foreign language, and understanding stats gives you the key to reading that language. I tell everyone I can the importance of learning stats because it’s the key to judge just about anything in the news or in science. You know when you’re being lied to, or at least when something is much weaker than the author is willing to admit.
There are some good sites out there that explain statistics … but of course, people have to care enough to visit them.
Good blog post. When I was in college I took Statistics. My teacher told us that statistics was how you lie to people with math. Since then I’ve been pretty wary of statistical information given in news articles. It gave me the mental flexibility to see how they could bend the numbers to get the answer they want you to hear. Math is so interesting and very useful.
Your stat teacher’s comment is the flipside of my physics teacher’s comment. Math is how you lie, so understanding math is how you spot the lies.
Taking statistics in college was the only worthwhile class I took. It was so useful and enlightening. I feel sad for normal people who have never understood it because reading any study is like reading a foreign language, and understanding stats gives you the key to reading that language. I tell everyone I can the importance of learning stats because it’s the key to judge just about anything in the news or in science. You know when you’re being lied to, or at least when something is much weaker than the author is willing to admit.
There are some good sites out there that explain statistics … but of course, people have to care enough to visit them.
Bravo. Thank you for explaining the good and bad science we are confronted with and need to make our decisions. My guess is as good as theirs if I’m getting good results. This also gives me a talking point with my doctors when they want me to go a direction I’m uncomfortable with.
You never know with doctors. Some consider themselves infallible and expect patients to be good little children and obey.
I read this book a while back, and like you found it interesting though not always practical. Still, the use of the word ‘significant’ is probably the most important point in the entire book for the general population, IMHO.
It’s a shame so many people in the media are science bubble heads. IOW, nothing much inside.
You’d think reporters on the science beat would be required to demonstrate some basic understanding of science before getting the job. That sure doesn’t seem to be the case.
Well it is reasonable to assume that those who generate news content be held to a higher standard of awareness. I am not sure it is reasonable to think that investigative journalism is going on. Ask Sharyl Attkisson how trying to get to the truth worked for her in the eyes of CBS.
A large percentage of the general population doesn’t even understand the probability of a coin flip. I heard someone boasting a few weeks ago that he had left the casino “up” 22 straight times (he mostly plays roulette). It’s the world we live in and people will continue to take advantage of other folk’s limitations as long as they can get away with it.
Bravo. Thank you for explaining the good and bad science we are confronted with and need to make our decisions. My guess is as good as theirs if I’m getting good results. This also gives me a talking point with my doctors when they want me to go a direction I’m uncomfortable with.
You never know with doctors. Some consider themselves infallible and expect patients to be good little children and obey.
I read this book a while back, and like you found it interesting though not always practical. Still, the use of the word ‘significant’ is probably the most important point in the entire book for the general population, IMHO.
It’s a shame so many people in the media are science bubble heads. IOW, nothing much inside.
You’d think reporters on the science beat would be required to demonstrate some basic understanding of science before getting the job. That sure doesn’t seem to be the case.
Well it is reasonable to assume that those who generate news content be held to a higher standard of awareness. I am not sure it is reasonable to think that investigative journalism is going on. Ask Sharyl Attkisson how trying to get to the truth worked for her in the eyes of CBS.
A large percentage of the general population doesn’t even understand the probability of a coin flip. I heard someone boasting a few weeks ago that he had left the casino “up” 22 straight times (he mostly plays roulette). It’s the world we live in and people will continue to take advantage of other folk’s limitations as long as they can get away with it.
Tom!
You should read this one:
http://rsos.royalsocietypublishing.org/content/1/3/140216
Love this conclusion at the end of his explanations:
“All of the approaches above suggest that if you use p=0.05 as a criterion for claiming that you have discovered an effect you will make a fool of yourself at least 30% of the time. This alone implies that many published claims are not true.”
Please, read Section 11 (about Berger’s approach) of the paper pointed out by Sten Rylander. So when you say (I don’t know if the error is already in Ellenberg’s book) “If the p-value is .10, there’s a 10% chance the results were due to chance” should be corrected to something like “If the p-value is .10, there’s a 10% chance of getting a difference of outcomes between intervention and control groups of such a magnitude as measured or greater when actually there is no real effect of the intervention”. You could say also from Table 1 at Appendix A “If the p-value is .10, there’s at least a 38.5% chance the results were due to chance”. This is hugely important. Really.
And this video:
https://www.youtube.com/watch?v=tRZMD1cYX_c
I remember the green jelly beans example from xkcd!
The xkcd post dates from 2011, and Ellenberg’s book dates from 2014.
I hope Ellenberg gives the credit in his book (in which case it would be nice for you to do the same).
There’s a cartoon in the book which he identifies as an “xkcd cartoon.” What is xkcd?
A comic. xkcd.com
http://www.explainxkcd.com/wiki/index.php?title=882:_Significant
Done derivation of the jelly bean example is pretty common. I got the same example in the 90s.
It’s an online comic strip. It’s rather intelligent and funny. I found a few comic strips I don’t agree with, but most is based on science and is hilarious at the same time.
http://xkcd.com/
It’s a webcomic of romance, sarcasm, math and language.
Tom, I’ve been looking at your recent interviews like the one my name links to. I was afraid you might have picked up a rural country accent but boy was I wrong; someone’s been practicing his speech!
Too bad you look a little worn out; it sounds like you’ve maybe been dedicating too much time to computer stuff… Also, nice aquarium!
I know you mentioned somewhere earlier that you did a bunch of podcast/video interviews recently, yet I’m having a hard time finding them on your site; I have to google your name and add things like “podcast” or “interview”. Maybe you should add an extra section on your sidebar close to the Fat Head News/Reviews section just for external video & podcast interviews.
Those can be interesting to listen to especially for people with wireless headphones who can listen while they work on something else.
Holy Toledo, I have a British accent now. Someone should have told me.
Well, it looks like I can get some good interview info by using your own site to search for “interview”. I wish I’d thought of that sooner. In my defense I find some websites’ internal search functions are really bad so I trained myself to go directly to google.
When I test the search field, “interview” finds some and “podcast” finds some. I probably should add a tag that makes it easy to find all of them.
Tom!
You should read this one:
http://rsos.royalsocietypublishing.org/content/1/3/140216
Love this conclusion at the end of his explanations:
“All of the approaches above suggest that if you use p=0.05 as a criterion for claiming that you have discovered an effect you will make a fool of yourself at least 30% of the time. This alone implies that many published claims are not true.”
Assuming the experiment was fair and the experimenters did not fudge the data, run many experiments to find one that gave the desired answer, nor Finagled the abstract.
Of course, there is no requirement to publish the raw data. How can anyone check an experiment without access to the ra data?
They can’t, and that’s the plan.
Point. They all know what happened to TCC.
Please, read Section 11 (about Berger’s approach) of the paper pointed out by Sten Rylander. So when you say (I don’t know if the error is already in Ellenberg’s book) “If the p-value is .10, there’s a 10% chance the results were due to chance” should be corrected to something like “If the p-value is .10, there’s a 10% chance of getting a difference of outcomes between intervention and control groups of such a magnitude as measured or greater when actually there is no real effect of the intervention”. You could say also from Table 1 at Appendix A “If the p-value is .10, there’s at least a 38.5% chance the results were due to chance”. This is hugely important. Really.
And this video:
https://www.youtube.com/watch?v=tRZMD1cYX_c
I remember the green jelly beans example from xkcd!
The xkcd post dates from 2011, and Ellenberg’s book dates from 2014.
I hope Ellenberg gives the credit in his book (in which case it would be nice for you to do the same).
There’s a cartoon in the book which he identifies as an “xkcd cartoon.” What is xkcd?
A comic. xkcd.com
http://www.explainxkcd.com/wiki/index.php?title=882:_Significant
Done derivation of the jelly bean example is pretty common. I got the same example in the 90s.
It’s an online comic strip. It’s rather intelligent and funny. I found a few comic strips I don’t agree with, but most is based on science and is hilarious at the same time.
http://xkcd.com/
It’s a webcomic of romance, sarcasm, math and language.
Tom, I’ve been looking at your recent interviews like the one my name links to. I was afraid you might have picked up a rural country accent but boy was I wrong; someone’s been practicing his speech!
Too bad you look a little worn out; it sounds like you’ve maybe been dedicating too much time to computer stuff… Also, nice aquarium!
I know you mentioned somewhere earlier that you did a bunch of podcast/video interviews recently, yet I’m having a hard time finding them on your site; I have to google your name and add things like “podcast” or “interview”. Maybe you should add an extra section on your sidebar close to the Fat Head News/Reviews section just for external video & podcast interviews.
Those can be interesting to listen to especially for people with wireless headphones who can listen while they work on something else.
Holy Toledo, I have a British accent now. Someone should have told me.
Well, it looks like I can get some good interview info by using your own site to search for “interview”. I wish I’d thought of that sooner. In my defense I find some websites’ internal search functions are really bad so I trained myself to go directly to google.
When I test the search field, “interview” finds some and “podcast” finds some. I probably should add a tag that makes it easy to find all of them.
You should put up a permanent link somewhere on your website (i.e. somewhere in the left sidebar) that’ll bring people to a search page with the keywords put in so they can see all your interviews/podcasts. That would be easier than listing all the interviews on a new sidebar section and it would remind/help people to use your site’s search function.
If you’re planning on doing this, remember people who want to read all your interviews won’t necessarily want to hear your podcasts and vice-versa, so you’d almost have to make two links would automatically enter the right keyword and search it; one for podcasts one for written interviews.
That’s assuming you’re thinking of doing that because honestly it doesn’t take THAT much brain power to put in “podcast” in the search field. Having a button there would just be a little more user-friendly for those who can’t find the search field, and it’s good for impulsive browsers to checkout your podcast on a whim once they see said link
There’s always money on the line, no matter who’s paying for the research. I don’t see why that’s reason enough to reject any study conducted by a drug company.
By the way, I think this book would be a good option for a review, Tom.
http://www.amazon.com/Pharmaphobia-Conflict-Interest-Undermines-Innovation/dp/1442244623
This is a big reason why we need to get government as far away from the scientific funding process as possible. We want more decentralized funding, where competition will force these controversies out and ultimately promote tighter standards and less fraud (no point in publishing less than honest results if your competitors are going to bust you for it right away)…we don’t get those built-in controls with centralized funding, where the only competition occurs before the experiment is done and the results are published.
We ‘Mericans would do well to apply this principle in most areas, including health care, for instance. But that suggestion tends to invite highly emotional screeches from people who can’t envision a society where government doesn’t waste everyone’s money. So I’m not going to hold my breath.
The more money on the line, the bigger the temptation to fudge. There are millions and sometimes billions on the line for new drugs. After reading how drug companies routinely fudge the data, cherry-pick subjects for what are supposed to be randomized trials, etc., I’m highly suspicious — especially since their results so often fail to be replicated in independent trials.
Dr. Beatrice Golomb also has an excellent lecture on the subject.
https://www.youtube.com/watch?v=nFtt-W3LROY
There’s corruption, no doubt. But if the pharmaceutical companies did nothing but develop ineffective drugs based on unethical research, they’d quickly go out of business, snuffed out by someone willing to develop real treatments.
As fun as it is to criticize drug companies, the reality is that they’ve introduced all kinds of life-saving medicines–vaccines, cancer treatments, anti-depressants–and made our lives better as a result.
Agreed. Take away modern drugs, many of us wouldn’t have lived to see age 20. But developing some truly stellar life-saving drugs is not a license to fudge when running trials on other drugs.
How could we have possibly evolved as a species without pharma? Pharma, to their credit, is creating band-aids for the problems of civilization. Cancer and depression are better treated by environment than by pharma. Nature and humans are not fundamentally broken; the environment we’re in, i.e. civilization, may be what’s broken.
There’s always money on the line, no matter who’s paying for the research. I don’t see why that’s reason enough to reject any study conducted by a drug company.
By the way, I think this book would be a good option for a review, Tom.
http://www.amazon.com/Pharmaphobia-Conflict-Interest-Undermines-Innovation/dp/1442244623
The more money on the line, the bigger the temptation to fudge. There are millions and sometimes billions on the line for new drugs. After reading how drug companies routinely fudge the data, cherry-pick subjects for what are supposed to be randomized trials, etc., I’m highly suspicious — especially since their results so often fail to be replicated in independent trials.
Dr. Beatrice Golomb also has an excellent lecture on the subject.
https://www.youtube.com/watch?v=nFtt-W3LROY
There’s corruption, no doubt. But if the pharmaceutical companies did nothing but develop ineffective drugs based on unethical research, they’d quickly go out of business, snuffed out by someone willing to develop real treatments.
As fun as it is to criticize drug companies, the reality is that they’ve introduced all kinds of life-saving medicines–vaccines, cancer treatments, anti-depressants–and made our lives better as a result.
Agreed. Take away modern drugs, many of us wouldn’t have lived to see age 20. But developing some truly stellar life-saving drugs is not a license to fudge when running trials on other drugs.
How could we have possibly evolved as a species without pharma? Pharma, to their credit, is creating band-aids for the problems of civilization. Cancer and depression are better treated by environment than by pharma. Nature and humans are not fundamentally broken; the environment we’re in, i.e. civilization, may be what’s broken.
This is a big reason why we need to get government as far away from the scientific funding process as possible. We want more decentralized funding, where competition will force these controversies out and ultimately promote tighter standards and less fraud (no point in publishing less than honest results if your competitors are going to bust you for it right away)…we don’t get those built-in controls with centralized funding, where the only competition occurs before the experiment is done and the results are published.
We ‘Mericans would do well to apply this principle in most areas, including health care, for instance. But that suggestion tends to invite highly emotional screeches from people who can’t envision a society where government doesn’t waste everyone’s money. So I’m not going to hold my breath.
Figures don’t lie. But liars figure. 🙂
“There are 3 types of lies. Lies, damn lies, and statistics.” – Disraeli
Prime example is the Lipitor statistics. 33% REDUCTION IN HEART ATTACK RISKS!! But actual numbers show only 1 person in 100 is/may be actually helped.
Yup, they trot out relative risk reduction because it’s so much more impressive than actual risk reduction.
And that one person who avoid a myocardial infarction may be suffering from muscle weakness or brain fog.
NNT>NNH
Figures don’t lie. But liars figure. 🙂
“There are 3 types of lies. Lies, damn lies, and statistics.” – Disraeli
Prime example is the Lipitor statistics. 33% REDUCTION IN HEART ATTACK RISKS!! But actual numbers show only 1 person in 100 is/may be actually helped.
Yup, they trot out relative risk reduction because it’s so much more impressive than actual risk reduction.
And that one person who avoid a myocardial infarction may be suffering from muscle weakness or brain fog.
NNT>NNH
Big fan of yours, but disappointed by the use of “my-friend’s-brother-said” as a resource/data point. No offense intended–I hold you to a pretty high standard.
I agree it would be hearsay in court.
Quick fix easy, but politically impossible:
No public data = no patent.
Cheers
I elected not to go there in the post, but even as a libertarian (thanks to some sibling who beat me into submission during a debate and then recommended some books at my request), I can make a case for guvmint acting to protect the public from fraud — a legitimate function of guvmint. No patent, or perhaps no FDA approval unless all clinical trials are announced at the outset (so the negative trials can’t just disappear), all trial results are published, and all trial data is made available for outside review.
Happy anniversary to you and The Older Wife, BTW. Ours is today, yours is tomorrow if memory serves.
Big fan of yours, but disappointed by the use of “my-friend’s-brother-said” as a resource/data point. No offense intended–I hold you to a pretty high standard.
I agree it would be hearsay in court.
Quick fix easy, but politically impossible:
No public data = no patent.
Cheers
I elected not to go there in the post, but even as a libertarian (thanks to some sibling who beat me into submission during a debate and then recommended some books at my request), I can make a case for guvmint acting to protect the public from fraud — a legitimate function of guvmint. No patent, or perhaps no FDA approval unless all clinical trials are announced at the outset (so the negative trials can’t just disappear), all trial results are published, and all trial data is made available for outside review.
Happy anniversary to you and The Older Wife, BTW. Ours is today, yours is tomorrow if memory serves.
And, as always seems to be the case: xkcd HAS the correct cartoon for you (and this blog entry!) Tom:
http://xkcd.com/1478/
Perfect.
And, as always seems to be the case: xkcd HAS the correct cartoon for you (and this blog entry!) Tom:
http://xkcd.com/1478/
Perfect.
Your memory is better than mine. Late happy anniversary to you and Chareva!
Cheers
Your memory is better than mine. Late happy anniversary to you and Chareva!
Cheers
Hi, Tom, this comment has nothing to do with math, but I ran across something that made me think of you and your chicken adventures, and I thought you’d appreciate this idea. It’s somebody’s solution for catching snakes in their chicken coop. Use fish traps with eggs for bait. Pretty cool! Take care, and have a happy and safe 4th!
Very nice idea. Simple and effective. Thanks for the link.
Hi, Tom, this comment has nothing to do with math, but I ran across something that made me think of you and your chicken adventures, and I thought you’d appreciate this idea. It’s somebody’s solution for catching snakes in their chicken coop. Use fish traps with eggs for bait. Pretty cool! Take care, and have a happy and safe 4th!
Very nice idea. Simple and effective. Thanks for the link.
Scientists need deep explanation. Without it, the studies mean nothing, no matter the sheer amount only measuring effects. The randomized clinical trial is not the be all, end all. Lots of confounding variables still exist in spades with humans-thousands of variables. Biology is hellish.
We need to understand HOW green jellybeans could cause acne for the study to be valid in your hypothetical example . We need deeper understanding of human cellular operations.
“Drug companies aren’t required to publish all their results, so they don’t. They aren’t required to share the raw data with other scientists, so they don’t.”
Hmmmm…say, that sounds an awful lot like climate scientists :)…and they’re TAXPAYER funded.
Don’t get me started.
Scientists need deep explanation. Without it, the studies mean nothing, no matter the sheer amount only measuring effects. The randomized clinical trial is not the be all, end all. Lots of confounding variables still exist in spades with humans-thousands of variables. Biology is hellish.
We need to understand HOW green jellybeans could cause acne for the study to be valid in your hypothetical example . We need deeper understanding of human cellular operations.
“Drug companies aren’t required to publish all their results, so they don’t. They aren’t required to share the raw data with other scientists, so they don’t.”
Hmmmm…say, that sounds an awful lot like climate scientists :)…and they’re TAXPAYER funded.
Don’t get me started.