For real people, if something works in theory, but not in practice, it doesn’t work. For academics, if something works in practice, but not in theory, it doesn’t exist. – Nassim Nicholas Taleb
I thought I was finished with this mini-series of follow-up posts on my recent speech, but I want to address one more comment I’ve heard here and there … the one that goes something like Why would you criticize observational studies, then turn around and praise people for making decisions on a bunch of anecdotes?
Okay, I understand the objection. But anecdotes and observational studies are not exactly the same thing. In an observational study, researchers try to figure out what people have been eating for the previous several years (usually with notoriously inaccurate food-recall surveys), then look for statistical correlations to health outcomes.
Aha! People who eat bacon have a marginally greater risk of developing colon cancer! Bacon must cause cancer!
That sort of thing. Those studies certainly keep academics busy, but as Dr. John Ioannidis and others have pointed out, researchers often find what they expect to find. As Ioannidis pointed out in one tongue-in-cheek article, most ingredients in a common cookbook have been linked to a higher rate of cancer … and also to a lower rate of cancer.
People in observational studies aren’t intentionally changing their diets in hopes of achieving a result. They’re just fodder for number-crunching researchers.
In a clinical study, there’s an intentional intervention: researchers have one group of people take a drug, or switch from animal fats to corn oil, or take fish oil, or whatever. Then they compare those people to a control group of people who didn’t change anything and look for differences in health outcomes. Clinical studies are the gold standard in research — and they should be.
But suppose you suffer from migraines and several doctors have failed to find the magic pill that provides relief. Now suppose someone at a dinner party says he’s heard from several people that their migraines went away when they stopped eating grains. (This happened with the wife of a co-worker of mine, as you may recall.)
Are you SERIOUSLY supposed to wait until someone designs a clinical study on grain-removal and migraines, receives funding, gets approval, conducts the study and reports the results before trying it out for yourself? Are you going to say to yourself, Well, those are just anecdotes, so I’ll continue going from doctor to doctor and suffering from the migraines until I find that magic pill. Not a chance. You’ll give up the grains and see what happens. If the migraines go away, you’ll no doubt tell some people.
Ka-boom … we have knowledge based on experience. (But gosh darn it, it’s only anecdotal evidence!)
I understand the problem with anecdotes, of course. What appears to be cause and effect could simply be coincidence. I have a nagging digestive problem, I happen to start drinking green tea, and the digestive problem – which was going to subside anyway for other reasons – goes away. But I’m all excited about the result and run off to share my belief that green tea fixed my digestive problems on the internet.
Sure, that happens. But that’s where the Marketplace of Ideas concept comes into it. People promote all kinds of ideas. Some are good, some are bad, some are just useless. But the good ideas are the ones that stick around. They’re the ideas that gain traction … because they’re the ideas that actually work.
When people try a new diet, they’re engaging in an experiment. There’s an intentional intervention – the new diet. There are results to compare – themselves on the old diet vs. themselves on the new diet. Yes, it’s just an n=1 experiment. We can’t necessarily extrapolate the results to everyone else. But it’s still an experiment.
In the Fat Head Kids book and movie, we recounted the n=1 experiment Sam Feltham conducted. For 21 days, he consumed more than 5,000 calories per day of low-carb/high-fat real foods. He gained less than three pounds and actually lost an inch around his waist. After a washout period, he again consumed more than 5,000 calories per day for 21 days – this time getting most of the calories from sugars and refined grains. He gained almost 16 pounds and got fatter around the waist. Same number of calories, very different results in terms of both weight and body composition.
When Feltham reported his results, the response from the “it’s all about the calories” crowd was predictable: This is just anecdotal evidence! Not a randomized controlled trial, so we can just ignore it! And my favorite: this is a jokexperiment!
Oh really? And why was it a joke? I only see three possibilities here:
- Feltham is a liar and didn’t actually eat what he recorded or get the results he reported.
- Feltham isn’t intelligent enough to accurately record what he consumed or measure his results.
- Feltham ate what says he ate and got the results he reported.
Feltham is clearly an intelligent guy, so we can dismiss #2. If people would rather assume he’s a liar than risk changing their minds … well, okay.
But I think the only likely possibility is that he ate what he says he ate and got the results he reported. So even though it’s just an n=1 experiment, it still tells us something important. If Sam Feltham can experience wildly different results from changing what he ate (as opposed to how much he ate), then so can other people. I’m pretty sure he’s not unique among humans. I’m also pretty sure we can’t write off the changes in weight as a coincidence.
Same goes for what I mentioned in my previous post about stuffing myself with meats, eggs and greens on the low-carb cruises but not gaining any weight. I’ve done it several years in a row now. I’ve heard from many other people who had the same experience. I corresponded with a doctor who saw that among his patients – not all of them, but many of them. Add up all those n=1 experiments, and you can’t help but notice a pattern emerge.
There will never be enough clinical trials conducted to tell us everything we’d like to know about diet and health – especially since so many trials are funded and conducted by people who want to sell us something. That’s why we have to largely rely on the Wisdom of Crowds. That wisdom is based on experience, and that’s what anecdotes are: people’s experiences. What’s changed in the internet era is that we can share experiences with countless people all over the world.
I wouldn’t look at anecdotes about the effects of a diet and declare This Is The Way It Is, Period. But I’ll certainly look at them as evidence that a particular change in diet might work. And if it works, it works. I don’t especially care if The Anointed in academia agree.
Besides, wait another 20 years or so, and they’ll be saying they knew it all along.