The latest “Low-Carb Diets Are Deadly!” study, the topic of my previous post, was based on data extracted from about a dozen food questionnaires mailed out over a 20-year span. Because so many studies rely on these questionnaires to draw headline-grabbing conclusions about associations between diets and disease, it’s important to understand just how lousy the accuracy rate is.
Twenty-five years ago, I completed one of those questionnaires as part of a wellness program sponsored by the company that employed me. (In fact, we were all required to complete one.) Many of us ended up laughing at the stupidity of the thing. Imagine being a young, single male who eats at least half his meals outside the home. Now try to accurately answer this question:
How many servings of ground beef did you consume in the previous month?
The truthful answer would’ve been “I have no friggin’ idea.” But I was required to answer the question. So my method of guesstimation went something like this:
Let’s see … I think I ate lunch at McDonald’s three times last week … no, wait … was it four times? Did I have Thai food with Frank last Friday, or was that the week before? Ah, hell, let’s say four lunches at McDonald’s. Okay … I think maybe I had the chicken sandwich once. No, it was twice. No, it was once. So I’ll say I had burgers three times. Did I get the Double Quarter Pounder that one day? I think I skipped breakfast and ended up eating the double. So that’s … well, if one burger patty is a serving, I must’ve had four patties. No idea about the whole month, so I’ll multiply four times last week by four weeks in a month and say I had sixteen burger patties. Maybe Mexican food twice, I might’ve had beef and broccoli once … Okay, I’ll just round it to up 20 servings of beef for the month.
By the time I was halfway through the questionnaire, I was so annoyed by all the guesstimation required, I just started filling in numbers that sounded sort of believable. I know several co-workers did likewise, because we talked about it later.
Here’s another example posted on the Diabetes Update site:
I know first hand how inaccurate the standard food frequency questionnaire is because several years ago I was a subject in a long term study that used the standard nutritionist designed food intake questionnaire. During this time I was logging my actual food intake, with weighed portions, trying to understand my own pattern of weight loss, so I knew exactly what I was eating during any given day or month.
The nutritionist associated with the study emailed me the nutritional breakdown that I had supposedly eaten, based on my answers to the standard food frequency questionnaire. It bore no relationship at all to what I had eaten either in terms of calories or the percentages of my diet represented by protein, carbs, or fat. When I offered to send the study my actual food intake, I was told that the questionnaire they were using had been carefully validated and was standard in all nutritional studies and that there was no point in looking at what I had actually eaten.
Some questionnaires lump all red meats into a single category — in other words, a fresh steak and a highly-processed frozen beef lasagna dinner are counted the same — and some even put butter and margarine into the same category … natural butter fat, artificially processed corn oil, hey, makes no difference if they both spread nicely on your dinner roll.
From this kind of data, researchers at Harvard believe they know what people ate over a 20-year period. And since he likes their conclusions, Dean Ornish believes it too.
If you enjoy my posts, please consider a small donation to the Fat Head Kids GoFundMe campaign.
The more I read, the more convinced I am that there’s very little much real science in so-called “Nutrition Science”. The standards are much much lower there than in other sciences. No experimental physicist could ever get away with this kind of garbage work. But in nutrition science it’s taken a high holy gospel.
Sigh. I suppose that the U.S. is doomed to get fatter and stupider.
I’d love it if anyone could point me to an example of one. I like to see a copy of an actual study version of a food intake questionaire.
I copied some examples in this post:
http://www.fathead-movie.com/index.php/2009/07/07/warning-bologna-may-cause-cancer-headlines/
“When I offered to send the study my actual food intake, I was told that the questionnaire they were using had been carefully validated and was standard in all nutritional studies and that there was no point in looking at what I had actually eaten.”
So they ignored what she said, and then lied about what she said to other people. Classy.
But it fit the data scheme, and apparently that’s what matters.
These researchers just end up outsmarting themselves with their little questionaires that are useless at getting any kind of accurate information for their data mining studies. Once again, it all boils down to garbage in, garbage out (and the gullible hogs out there just love feeding on it).
GIGO, as we programmers call it.
The more I read, the more convinced I am that there’s very little much real science in so-called “Nutrition Science”. The standards are much much lower there than in other sciences. No experimental physicist could ever get away with this kind of garbage work. But in nutrition science it’s taken a high holy gospel.
Sigh. I suppose that the U.S. is doomed to get fatter and stupider.
I’m obsessed with food, buy all the food at my house, pack my lunches, and rarely eat out. I read labels, avoid certain ingredients, and read scientific papers on food. (The older ones are less ambitious than today’s big epidemiological studies, but they’re specific about the experiments and they’re written in plain English.)
I couldn’t tell you what I ate for lunch and dinner two days ago, though.
I’d love it if anyone could point me to an example of one. I like to see a copy of an actual study version of a food intake questionaire.
I copied some examples in this post:
http://www.fathead-movie.com/index.php/2009/07/07/warning-bologna-may-cause-cancer-headlines/
“When I offered to send the study my actual food intake, I was told that the questionnaire they were using had been carefully validated and was standard in all nutritional studies and that there was no point in looking at what I had actually eaten.”
So they ignored what she said, and then lied about what she said to other people. Classy.
But it fit the data scheme, and apparently that’s what matters.
These researchers just end up outsmarting themselves with their little questionaires that are useless at getting any kind of accurate information for their data mining studies. Once again, it all boils down to garbage in, garbage out (and the gullible hogs out there just love feeding on it).
GIGO, as we programmers call it.
And of course, there’s the question of how honestly you choose to complete them too. Do you confess to the glass of wine each night? What if you regularly have three? How about the king size chocolate bar? Did you have two helpings of dessert at the buffet last weekend? Or just load one plate up so high you might as well have? Did you really only have a teaspoon of butter on your four slices of grainy toast for breakfast. How many vegetables did you really eat?
It’s human nature to want to make a good impression, so when estimating on your food recall chances are you’ll overestimate the ‘good’ stuff and underestimate (or totally ‘forget’) about the ‘bad’ stuff.
Just an aside Tom, I’m from Christchurch which you may or may not have heard was hit by a major earthquake last weekend – and we’ve had hundreds of aftershocks since. Here’s some pics of the amazing power of Mother Nature:
http://www.crashbang.co.nz/quake040910/index.html?ref=nfMo
Bizarrely, in our neighbourhood, you’re hard pressed to find much evidence other than odd missing chimney – but it does tend to throw a healthy dose of perspective into your day (which I’m sure you understand from the flooding in your town recently).
PS feel free to edit the earthquake stuff out of my comment as I know it’s off topic – just thought you might like a wee peek 🙂
I think people definitely forget a few items on purpose. That was quite an earthquake. As the Nashville-area flood reminded us, Mother Nature can still kick our butts.
I’m obsessed with food, buy all the food at my house, pack my lunches, and rarely eat out. I read labels, avoid certain ingredients, and read scientific papers on food. (The older ones are less ambitious than today’s big epidemiological studies, but they’re specific about the experiments and they’re written in plain English.)
I couldn’t tell you what I ate for lunch and dinner two days ago, though.
And of course, there’s the question of how honestly you choose to complete them too. Do you confess to the glass of wine each night? What if you regularly have three? How about the king size chocolate bar? Did you have two helpings of dessert at the buffet last weekend? Or just load one plate up so high you might as well have? Did you really only have a teaspoon of butter on your four slices of grainy toast for breakfast. How many vegetables did you really eat?
It’s human nature to want to make a good impression, so when estimating on your food recall chances are you’ll overestimate the ‘good’ stuff and underestimate (or totally ‘forget’) about the ‘bad’ stuff.
Just an aside Tom, I’m from Christchurch which you may or may not have heard was hit by a major earthquake last weekend – and we’ve had hundreds of aftershocks since. Here’s some pics of the amazing power of Mother Nature:
http://www.crashbang.co.nz/quake040910/index.html?ref=nfMo
Bizarrely, in our neighbourhood, you’re hard pressed to find much evidence other than odd missing chimney – but it does tend to throw a healthy dose of perspective into your day (which I’m sure you understand from the flooding in your town recently).
PS feel free to edit the earthquake stuff out of my comment as I know it’s off topic – just thought you might like a wee peek 🙂
I think people definitely forget a few items on purpose. That was quite an earthquake. As the Nashville-area flood reminded us, Mother Nature can still kick our butts.
I just hope to God that nobody every wants my cooperation in filling out food questionnaires let alone require my assistance in a criminal investigation of some sorts. “Where were you on the 22nd of July and what were you doing between three and six in the afternoon?”
How am I supposed to know? My diary might provide some general idea but other than that my mind would be a complete blank. Unless I’d done the murder, of course.
If you asked me how many servings of beef, chicken, eggs, nuts, vegetables, etc. I ate last month, I’d have no flippin’ clue. I don’t even remember what I ate two days ago.
I just hope to God that nobody every wants my cooperation in filling out food questionnaires let alone require my assistance in a criminal investigation of some sorts. “Where were you on the 22nd of July and what were you doing between three and six in the afternoon?”
How am I supposed to know? My diary might provide some general idea but other than that my mind would be a complete blank. Unless I’d done the murder, of course.
If you asked me how many servings of beef, chicken, eggs, nuts, vegetables, etc. I ate last month, I’d have no flippin’ clue. I don’t even remember what I ate two days ago.
Interestingly, I’ve gotten some of my family to understand that sugar and refined grains are the true reasons behind their heart disease and diabetes. Yet we’ll start talking about the evils of such, and they’ll say, “Yes, I am so glad I stopped eating sugar!” And I will ask them, “But what about the pie and ice cream you had last night? And didn’t you have pancakes for breakfast?” “Oh, I forgot about those! …well that’s an exception!” So if you ask me, I’d say that those who wish to be healthy (but lack follow-thru) put down in these questionnaires how they wanted to eat, rather than how they really did eat. (Not to mention the questionnaires being suspect as well.)
And it doesn’t have to be intentional. People who fool themselves into believing they eat healthy may forget a few foods when filling out the questionnaires.
@Tracey, I think someone mentioned it in the comments to the last post here, but Chris Masterjohn has a terrific post about the accuracy of those food questionnaires.
http://blog.cholesterol-and-health.com/2010/09/new-study-shows-that-lying-about-your.html
Scroll down halfway and you’ll see that in the Nurses’ Health Study, daily food logs indicated that the subjects’ monthly questionnaires showed a 1.4% accuracy re: hamburger consumption.
I don’t think the subjects are lying, necessarily, they probably really don’t remember. The thought process is probably, “I eat healthfully. Burgers aren’t healthful. Therefore, I didn’t eat that many–maybe just one last month.” This does go to show that for someone who is struggling with weight loss or food intolerance, a daily food log and a food scale could shed some light on the problem.
The data gathered for this study is especially prone to the type of errors he describes because the subjects WERE NOT ON A DIET. They were simply asked what they ate. When we choose to follow a specific diet, such as Atkins, we we’re more consciously aware of our daily choices. But for this study, they just grabbed thousands of questionnaires and labeled the “low carb” (ahem, ahem) eaters as Atkins dieters.
Interestingly, I’ve gotten some of my family to understand that sugar and refined grains are the true reasons behind their heart disease and diabetes. Yet we’ll start talking about the evils of such, and they’ll say, “Yes, I am so glad I stopped eating sugar!” And I will ask them, “But what about the pie and ice cream you had last night? And didn’t you have pancakes for breakfast?” “Oh, I forgot about those! …well that’s an exception!” So if you ask me, I’d say that those who wish to be healthy (but lack follow-thru) put down in these questionnaires how they wanted to eat, rather than how they really did eat. (Not to mention the questionnaires being suspect as well.)
And it doesn’t have to be intentional. People who fool themselves into believing they eat healthy may forget a few foods when filling out the questionnaires.
@Tracey, I think someone mentioned it in the comments to the last post here, but Chris Masterjohn has a terrific post about the accuracy of those food questionnaires.
http://blog.cholesterol-and-health.com/2010/09/new-study-shows-that-lying-about-your.html
Scroll down halfway and you’ll see that in the Nurses’ Health Study, daily food logs indicated that the subjects’ monthly questionnaires showed a 1.4% accuracy re: hamburger consumption.
I don’t think the subjects are lying, necessarily, they probably really don’t remember. The thought process is probably, “I eat healthfully. Burgers aren’t healthful. Therefore, I didn’t eat that many–maybe just one last month.” This does go to show that for someone who is struggling with weight loss or food intolerance, a daily food log and a food scale could shed some light on the problem.
The data gathered for this study is especially prone to the type of errors he describes because the subjects WERE NOT ON A DIET. They were simply asked what they ate. When we choose to follow a specific diet, such as Atkins, we we’re more consciously aware of our daily choices. But for this study, they just grabbed thousands of questionnaires and labeled the “low carb” (ahem, ahem) eaters as Atkins dieters.
To follow up on Lori’s comment, Chris Masterjohn concluded that lying about hamburger intake gives you heart disease. 🙂
Makes sense. If there’s an association, it must be cause and effect.
We used those questionnaires when I worked in market research. We’d have people come back every week for 6 weeks to answer a series of questions. Fortunately, it’s easier to remember what you ate in the last SEVEN DAYS than in the last THIRTY ONE.
It had stuff like, “How many servings of dark green, leafy vegetables have you eating since your last questionnaire?”
People routinely looked at us like we were nuts. Then, the real fun started. “What’s a “Dark green, leafy vegetable? Does iceberg lettuce count? It’s green and leafy, but it’s light green.” (Sure why not!)
“Do hotdogs count as beef, pork or turkey? The ones I had have all three, I think.” (You’re a little short on beef. It goes there.)
“I had a bunch of broccoli. Where do I record that?” (Nowhere)
You do realize, of course, this means you’re officially qualified to write academic papers using that data as if it were true and meaningful.
To follow up on Lori’s comment, Chris Masterjohn concluded that lying about hamburger intake gives you heart disease. 🙂
Makes sense. If there’s an association, it must be cause and effect.
We used those questionnaires when I worked in market research. We’d have people come back every week for 6 weeks to answer a series of questions. Fortunately, it’s easier to remember what you ate in the last SEVEN DAYS than in the last THIRTY ONE.
It had stuff like, “How many servings of dark green, leafy vegetables have you eating since your last questionnaire?”
People routinely looked at us like we were nuts. Then, the real fun started. “What’s a “Dark green, leafy vegetable? Does iceberg lettuce count? It’s green and leafy, but it’s light green.” (Sure why not!)
“Do hotdogs count as beef, pork or turkey? The ones I had have all three, I think.” (You’re a little short on beef. It goes there.)
“I had a bunch of broccoli. Where do I record that?” (Nowhere)
You do realize, of course, this means you’re officially qualified to write academic papers using that data as if it were true and meaningful.
It is the case with a lot of market surveys. I get to a point that I worry more that my math works out than if the response is an accurate representation of the truth. You would think that some level of validation is applied though you know it is not. Responses aren’t rejected if it turns out that some fool is eating 3 sides of beef a week.
Ailu, I understand your point but you must remember that even if people don’t always eat well, half the battle is them knowing what eating well is (and isn’t). As long as you are moving them in the right direction there you are a hero!
It is the case with a lot of market surveys. I get to a point that I worry more that my math works out than if the response is an accurate representation of the truth. You would think that some level of validation is applied though you know it is not. Responses aren’t rejected if it turns out that some fool is eating 3 sides of beef a week.
Ailu, I understand your point but you must remember that even if people don’t always eat well, half the battle is them knowing what eating well is (and isn’t). As long as you are moving them in the right direction there you are a hero!