To feign credibility for their holy propaganda, religious professionals love to preface their sales talk by saying something like “in the original language our holy book says …” Read what I wrote here about that deceptive trick.
Such deception is not monopolized only by referring to religious texts. People use science jargon in the same way. People who preface their sales points by saying “studies show” may also be trying to say, “look, this is science, don’t question it.” or “I know my stuff, don’t question me.” So it is not just the religion hucksters who play the game.
I once taught a course on “Medical Research” to graduate students. My course goal was to teach the future physician assistant students to see through the bullshit of drug reps and to not fall prey to smokescreens of “science” talk. I wasn’t trying to teach them to be researchers but to see through the smokescreens of researchers or the hucksters who cherry pick research.
Unfortunately, learning to analytically dissect supposed scientific studies and to see through their shortcomings takes a lot of training. But the first step to being objective is to never believe someone who says “studies show” without demanding much more evidence. And if the science salesman or woman actually happens to hand you the study they are quoting, the first things to examine include:
- the reputation of the source journal
- the research funding source and the vested interests of the researcher
- the number of people studied and the population chosen
- the type of study (qualitative vs. quantitative, epidemiological vs experimental, retrospective vs prospective, etc, etc). Understanding study types is a bit tough. But it allows you to see through the common deceptions:
- Correlation vs Causation deceptions
- Generalization deceptions
- The Ecological Fallacy
So beware: In developed countries, among the highly educated, science is the new Bible and “Studies shows” is the equivalent of “In the original Greek” for Bible pushers. Don’t be hoodwinked by either.
If you want to watch a professional researcher analyze studies, go to Tom Reese’s site “Epiphenom” where he reviews studies on religion done in all sorts of journals.
Questions to readers: Do you have other cautions for beginners to consider when questioning a study?
Triangulations 
Maybe I’m not seeing it at the right angle, but I’m having a hard time meshing “Studies shows” as the equivalent of “In the original Greek”. I see original language cases as a different kind of strategy than citing studies.
Don’t get me wrong. I completely agree with the need to be cautious about what “studies show”. I would venture to add to that the recommendation of doubling the level of caution when those studies are showing results that you want them to show.
But as for the Greek, or Hebrew, or Arabic, or Sanskrit, I see that as a method of distracting someone from the big picture, and trying to build an attractive richness to the story which, if taken just in its outline, is a bit flimsy in its veracity.
@TWF,
Good question and doubt.
This post was meant as a preface to another which may help.
Yet I write far more prefaces than follow-ups. ;-(
So in case I don’t write it, let me try part of the explanation now:
Non-scientist types (religious and atheist alike) use the phrase “studies show” often without any support — first, they can’t name they article, second if they can , they probably never read it but only someone’s spin on it. Heck, even science types use the phrase with the same weak support.
The point, these folks use the phrase “studies show” in hope that it will add credibility to their opinion and even perhaps shut down disagreement when it does. Likewise, many religious professionals throw around “in the original it says …” in a similar way.
Does that make it clear?
Indeed “studies” are suppose to be open to empirical dialogue which is far better than any religious dialogue, so in that sense (as I think you are eluding), they are very different.
Ahhhh, yes. That’s definitely true. And I know I’ve been guilty of that “sin” myself! 😉
TWF: Great, glad it works and today, just for you, I put up a post illustrating this trick.
I agree. Some other things to look for
Blinding. Double blinded RCTs are standard in Medicine, but very rare elsewhere.
Objectivity of measures.
Number of measures (a 95% confidence in a result, when there are 10 measures, is not impressive). Not just in medical research – that one’s a doozy in all kinds of research. “We looked at 12 different ways to do X, and found two that are particularly promising” yeah, no kidding.
Duration of effect. Does the study only show very short-term reaction, but are being touted as if they are long-term solutions?
Drop out rate. Someone I knew who did drug trials for a big pharma co. said to watch these numbers. In there a whole legion of problems are hidden.