When headlines hit the newsstands last month about a recent study showing that “religion is good for kids,” I got a little cranky.  Fortunately, Dr. Jean Mercer, a developmental psychologist, wrote an excellent white paper on the study. Her white paper is the inaugural contribution to the Secular Parenting Think Tank Project led by the Institute for Humanist Studies and the authors of Parenting Beyond Belief.    Dale McGowan, editor and co-author of Parenting Beyond Belief, also wrote an informative critique of the study for the Humanist Network News.   Thanks to their excellent work, my own critique of the study would be redundant.  Instead, I want to offer some tips and tricks for the skeptical reader of research findings.  You don’t have to be an expert in the field of study to use these tactics to help you assess the quality of research.

Top Ten Tactics for Textual Thomases

1. 200 Scientists Named Steve – Media outlets often pick up on a new study that has come out and report on its findings.  In reality, one study in science tells us very little. It is just one part of an ongoing investigation into an issue in which experts debate and disagree.  After a lot of testing and critiquing and many studies a scientific consensus starts to emerge, and that is how knowledge is built. Of course, even that consensus can be overturned in the light of new evidence, that’s the whole point.  When intelligent design advocates tried to claim a scientific consensus by trotting out a list of 100 scientists who questioned evolution, some folks in the scientific community pointed out the absurdity of this claim by creating a list of more than 200 scientists named Steve who supported evolution.  Since only about 1% of scientists are named Steve, and there is no reason to suspect they are disproportionately supportive of evolution, this is a pretty serious consensus.  When you are presented with one study in isolation, find out how much other research is out there that comes to similar conclusions, as opposed to how much other research comes to different conclusions.

2. Lack of Pirates causes Global Warming – Many statistical analyses are correlational. What that means is that they demonstrate that two things are related to one another, but not that one of them causes the other.  There are some types of study that can demonstrate causation, such as experimental designs that involve a control group and random assignment of study participants to the conditions in the study.  These types of experiments are used in most medical research on the effectiveness of treatments for diseases, for example. Surveys and other types of statistical research that doesn’t involve an experiment usually can’t assess causation.  Watch out for statements that claim X causes Y, when really all we know is that X and Y are linked somehow.  

The Church of the Invisible Flying Spaghetti Monster does a good job poking fun at the absurd conclusions that can be reached when correlation is confused with causation:

“You may be interested to know that global warming, earthquakes, hurricanes, and other natural disasters are a direct effect of the shrinking numbers of Pirates since the 1800s. For your interest, I have included a graph of the approximate number of pirates versus the average global temperature over the last 200 years. As you can see, there is a statistically significant inverse relationship between pirates and global temperature.” (see graph: http://www.venganza.org/piratesarecool4.jpg)

There are four possibilities here: (A.) decreasing numbers of pirates causes increasing temperatures. (B.) increasing temperatures causes decreasing numbers of pirates (reverse causality), (C.) something else causes decreasing pirates to correlate with increasing temperatures (spurious relationship or third-variable cause), and, of course, (D.) mere coincidence.  Keep in mind the possibility of reverse causality, or another factor causing the relationship whenever you read about new research findings from correlational studies.

3. It’s Significant, but is it Important? – Most statistical research uses a method called “null hypothesis significance testing” to determine whether the results achieved in the study are likely to be related to some actual variation in the population under study, or whether they occurred due to chance.  A typical standard in social scientific research is that things are considered significant if there is a less than 5% chance that those results would be achieved by chance if there was no relationship between the variables being tested.  Note two things here, first off, there is still a chance, albeit a small one, that “statistically significant” results may be achieved by chance.  By the same token, there is also a chance that researchers will find “no significant results” when there is an actual relationship.  Secondly, and more importantly though, the word “significant” does not mean that the effect is large, or important, or any of those other things that normal people mean when they use the word significant.  A relationship between variables in research can be statistically significant, and still be quite small, and thus not practically very important.  Try to get a sense of how large the relationship between the factors is, rather than assuming that “significance” means it’s an important relationship.

4. “You keep using that word, I do not think it means what you think it means”¹  – Often researchers are seeking to measure some concept that is not easy to quantify, but they have to quantify it in order to subject it to statistical analysis.  Sometimes this creates a serious disconnect between what the researcher is interested in measuring, and what s/he actually measures.  For example if we decided to measure whether a country was a democracy by looking only at whether the people there were allowed to vote, we’d get some really weird results.  Many places that are clearly not what we think of as democracies, including the Soviet Union before 1991, had people vote for candidates.  The trick is that the only candidates who were allowed on the ballot were those selected by the leadership, and they always ran unopposed.  What we think of as democracy is more than just people voting; if our measure doesn’t capture that, and we go on to say “Democracies behave like this,” people are probably going to get the wrong idea.  If a measurement for a concept doesn’t really capture the concept or make sense, we will get results that don’t mean what we think they mean.  Find out what is actually being measured when you read research because sometimes what the concept means and how it’s measured don’t match up.

¹ Inigo Montoya in The Princess Bride, commenting on his boss Vizzini’s continuing use of the word "inconceivable"for things that subsequently happen.

5.  “And all the children are above average,”²  – Sometimes measures may be closely related to the concept, but they can be problematic in other ways.  For example, in the Bartkowski et al (2007) study that I started off this piece with, the only measures are subjective – they are what parents and teachers say about children’s behavior when asked on a questionnaire, and how often parents say that they attend church.  Subjective measures can be useful for something, but they can also be prone to biases.  For example, the social desirability bias – people are prone to exaggerate their virtues, and minimize their vices, even on an anonymous survey.  This is one of the reasons that it’s hard to ask people about sensitive topics like prejudice, drug use, or sexual activity.  In the case of the Bartkowski et al (2007) study parents who may be prone to exaggerate the good behavior of their children, may be the same parents who are prone to exaggerate their church attendance.  Half of the Bartkowski results can be thus restated as “Parents who say that they attend church more often also say that their children are better behaved.”  Clearly, the teachers’ subjective responses may be less prone to that kind of bias, but teachers’ opinions of students can be skewed by factors like the socio-economic status of the family, and other unconscious biases.  Try to find out if measures being used are subjective or objective, and assess how accurate you think those measures are.

² Garrison Keillor’s description of the residents of Lake Wobegon, the fictional community in which his radio show, A Prairie Home Companion, takes place.

6. And as my results clearly show… – Typically a researcher will present arguments about why they found the results they found.  Their arguments are partially hypotheses they developed before conducting the research which they designed the research to test, and partially an explanation they crafted after looking at the results of the research.  Often some elements of their explanation will be tested in their research design, but others remain untested.  This is fine, but you should look carefully at the test, and see if you think there are other explanations that would also lead you to expect the same test results.  Perhaps the data that is being presented by the researcher is really good, but there is a better explanation for it.  Also, pay attention to what parts of the argument are being tested, and which parts are yet untested.

7. It’s not me, it’s you – Eschew Obfuscation means avoid making things difficult to understand.  Good research should follow that principle.  Cleary, many of the topics dealt with by scientific research are very complex and technical, and so someone unfamiliar with the field of study and its jargon may have trouble understanding research in it.  Sometimes though, research is difficult to understand because it poorly written or the concepts in it don’t quite make sense.  In 1996 physicist Alan Sokal got an article published in a cultural studies journal called Social Text that was total bunk.  He pulled this hoax on the journal to raise the issue of how scientific concepts were being misapplied in humanities disciplines.  The article was filled with erudite references, and complex jargon, and although it really didn’t say anything that made any sense, the editors of the journal were fooled.  Don’t assume that just because you don’t understand what is being said, that it must be brilliant beyond your comprehension.  Researchers should try to make things intelligible both in their graphs and in their text, because the goal is to communicate new knowledge to others. Don’t be afraid to ask for an explanation if something is unclear, and assess the quality of the explanation.  If you aren’t in a position to ask directly, look at what other experts in the field are saying about the article, and check for summaries of the article in textbooks or other materials aimed at a less specialized audience and see if those makes sense.   

8. Fair and Balanced? – Beware of agendas that the researcher might have.  Dale McGowan points out in his article that Bartkowski is clearly supportive of an Evangelical Christian worldview, which is evidenced by his other research.  While this certainly does not automatically render his conclusions invalid, it means that he comes from a certain perspective, and that his study may be influenced by that perspective.  In fact, it may be influenced by this perspective, even if he is doing his absolute best to conduct a rigorous and fair scientific test.  I believe that he probably is doing his best to conduct a fair test, but people’s beliefs and predispositions shape a lot of things in their research including what assumptions they may make, what topics they choose to study, what kinds of claims require evidence for them, and how much evidence it takes to convince them.  Scientific researchers are human beings, just like the rest of us, and as such they are not totally free of bias.  Although researchers do their best to put these biases aside when doing their work, they can creep in all the same.  Find out what agendas the researcher might bring to the table - this can be accomplished by doing a Google search on them, looking at a list of what they’ve published, finding out about the organization they work for, and finding out about the publication that published the research. Think about how these things might affect the study.   If you are reading media coverage of a study, consider agendas the media outlet may have, and how that may affect their coverage.  BUT don’t dismiss the study out of hand because of these things, because although there may be a spin there may be valid results and conclusions as well.

9. What’s your agenda? – Beware of the agendas that you might have.  Just as researchers can be biased by their agendas, so can readers.  We are much more likely to accept uncritically results that agree with our previously held beliefs, and criticize mercilessly results that disagree with our opinions.  I have to admit, that I probably wouldn’t have been so quick to do all the work of reading thoroughly and critiquing the Bartkowski et al (2007) study if its conclusion was something I agree with like “critical thinking is good for kids.”  We have to work hard to fight that tendency.  It is important to apply the same rigorous standards to research that supports things we agree with, not only to research that comes to conclusions that counter our beliefs.  Make an effort to critically evaluate research you read, regardless of whether you like the study’s conclusions.

10. Don’t Throw Out the Baby with the Bathwater³  – If you’ve read to this point, you might be reaching the conclusion that no research is ever valid, and that it all has all these problems, and that we can’t really know anything.  If you’re thinking that, you’re going too far.  The goal here is to read with a skeptical eye, not one that is so beset by doubts that no new information gets through.  There is a lot of good research being conducted.  The goal of these tactics is to help you assess the quality of research and be aware of some of the typical pitfalls.  If you find a problem in a study, don’t dismiss the whole thing out of hand.  Note the problem, and reduce your confidence in the results accordingly.  This isn’t a black and white sort of game.  You’ll find a wide range of quality of research, and your assessment of it should vary through a range as well.  If you find a study with a major methodological problem in it, see if anyone has learned from those mistakes and conducted similar research with a better research design.  It’s not all forward progress in science, but slowly through more research and more testing and more evidence science can correct itself, and winnow out bad ideas while keeping the good ones.  

 

 

This article originally appeared in the SSA eMpirical No. 20 - June 2007.