Types of Research Studies and How To Interpret Them

The field of nutrition is dynamic, and our understanding and practices are always evolving. Nutrition scientists are continuously conducting new research and publishing their findings in peer-reviewed journals. This adds to scientific knowledge, but it’s also of great interest to the public, so nutrition research often shows up in the news and other media sources. You might be interested in nutrition research to inform your own eating habits, or if you work in a health profession, so that you can give evidence-based advice to others. Making sense of science requires that you understand the types of research studies used and their limitations.

The Hierarchy of Nutrition Evidence

Researchers use many different types of study designs depending on the question they are trying to answer, as well as factors such as time, funding, and ethical considerations. The study design affects how we interpret the results and the strength of the evidence as it relates to real-life nutrition decisions. It can be helpful to think about the types of studies within a pyramid representing a hierarchy of evidence, where the studies at the bottom of the pyramid usually give us the weakest evidence with the least relevance to real-life nutrition decisions, and the studies at the top offer the strongest evidence, with the most relevance to real-life nutrition decisions.

The image shows a triangle, divided horizontally into 4 sections, from bottom to top, labeled as follows: non-human studies in red color; observational studies in blue color; intervention studies in green color, and meta-analyses and systematic reviews in yellow color. At left is an arrow pointing diagonally from bottom to top, labeled "LOW--Strength of evidence/Relevance to real-life nutrition decisions--HIGH."

Figure 2.3. The hierarchy of evidence shows types of research studies relative to their strength of evidence and relevance to real-life nutrition decisions, with the strongest studies at the top and the weakest at the bottom.

The pyramid also represents a few other general ideas. There tend to be more studies published using the methods at the bottom of the pyramid, because they require less time, money, and other resources. When researchers want to test a new hypothesis, they often start with the study designs at the bottom of the pyramid, such as in vitro, animal, or observational studies. Intervention studies are more expensive and resource-intensive, so there are fewer of these types of studies conducted. But they also give us higher quality evidence, so they’re an important next step if observational and non-human studies have shown promising results. Meta-analyses and systematic reviews combine the results of many studies already conducted, so they help researchers summarize scientific knowledge on a topic.

Non-Human Studies: In Vitro & Animal Studies

The simplest form of nutrition research is an in vitro study. In vitro means “within glass,” (although plastic is used more commonly today) and these experiments are conducted within flasks, dishes, plates, and test tubes. These studies are performed on isolated cells or tissue samples, so they’re less expensive and time-intensive than animal or human studies. In vitro studies are vital for zooming in on biological mechanisms, to see how things work at the cellular or molecular level. However, these studies shouldn’t be used to draw conclusions about how things work in humans (or even animals), because we can’t assume that the results will apply to a whole, living organism.

Two photos representing lab research. At left, a person appearing to be a woman with long dark hair and dark skin handles tiny tubes in a black bucket of ice. More tubes surround the bucket on the table. At right, a white mouse with red eyes peers out of an opening of a cage.

Animal studies are one form of in vivo research, which translates to “within the living.” Rats and mice are the most common animals used in nutrition research. Animals are often used in research that would be unethical to conduct in humans. Another advantage of animal dietary studies is that researchers can control exactly what the animals eat. In human studies, researchers can tell subjects what to eat and even provide them with the food, but they may not stick to the planned diet. People are also not very good at estimating, recording, or reporting what they eat and in what quantities. In addition, animal studies typically do not cost as much as human studies.

There are some important limitations of animal research. First, an animal’s metabolism and physiology are different from humans. Plus, animal models of disease (cancer, cardiovascular disease, etc.), although similar, are different from human diseases. Animal research is considered preliminary, and while it can be very important to the process of building scientific understanding and informing the types of studies that should be conducted in humans, animal studies shouldn’t be considered relevant to real-life decisions about how people eat.

Observational Studies

Observational studies in human nutrition collect information on people’s dietary patterns or nutrient intake and look for associations with health outcomes. Observational studies do not give participants a treatment or intervention; instead, they look at what they’re already doing and see how it relates to their health. These types of study designs can only identify correlations (relationships) between nutrition and health; they can’t show that one factor causes another. (For that, we need intervention studies, which we’ll discuss in a moment.) Observational studies that describe factors correlated with human health are also called epidemiological studies.1

One example of a nutrition hypothesis that has been investigated using observational studies is that eating a Mediterranean diet reduces the risk of developing cardiovascular disease. (A Mediterranean diet focuses on whole grains, fruits and vegetables, beans and other legumes, nuts, olive oil, herbs, and spices. It includes small amounts of animal protein (mostly fish), dairy, and red wine.2) There are three main types of observational studies, all of which could be used to test hypotheses about the Mediterranean diet:

  • Cohort studies follow a group of people (a cohort) over time, measuring factors such as diet and health outcomes. A cohort study of the Mediterranean diet would ask a group of people to describe their diet, and then researchers would track them over time to see if those eating a Mediterranean diet had a lower incidence of cardiovascular disease.
  • Case-control studies compare a group of cases and controls, looking for differences between the two groups that might explain their different health outcomes. For example, researchers might compare a group of people with cardiovascular disease with a group of healthy controls to see whether there were more controls or cases that followed a Mediterranean diet.
  • Cross-sectional studies collect information about a population of people at one point in time. For example, a cross-sectional study might compare the dietary patterns of people from different countries to see if diet correlates with the prevalence of cardiovascular disease in the different countries.

Prospective cohort studies, which enroll a cohort and follow them into the future, are usually considered the strongest type of observational study design. Retrospective studies look at what happened in the past, and they’re considered weaker because they rely on people’s memory of what they ate or how they felt in the past. There are several well-known examples of prospective cohort studies that have described important correlations between diet and disease:

  • Framingham Heart Study: Beginning in 1948, this study has followed the residents of Framingham, Massachusetts to identify risk factors for heart disease.
  • Health Professionals Follow-Up Study: This study started in 1986 and enrolled 51,529 male health professionals (dentists, pharmacists, optometrists, osteopathic physicians, podiatrists, and veterinarians), who complete diet questionnaires every 2 years.
  • Nurses Health Studies: Beginning in 1976, these studies have enrolled three large cohorts of nurses with a total of 280,000 participants. Participants have completed detailed questionnaires about diet, other lifestyle factors (smoking and exercise, for example), and health outcomes.

Observational studies have the advantage of allowing researchers to study large groups of people in the real world, looking at the frequency and pattern of health outcomes and identifying factors that correlate with them. But even very large observational studies may not apply to the population as a whole. For example, the Health Professionals Follow-Up Study and the Nurses Health Studies include people with above-average knowledge of health. In many ways, this makes them ideal study subjects, because they may be more motivated to be part of the study and to fill out detailed questionnaires for years. However, the findings of these studies may not apply to people with less baseline knowledge of health.

We’ve already mentioned another important limitation of observational studies—that they can only determine correlation, not causation. A prospective cohort study that finds that people eating a Mediterranean diet have a lower incidence of heart disease can only show that the Mediterranean diet is correlated with lowered risk of heart disease. It can’t show that the Mediterranean diet directly prevents heart disease. Why? There are a huge number of factors that determine health outcomes such as heart disease, and other factors might explain a correlation found in an observational study. For example, people who eat a Mediterranean diet might also be the same kind of people who exercise more, sleep more, have higher income (fish and nuts can be expensive!), or be less stressed. These are called confounding factors; they’re factors that can affect the outcome in question (i.e. heart disease) and also vary with the factor being studied (i.e. Mediterranean diet).

Intervention Studies

Intervention studies, also sometimes called experimental studies or clinical trials, include some type of treatment or change imposed by the researcher. Examples of interventions in nutrition research include asking participants to change their diet, take a supplement, or change the time of day that they eat. Unlike observational studies, intervention studies can provide evidence of cause and effect, so they are higher in the hierarchy of evidence pyramid.

The gold standard for intervention studies is the randomized controlled trial (RCT). In an RCT, study subjects are recruited to participate in the study. They are then randomly assigned into one of at least two groups, one of which is a control group (this is what makes the study controlled). In an RCT to study the effects of the Mediterranean diet on cardiovascular disease development, researchers might ask the control group to follow a low-fat diet (typically recommended for heart disease prevention) and the intervention group to eat a Mediterranean diet. The study would continue for a defined period of time (usually years to study an outcome like heart disease), at which point the researchers would analyze their data to see if more people in the control or Mediterranean diet had heart attacks or strokes. Because the treatment and control groups were randomly assigned, they should be alike in every other way except for diet, so differences in heart disease could be attributed to the diet. This eliminates the problem of confounding factors found in observational research, and it’s why RCTs can provide evidence of causation, not just correlation.

Imagine for a moment what would happen if the two groups weren’t randomly assigned. What if the researchers let study participants choose which diet they’d like to adopt for the study? They might, for whatever reason, end up with more overweight people who smoke and have high blood pressure in the low-fat diet group, and more people who exercised regularly and had already been eating lots of olive oil and nuts for years in the Mediterranean diet group. If they found that the Mediterranean diet group had fewer heart attacks by the end of the study, they would have no way of knowing if this was because of the diet or because of the underlying differences in the groups. In other words, without randomization, their results would be compromised by confounding factors, with many of the same limitations as observational studies.

In an RCT of a supplement, the control group would receive a placebo—a “fake” treatment that contains no active ingredients, such as a sugar pill. The use of a placebo is necessary in medical research because of a phenomenon known as the placebo effect. The placebo effect results in a beneficial effect because of a subject’s belief in the treatment, even though there is no treatment actually being administered.

For example, imagine an athlete who consumes a sports drink and then runs 100 meters in 11.0 seconds. On a different day, under the exact same conditions, the athlete is given a Super Duper Sports Drink and again runs 100 meters, this time in 10.5 seconds. But what the athlete didn’t know was that the Super Duper Sports Drink was the same as the regular sports drink—it just had a bit of food coloring added. There was nothing different between the drinks, but the athlete believed that the Super Duper Sports Drink was going to help him run faster, so he did. This improvement is due to the placebo effect. Ironically, a study similar to this example was published in 2015, demonstrating the power of the placebo effect on athletic performance.3

A cartoon depicts the study described in the text. At left is shown the "super duper sports drink" (sports drink plus food coloring) in orange. At right is the regular sports drink in green. A cartoon guy with yellow hair is pictured sprinting. The time with the super duper sports drink is 10.50 seconds, and the time with the regular sports drink is 11.00 seconds. The image reads "the improvement is the placebo effect."

Figure 2.4. An example of the placebo effect

Blinding is a technique to prevent bias in intervention studies. In a study without blinding, the subject and the researchers both know what treatment the subject is receiving. This can lead to bias if the subject or researcher have expectations about the treatment working, so these types of trials are used less frequently. It’s best if a study is double-blind, meaning that neither the researcher nor the subject know what treatment the subject is receiving. It’s relatively simple to double-blind a study where subjects are receiving a placebo or treatment pill, because they could be formulated to look and taste the same. In a single-blind study, either the researcher or the subject knows what treatment they’re receiving, but not both. Studies of diets—such as the Mediterranean diet example—often can’t be double-blinded because the study subjects know whether or not they’re eating a lot of olive oil and nuts. However, the researchers who are checking participants’ blood pressure or evaluating their medical records could be blinded to their treatment group, reducing the chance of bias.

Like all studies, RCTs and other intervention studies do have some limitations. They can be difficult to carry on for long periods of time and require that participants remain compliant with the intervention. They’re also costly and often have smaller sample sizes. Furthermore, it is unethical to study certain interventions. (An example of an unethical intervention would be to advise one group of pregnant mothers to drink alcohol to determine its effects on pregnancy outcomes, because we know that alcohol consumption during pregnancy damages the developing fetus.)

VIDEO: “Not all scientific studies are created equal” by David H. Schwartz, YouTube (April 28, 2014), 4:26.

Meta Analyses and Systematic Reviews

At the top of the hierarchy of evidence pyramid are systematic reviews and meta-analyses. You can think of these as “studies of studies.” They attempt to combine all of the relevant studies that have been conducted on a research question and summarize their overall conclusions. Researchers conducting a systematic review formulate a research question and then systematically and independently identify, select, evaluate, and synthesize all high-quality evidence that relates to the research question. Since systematic reviews combine the results of many studies, they help researchers produce more reliable findings. A meta-analysis is a type of systematic review that goes one step further, combining the data from multiple studies and using statistics to summarize it, as if creating a mega-study from many smaller studies.4

However, even systematic reviews and meta-analyses aren’t the final word on scientific questions. For one thing, they’re only as good as the studies that they include. The Cochrane Collaboration is an international consortium of researchers who conduct systematic reviews in order to inform evidence-based healthcare, including nutrition, and their reviews are among the most well-regarded and rigorous in science. For the most recent Cochrane review of the Mediterranean diet and cardiovascular disease, two authors independently reviewed studies published on this question. Based on their inclusion criteria, 30 RCTs with a total of 12,461 participants were included in the final analysis. However, after evaluating and combining the data, the authors concluded that “despite the large number of included trials, there is still uncertainty regarding the effects of a Mediterranean‐style diet on cardiovascular disease occurrence and risk factors in people both with and without cardiovascular disease already.” Part of the reason for this uncertainty is that different trials found different results, and the quality of the studies was low to moderate. Some had problems with their randomization procedures, for example, and others were judged to have unreliable data. That doesn’t make them useless, but it adds to the uncertainty about this question, and uncertainty pushes the field forward towards more and better studies. The Cochrane review authors noted that they found seven ongoing trials of the Mediterranean diet, so we can hope that they’ll add more clarity to this question in the future.5

Science is an ongoing and often slow process, and uncertainty and evolving questions are inherently a part of that process. The scientific method is our best approach for building our understanding of how the world and human life works. Many different types of studies can contribute to scientific knowledge. None are perfect—all have limitations—and a single study is never the final word on a scientific question. Part of what advances science is that researchers are constantly checking each other’s work, asking how it can be improved and what new questions it raises. And all of this moves us closer to consensus, usually built on a foundation of many different studies from every level of the hierarchy of evidence pyramid, and usually multiple meta-analyses of those studies.

One final note: If there’s a lot of debate on a scientific question, then that probably means that the science is still young and messy, and there’s not enough evidence for experts to settle on a consensus. In this scenario, be skeptical of anyone who claims to know the absolute truth or makes promises that seem too good to be true. We often see such claims in the field of nutrition, where book authors, podcasters, celebrity influencers, and companies selling products for a profit say they know the secrets of good health. They’re often operating in areas of scientific uncertainty, extrapolating from a few interesting findings, or outside of consensus entirely.

Self-Check:

Attributions:

References:

Images:

definition

License

Icon for the Creative Commons Attribution-NonCommercial 4.0 International License

Nutrition: Science and Everyday Application Copyright © 2020 by Alice Callahan, PhD; Heather Leonard, PhD, RDN; and Tamberly Powell, MS, RDN is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License, except where otherwise noted.

Share This Book