quasi experimental design control group

• Skip to secondary menu
• Skip to main content
• Skip to primary sidebar

Statistics By Jim

Making statistics intuitive

Quasi Experimental Design Overview & Examples

By Jim Frost Leave a Comment

What is a Quasi Experimental Design?

A quasi experimental design is a method for identifying causal relationships that does not randomly assign participants to the experimental groups. Instead, researchers use a non-random process. For example, they might use an eligibility cutoff score or preexisting groups to determine who receives the treatment.

Quasi-experimental research is a design that closely resembles experimental research but is different. The term “quasi” means “resembling,” so you can think of it as a cousin to actual experiments. In these studies, researchers can manipulate an independent variable — that is, they change one factor to see what effect it has. However, unlike true experimental research, participants are not randomly assigned to different groups.

Learn more about Experimental Designs: Definition & Types .

When to Use Quasi-Experimental Design

Researchers typically use a quasi-experimental design because they can’t randomize due to practical or ethical concerns. For example:

• Practical Constraints : A school interested in testing a new teaching method can only implement it in preexisting classes and cannot randomly assign students.
• Ethical Concerns : A medical study might not be able to randomly assign participants to a treatment group for an experimental medication when they are already taking a proven drug.

Quasi-experimental designs also come in handy when researchers want to study the effects of naturally occurring events, like policy changes or environmental shifts, where they can’t control who is exposed to the treatment.

Quasi-experimental designs occupy a unique position in the spectrum of research methodologies, sitting between observational studies and true experiments. This middle ground offers a blend of both worlds, addressing some limitations of purely observational studies while navigating the constraints often accompanying true experiments.

A significant advantage of quasi-experimental research over purely observational studies and correlational research is that it addresses the issue of directionality, determining which variable is the cause and which is the effect. In quasi-experiments, an intervention typically occurs during the investigation, and the researchers record outcomes before and after it, increasing the confidence that it causes the observed changes.

However, it’s crucial to recognize its limitations as well. Controlling confounding variables is a larger concern for a quasi-experimental design than a true experiment because it lacks random assignment.

In sum, quasi-experimental designs offer a valuable research approach when random assignment is not feasible, providing a more structured and controlled framework than observational studies while acknowledging and attempting to address potential confounders.

Types of Quasi-Experimental Designs and Examples

Quasi-experimental studies use various methods, depending on the scenario.

Natural Experiments

This design uses naturally occurring events or changes to create the treatment and control groups. Researchers compare outcomes between those whom the event affected and those it did not affect. Analysts use statistical controls to account for confounders that the researchers must also measure.

Natural experiments are related to observational studies, but they allow for a clearer causality inference because the external event or policy change provides both a form of quasi-random group assignment and a definite start date for the intervention.

For example, in a natural experiment utilizing a quasi-experimental design, researchers study the impact of a significant economic policy change on small business growth. The policy is implemented in one state but not in neighboring states. This scenario creates an unplanned experimental setup, where the state with the new policy serves as the treatment group, and the neighboring states act as the control group.

Researchers are primarily interested in small business growth rates but need to record various confounders that can impact growth rates. Hence, they record state economic indicators, investment levels, and employment figures. By recording these metrics across the states, they can include them in the model as covariates and control them statistically. This method allows researchers to estimate differences in small business growth due to the policy itself, separate from the various confounders.

Nonequivalent Groups Design

This method involves matching existing groups that are similar but not identical. Researchers attempt to find groups that are as equivalent as possible, particularly for factors likely to affect the outcome.

For instance, researchers use a nonequivalent groups quasi-experimental design to evaluate the effectiveness of a new teaching method in improving students’ mathematics performance. A school district considering the teaching method is planning the study. Students are already divided into schools, preventing random assignment.

The researchers matched two schools with similar demographics, baseline academic performance, and resources. The school using the traditional methodology is the control, while the other uses the new approach. Researchers are evaluating differences in educational outcomes between the two methods.

They perform a pretest to identify differences between the schools that might affect the outcome and include them as covariates to control for confounding. They also record outcomes before and after the intervention to have a larger context for the changes they observe.

Regression Discontinuity

This process assigns subjects to a treatment or control group based on a predetermined cutoff point (e.g., a test score). The analysis primarily focuses on participants near the cutoff point, as they are likely similar except for the treatment received. By comparing participants just above and below the cutoff, the design controls for confounders that vary smoothly around the cutoff.

For example, in a regression discontinuity quasi-experimental design focusing on a new medical treatment for depression, researchers use depression scores as the cutoff point. Individuals with depression scores just above a certain threshold are assigned to receive the latest treatment, while those just below the threshold do not receive it. This method creates two closely matched groups: one that barely qualifies for treatment and one that barely misses out.

By comparing the mental health outcomes of these two groups over time, researchers can assess the effectiveness of the new treatment. The assumption is that the only significant difference between the groups is whether they received the treatment, thereby isolating its impact on depression outcomes.

Controlling Confounders in a Quasi-Experimental Design

Accounting for confounding variables is a challenging but essential task for a quasi-experimental design.

In a true experiment, the random assignment process equalizes confounders across the groups to nullify their overall effect. It’s the gold standard because it works on all confounders, known and unknown.

Unfortunately, the lack of random assignment can allow differences between the groups to exist before the intervention. These confounding factors might ultimately explain the results rather than the intervention.

Consequently, researchers must use other methods to equalize the groups roughly using matching and cutoff values or statistically adjust for preexisting differences they measure to reduce the impact of confounders.

A key strength of quasi-experiments is their frequent use of “pre-post testing.” This approach involves conducting initial tests before collecting data to check for preexisting differences between groups that could impact the study’s outcome. By identifying these variables early on and including them as covariates, researchers can more effectively control potential confounders in their statistical analysis.

Additionally, researchers frequently track outcomes before and after the intervention to better understand the context for changes they observe.

Statisticians consider these methods to be less effective than randomization. Hence, quasi-experiments fall somewhere in the middle when it comes to internal validity , or how well the study can identify causal relationships versus mere correlation . They’re more conclusive than correlational studies but not as solid as true experiments.

In conclusion, quasi-experimental designs offer researchers a versatile and practical approach when random assignment is not feasible. This methodology bridges the gap between controlled experiments and observational studies, providing a valuable tool for investigating cause-and-effect relationships in real-world settings. Researchers can address ethical and logistical constraints by understanding and leveraging the different types of quasi-experimental designs while still obtaining insightful and meaningful results.

Cook, T. D., & Campbell, D. T. (1979).  Quasi-experimentation: Design & analysis issues in field settings . Boston, MA: Houghton Mifflin

Share this:

Reader Interactions

Comments and questions cancel reply.

• Privacy Policy

Home » Quasi-Experimental Research Design – Types, Methods

Quasi-Experimental Research Design – Types, Methods

Table of Contents

Quasi-Experimental Design

Quasi-experimental design is a research method that seeks to evaluate the causal relationships between variables, but without the full control over the independent variable(s) that is available in a true experimental design.

In a quasi-experimental design, the researcher uses an existing group of participants that is not randomly assigned to the experimental and control groups. Instead, the groups are selected based on pre-existing characteristics or conditions, such as age, gender, or the presence of a certain medical condition.

Types of Quasi-Experimental Design

There are several types of quasi-experimental designs that researchers use to study causal relationships between variables. Here are some of the most common types:

Non-Equivalent Control Group Design

This design involves selecting two groups of participants that are similar in every way except for the independent variable(s) that the researcher is testing. One group receives the treatment or intervention being studied, while the other group does not. The two groups are then compared to see if there are any significant differences in the outcomes.

Interrupted Time-Series Design

This design involves collecting data on the dependent variable(s) over a period of time, both before and after an intervention or event. The researcher can then determine whether there was a significant change in the dependent variable(s) following the intervention or event.

Pretest-Posttest Design

This design involves measuring the dependent variable(s) before and after an intervention or event, but without a control group. This design can be useful for determining whether the intervention or event had an effect, but it does not allow for control over other factors that may have influenced the outcomes.

Regression Discontinuity Design

This design involves selecting participants based on a specific cutoff point on a continuous variable, such as a test score. Participants on either side of the cutoff point are then compared to determine whether the intervention or event had an effect.

Natural Experiments

This design involves studying the effects of an intervention or event that occurs naturally, without the researcher’s intervention. For example, a researcher might study the effects of a new law or policy that affects certain groups of people. This design is useful when true experiments are not feasible or ethical.

Data Analysis Methods

Here are some data analysis methods that are commonly used in quasi-experimental designs:

Descriptive Statistics

This method involves summarizing the data collected during a study using measures such as mean, median, mode, range, and standard deviation. Descriptive statistics can help researchers identify trends or patterns in the data, and can also be useful for identifying outliers or anomalies.

Inferential Statistics

This method involves using statistical tests to determine whether the results of a study are statistically significant. Inferential statistics can help researchers make generalizations about a population based on the sample data collected during the study. Common statistical tests used in quasi-experimental designs include t-tests, ANOVA, and regression analysis.

Propensity Score Matching

This method is used to reduce bias in quasi-experimental designs by matching participants in the intervention group with participants in the control group who have similar characteristics. This can help to reduce the impact of confounding variables that may affect the study’s results.

Difference-in-differences Analysis

This method is used to compare the difference in outcomes between two groups over time. Researchers can use this method to determine whether a particular intervention has had an impact on the target population over time.

Interrupted Time Series Analysis

This method is used to examine the impact of an intervention or treatment over time by comparing data collected before and after the intervention or treatment. This method can help researchers determine whether an intervention had a significant impact on the target population.

Regression Discontinuity Analysis

This method is used to compare the outcomes of participants who fall on either side of a predetermined cutoff point. This method can help researchers determine whether an intervention had a significant impact on the target population.

Steps in Quasi-Experimental Design

Here are the general steps involved in conducting a quasi-experimental design:

• Identify the research question: Determine the research question and the variables that will be investigated.
• Choose the design: Choose the appropriate quasi-experimental design to address the research question. Examples include the pretest-posttest design, non-equivalent control group design, regression discontinuity design, and interrupted time series design.
• Select the participants: Select the participants who will be included in the study. Participants should be selected based on specific criteria relevant to the research question.
• Measure the variables: Measure the variables that are relevant to the research question. This may involve using surveys, questionnaires, tests, or other measures.
• Implement the intervention or treatment: Implement the intervention or treatment to the participants in the intervention group. This may involve training, education, counseling, or other interventions.
• Collect data: Collect data on the dependent variable(s) before and after the intervention. Data collection may also include collecting data on other variables that may impact the dependent variable(s).
• Analyze the data: Analyze the data collected to determine whether the intervention had a significant impact on the dependent variable(s).
• Draw conclusions: Draw conclusions about the relationship between the independent and dependent variables. If the results suggest a causal relationship, then appropriate recommendations may be made based on the findings.

Quasi-Experimental Design Examples

Here are some examples of real-time quasi-experimental designs:

• Evaluating the impact of a new teaching method: In this study, a group of students are taught using a new teaching method, while another group is taught using the traditional method. The test scores of both groups are compared before and after the intervention to determine whether the new teaching method had a significant impact on student performance.
• Assessing the effectiveness of a public health campaign: In this study, a public health campaign is launched to promote healthy eating habits among a targeted population. The behavior of the population is compared before and after the campaign to determine whether the intervention had a significant impact on the target behavior.
• Examining the impact of a new medication: In this study, a group of patients is given a new medication, while another group is given a placebo. The outcomes of both groups are compared to determine whether the new medication had a significant impact on the targeted health condition.
• Evaluating the effectiveness of a job training program : In this study, a group of unemployed individuals is enrolled in a job training program, while another group is not enrolled in any program. The employment rates of both groups are compared before and after the intervention to determine whether the training program had a significant impact on the employment rates of the participants.
• Assessing the impact of a new policy : In this study, a new policy is implemented in a particular area, while another area does not have the new policy. The outcomes of both areas are compared before and after the intervention to determine whether the new policy had a significant impact on the targeted behavior or outcome.

Applications of Quasi-Experimental Design

Here are some applications of quasi-experimental design:

• Educational research: Quasi-experimental designs are used to evaluate the effectiveness of educational interventions, such as new teaching methods, technology-based learning, or educational policies.
• Health research: Quasi-experimental designs are used to evaluate the effectiveness of health interventions, such as new medications, public health campaigns, or health policies.
• Social science research: Quasi-experimental designs are used to investigate the impact of social interventions, such as job training programs, welfare policies, or criminal justice programs.
• Business research: Quasi-experimental designs are used to evaluate the impact of business interventions, such as marketing campaigns, new products, or pricing strategies.
• Environmental research: Quasi-experimental designs are used to evaluate the impact of environmental interventions, such as conservation programs, pollution control policies, or renewable energy initiatives.

When to use Quasi-Experimental Design

Here are some situations where quasi-experimental designs may be appropriate:

• When the research question involves investigating the effectiveness of an intervention, policy, or program : In situations where it is not feasible or ethical to randomly assign participants to intervention and control groups, quasi-experimental designs can be used to evaluate the impact of the intervention on the targeted outcome.
• When the sample size is small: In situations where the sample size is small, it may be difficult to randomly assign participants to intervention and control groups. Quasi-experimental designs can be used to investigate the impact of an intervention without requiring a large sample size.
• When the research question involves investigating a naturally occurring event : In some situations, researchers may be interested in investigating the impact of a naturally occurring event, such as a natural disaster or a major policy change. Quasi-experimental designs can be used to evaluate the impact of the event on the targeted outcome.
• When the research question involves investigating a long-term intervention: In situations where the intervention or program is long-term, it may be difficult to randomly assign participants to intervention and control groups for the entire duration of the intervention. Quasi-experimental designs can be used to evaluate the impact of the intervention over time.
• When the research question involves investigating the impact of a variable that cannot be manipulated : In some situations, it may not be possible or ethical to manipulate a variable of interest. Quasi-experimental designs can be used to investigate the relationship between the variable and the targeted outcome.

Purpose of Quasi-Experimental Design

The purpose of quasi-experimental design is to investigate the causal relationship between two or more variables when it is not feasible or ethical to conduct a randomized controlled trial (RCT). Quasi-experimental designs attempt to emulate the randomized control trial by mimicking the control group and the intervention group as much as possible.

The key purpose of quasi-experimental design is to evaluate the impact of an intervention, policy, or program on a targeted outcome while controlling for potential confounding factors that may affect the outcome. Quasi-experimental designs aim to answer questions such as: Did the intervention cause the change in the outcome? Would the outcome have changed without the intervention? And was the intervention effective in achieving its intended goals?

Quasi-experimental designs are useful in situations where randomized controlled trials are not feasible or ethical. They provide researchers with an alternative method to evaluate the effectiveness of interventions, policies, and programs in real-life settings. Quasi-experimental designs can also help inform policy and practice by providing valuable insights into the causal relationships between variables.

Overall, the purpose of quasi-experimental design is to provide a rigorous method for evaluating the impact of interventions, policies, and programs while controlling for potential confounding factors that may affect the outcome.

Advantages of Quasi-Experimental Design

Quasi-experimental designs have several advantages over other research designs, such as:

• Greater external validity : Quasi-experimental designs are more likely to have greater external validity than laboratory experiments because they are conducted in naturalistic settings. This means that the results are more likely to generalize to real-world situations.
• Ethical considerations: Quasi-experimental designs often involve naturally occurring events, such as natural disasters or policy changes. This means that researchers do not need to manipulate variables, which can raise ethical concerns.
• More practical: Quasi-experimental designs are often more practical than experimental designs because they are less expensive and easier to conduct. They can also be used to evaluate programs or policies that have already been implemented, which can save time and resources.
• No random assignment: Quasi-experimental designs do not require random assignment, which can be difficult or impossible in some cases, such as when studying the effects of a natural disaster. This means that researchers can still make causal inferences, although they must use statistical techniques to control for potential confounding variables.
• Greater generalizability : Quasi-experimental designs are often more generalizable than experimental designs because they include a wider range of participants and conditions. This can make the results more applicable to different populations and settings.

Limitations of Quasi-Experimental Design

There are several limitations associated with quasi-experimental designs, which include:

• Lack of Randomization: Quasi-experimental designs do not involve randomization of participants into groups, which means that the groups being studied may differ in important ways that could affect the outcome of the study. This can lead to problems with internal validity and limit the ability to make causal inferences.
• Selection Bias: Quasi-experimental designs may suffer from selection bias because participants are not randomly assigned to groups. Participants may self-select into groups or be assigned based on pre-existing characteristics, which may introduce bias into the study.
• History and Maturation: Quasi-experimental designs are susceptible to history and maturation effects, where the passage of time or other events may influence the outcome of the study.
• Lack of Control: Quasi-experimental designs may lack control over extraneous variables that could influence the outcome of the study. This can limit the ability to draw causal inferences from the study.
• Limited Generalizability: Quasi-experimental designs may have limited generalizability because the results may only apply to the specific population and context being studied.

About the author

Muhammad Hassan

Researcher, Academic Writer, Web developer

You may also like

Triangulation in Research – Types, Methods and...

Survey Research – Types, Methods, Examples

Mixed Methods Research – Types & Analysis

Focus Groups – Steps, Examples and Guide

Experimental Design – Types, Methods, Guide

Basic Research – Types, Methods and Examples

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• HHS Author Manuscripts

Experimental and Quasi-Experimental Designs in Implementation Research

Christopher j. miller.

a VA Boston Healthcare System, Center for Healthcare Organization and Implementation Research (CHOIR), United States Department of Veterans Affairs, Boston, MA, USA

b Department of Psychiatry, Harvard Medical School, Boston, MA, USA

Shawna N. Smith

c Department of Psychiatry, University of Michigan Medical School, Ann Arbor, MI, USA

d Survey Research Center, Institute for Social Research, University of Michigan, Ann Arbor, MI, USA

Marianne Pugatch

Implementation science is focused on maximizing the adoption, appropriate use, and sustainability of effective clinical practices in real world clinical settings. Many implementation science questions can be feasibly answered by fully experimental designs, typically in the form of randomized controlled trials (RCTs). Implementation-focused RCTs, however, usually differ from traditional efficacy- or effectiveness-oriented RCTs on key parameters. Other implementation science questions are more suited to quasi-experimental designs, which are intended to estimate the effect of an intervention in the absence of randomization. These designs include pre-post designs with a non-equivalent control group, interrupted time series (ITS), and stepped wedges, the last of which require all participants to receive the intervention, but in a staggered fashion. In this article we review the use of experimental designs in implementation science, including recent methodological advances for implementation studies. We also review the use of quasi-experimental designs in implementation science, and discuss the strengths and weaknesses of these approaches. This article is therefore meant to be a practical guide for researchers who are interested in selecting the most appropriate study design to answer relevant implementation science questions, and thereby increase the rate at which effective clinical practices are adopted, spread, and sustained.

1. Background

The first documented clinical trial was conducted in 1747 by James Lind, a royal navy physician, who tested the hypothesis that citrus fruit could cure scurvy. Since then, based on foundational work by Fisher and others (1935), the randomized controlled trial (RCT) has emerged as the gold standard for testing the efficacy of treatment versus a control condition for individual patients. Randomization of patients is seen as a crucial to reducing the impact of measured or unmeasured confounding variables, in turn allowing researchers to draw conclusions regarding causality in clinical trials.

As described elsewhere in this special issue, implementation science is ultimately focused on maximizing the adoption, appropriate use, and sustainability of effective clinical practices in real world clinical settings. As such, some implementation science questions may be addressed by experimental designs. For our purposes here, we use the term “experimental” to refer to designs that feature two essential ingredients: first, manipulation of an independent variable; and second, random assignment of subjects. This corresponds to the definition of randomized experiments originally championed by Fisher (1925) . From this perspective, experimental designs usually take the form of RCTs—but implementation- oriented RCTs typically differ in important ways from traditional efficacy- or effectiveness-oriented RCTs. Other implementation science questions require different methodologies entirely: specifically, several forms of quasi-experimental designs may be used for implementation research in situations where an RCT would be inappropriate. These designs are intended to estimate the effect of an intervention despite a lack of randomization. Quasi-experimental designs include pre-post designs with a nonequivalent control group, interrupted time series (ITS), and stepped wedge designs. Stepped wedges are studies in which all participants receive the intervention, but in a staggered fashion. It is important to note that quasi-experimental designs are not unique to implementation science. As we will discuss below, however, each of them has strengths that make them particularly useful in certain implementation science contexts.

Our goal for this manuscript is two-fold. First, we will summarize the use of experimental designs in implementation science. This will include discussion of ways that implementation-focused RCTs may differ from efficacy- or effectiveness-oriented RCTs. Second, we will summarize the use of quasi-experimental designs in implementation research. This will include discussion of the strengths and weaknesses of these types of approaches in answering implementation research questions. For both experimental and quasi-experimental designs, we will discuss a recent implementation study as an illustrative example of one approach.

1. Experimental Designs in Implementation Science

RCTs in implementation science share the same basic structure as efficacy- or effectiveness-oriented RCTs, but typically feature important distinctions. In this section we will start by reviewing key factors that separate implementation RCTs from more traditional efficacy- or effectiveness-oriented RCTs. We will then discuss optimization trials, which are a type of experimental design that is especially useful for certain implementation science questions. We will then briefly turn our attention to single subject experimental designs (SSEDs) and on-off-on (ABA) designs.

The first common difference that sets apart implementation RCTs from more traditional clinical trials is the primary research question they aim to address. For most implementation trials, the primary research question is not the extent to which a particular treatment or evidence-based practice is more effective than a comparison condition, but instead the extent to which a given implementation strategy is more effective than a comparison condition. For more detail on this pivotal issue, see Drs. Bauer and Kirchner in this special issue.

Second, as a corollary of this point, implementation RCTs typically feature different outcome measures than efficacy or effectiveness RCTs, with an emphasis on the extent to which a health intervention was successfully implemented rather than an evaluation of the health effects of that intervention ( Proctor et al., 2011 ). For example, typical implementation outcomes might include the number of patients who receive the intervention, or the number of providers who administer the intervention as intended. A variety of evaluation-oriented implementation frameworks may guide the choices of such measures (e.g. RE-AIM; Gaglio et al., 2013 ; Glasgow et al., 1999 ). Hybrid implementation-effectiveness studies attend to both effectiveness and implementation outcomes ( Curran et al., 2012 ); these designs are also covered in more detail elsewhere in this issue (Landes, this issue).

Third, given their focus, implementation RCTs are frequently cluster-randomized (i.e. with sites or clinics as the unit of randomization, and patients nested within those sites or clinics). For example, consider a hypothetical RCT that aims to evaluate the implementation of a training program for cognitive behavioral therapy (CBT) in community clinics. Randomizing at the patient level for such a trial would be inappropriate due to the risk of contamination, as providers trained in CBT might reasonably be expected to incorporate CBT principles into their treatment even to patients assigned to the control condition. Randomizing at the provider level would also risk contamination, as providers trained in CBT might discuss this treatment approach with their colleagues. Thus, many implementation trials are cluster randomized at the site or clinic level. While such clustering minimizes the risk of contamination, it can unfortunately create commensurate problems with confounding, especially for trials with very few sites to randomize. Stratification may be used to at least partially address confounding issues in cluster- randomized and more traditional trials alike, by ensuring that intervention and control groups are broadly similar on certain key variables. Furthermore, such allocation schemes typically require analytic models that account for this clustering and the resulting correlations among error structures (e.g., generalized estimating equations [GEE] or mixed-effects models; Schildcrout et al., 2018 ).

1.1. Optimization trials

Key research questions in implementation science often involve determining which implementation strategies to provide, to whom, and when, to achieve optimal implementation success. As such, trials designed to evaluate comparative effectiveness, or to optimize provision of different types or intensities of implementation strategies, may be more appealing than traditional effectiveness trials. The methods described in this section are not unique to implementation science, but their application in the context of implementation trials may be particularly useful for informing implementation strategies.

While two-arm RCTs can be used to evaluate comparative effectiveness, trials focused on optimizing implementation support may use alternative experimental designs ( Collins et al., 2005 ; Collins et al., 2007 ). For example, in certain clinical contexts, multi-component “bundles” of implementation strategies may be warranted (e.g. a bundle consisting of clinician training, technical assistance, and audit/feedback to encourage clinicians to use a new evidence-based practice). In these situations, implementation researchers might consider using factorial or fractional-factorial designs. In the context of implementation science, these designs randomize participants (e.g. sites or providers) to different combinations of implementation strategies, and can be used to evaluate the effectiveness of each strategy individually to inform an optimal combination (e.g. Coulton et al., 2009 ; Pellegrini et al., 2014 ; Wyrick, et al., 2014 ). Such designs can be particularly useful in informing multi-component implementation strategies that are not redundant or overly burdensome ( Collins et al., 2014a ; Collins et al., 2009 ; Collins et al., 2007 ).

Researchers interested in optimizing sequences of implementation strategies that adapt to ongoing needs over time may be interested in a variant of factorial designs known as the sequential, multiple-assignment randomized trial (SMART; Almirall et al., 2012 ; Collins et al., 2014b ; Kilbourne et al., 2014b ; Lei et al., 2012 ; Nahum-Shani et al., 2012 ; NeCamp et al., 2017 ). SMARTs are multistage randomized trials in which some or all participants are randomized more than once, often based on ongoing information (e.g., treatment response). In implementation research, SMARTs can inform optimal sequences of implementation strategies to maximize downstream clinical outcomes. Thus, such designs are well-suited to answering questions about what implementation strategies should be used, in what order, to achieve the best outcomes in a given context.

One example of an implementation SMART is the Adaptive Implementation of Effective Program Trial (ADEPT; Kilbourne et al., 2014a ). ADEPT was a clustered SMART ( NeCamp et al., 2017 ) designed to inform an adaptive sequence of implementation strategies for implementing an evidence-based collaborative chronic care model, Life Goals ( Kilbourne et al., 2014c ; Kilbourne et al., 2012a ), into community-based practices. Life Goals, the clinical intervention being implemented, has proven effective at improving physical and mental health outcomes for patients with unipolar and bipolar depression by encouraging providers to instruct patients in self-management, and improving clinical information systems and care management across physical and mental health providers ( Bauer et al., 2006 ; Kilbourne et al., 2012a ; Kilbourne et al., 2008 ; Simon et al., 2006 ). However, in spite of its established clinical effectiveness, community-based clinics experienced a number of barriers in trying to implement the Life Goals model, and there were questions about how best to efficiently and effectively augment implementation strategies for clinics that struggled with implementation.

The ADEPT study was thus designed to determine the best sequence of implementation strategies to offer sites interested in implementing Life Goals. The ADEPT study involved use of three different implementation strategies. First, all sites received implementation support based on Replicating Effective Programs (REP), which offered an implementation manual, brief training, and low- level technical support ( Kilbourne et al., 2007 ; Kilbourne et al., 2012b ; Neumann and Sogolow, 2000 ). REP implementation support had been previously found to be low-cost and readily scalable, but also insufficient for uptake for many community-based settings ( Kilbourne et al., 2015 ). For sites that failed to implement Life Goals under REP, two additional implementation strategies were considered as augmentations to REP: External Facilitation (EF; Kilbourne et al., 2014b ; Stetler et al., 2006 ), consisting of phone-based mentoring in strategic skills from a study team member; and Internal Facilitation (IF; Kirchner et al., 2014 ), which supported protected time for a site employee to address barriers to program adoption.

The ADEPT study was designed to evaluate the best way to augment support for these sites that were not able to implement Life Goals under REP, specifically querying whether it was better to augment REP with EF only or the more intensive EF/IF, and whether augmentations should be provided all at once, or staged. Intervention assignments are mapped in Figure 1 . Seventy-nine community-based clinics across Michigan and Colorado were provided with initial implementation support under REP. After six months, implementation of the clinical intervention, Life Goals, was evaluated at all sites. Sites that had failed to reach an adequate level of delivery (defined as those sites enrolling fewer than ten patients in Life Goals, or those at which fewer than 50% of enrolled patients had received at least three Life Goals sessions) were considered non-responsive to REP and randomized to receive additional support through either EF or combined EF/IF. After six further months, Life Goals implementation at these sites was again evaluated. Sites surpassing the implementation response benchmark had their EF or EF/IF support discontinued. EF/IF sites that remained non-responsive continued to receive EF/IF for an additional six months. EF sites that remained non-responsive were randomized a second time to either continue with EF or further augment with IF. This design thus allowed for comparison of three different adaptive implementation interventions for sites that were initially non-responsive to REP to determine the best adaptive sequence of implementation support for sites that were initially non-responsive under REP:

SMART design from ADEPT trial.

• Provide EF for 6 months; continue EF for a further six months for sites that remain nonresponsive; discontinue EF for sites that are responsive;
• Provide EF/IF for 6 months; continue EF/IF for a further six months for sites that remain non-responsive; discontinue EF/IF for sites that are responsive; and
• Provide EF for 6 months; step up to EF/IF for a further six months for sites that remain non-responsive; discontinue EF for sites that are responsive.

While analyses of this study are still ongoing, including the comparison of these three adaptive sequences of implementation strategies, results have shown that patients at sites that were randomized to receive EF as the initial augmentation to REP saw more improvement in clinical outcomes (SF-12 mental health quality of life and PHQ-9 depression scores) after 12 months than patients at sites that were randomized to receive the more intensive EF/IF augmentation.

1.2. Single Subject Experimental Designs and On-Off-On (ABA) Designs

We also note that there are a variety of Single Subject Experimental Designs (SSEDs; Byiers et al., 2012 ), including withdrawal designs and alternating treatment designs, that can be used in testing evidence-based practices. Similarly, an implementation strategy may be used to encourage the use of a specific treatment at a particular site, followed by that strategy’s withdrawal and subsequent reinstatement, with data collection throughout the process (on-off-on or ABA design). A weakness of these approaches in the context of implementation science, however, is that they usually require reversibility of the intervention (i.e. that the withdrawal of implementation support truly allows the healthcare system to revert to its pre-implementation state). When this is not the case—for example, if a hypothetical study is focused on training to encourage use of an evidence-based psychotherapy—then these designs may be less useful.

2. Quasi-Experimental Designs in Implementation Science

In some implementation science contexts, policy-makers or administrators may not be willing to have a subset of participating patients or sites randomized to a control condition, especially for high-profile or high-urgency clinical issues. Quasi-experimental designs allow implementation scientists to conduct rigorous studies in these contexts, albeit with certain limitations. We briefly review the characteristics of these designs here; other recent review articles are available for the interested reader (e.g. Handley et al., 2018 ).

2.1. Pre-Post with Non-Equivalent Control Group

The pre-post with non-equivalent control group uses a control group in the absence of randomization. Ideally, the control group is chosen to be as similar to the intervention group as possible (e.g. by matching on factors such as clinic type, patient population, geographic region, etc.). Theoretically, both groups are exposed to the same trends in the environment, making it plausible to decipher if the intervention had an effect. Measurement of both treatment and control conditions classically occurs pre- and post-intervention, with differential improvement between the groups attributed to the intervention. This design is popular due to its practicality, especially if data collection points can be kept to a minimum. It may be especially useful for capitalizing on naturally occurring experiments such as may occur in the context of certain policy initiatives or rollouts—specifically, rollouts in which it is plausible that a control group can be identified. For example, Kirchner and colleagues (2014) used this type of design to evaluate the integration of mental health services into primary care clinics at seven US Department of Veterans Affairs (VA) medical centers and seven matched controls.

One overarching drawback of this design is that it is especially vulnerable to threats to internal validity ( Shadish, 2002 ), because pre-existing differences between the treatment and control group could erroneously be attributed to the intervention. While unmeasured differences between treatment and control groups are always a possibility in healthcare research, such differences are especially likely to occur in the context of these designs due to the lack of randomization. Similarly, this design is particularly sensitive to secular trends that may differentially affect the treatment and control groups ( Cousins et al., 2014 ; Pape et al., 2013 ), as well as regression to the mean confounding study results ( Morton and Torgerson, 2003 ). For example, if a study site is selected for the experimental condition precisely because it is underperforming in some way, then regression to the mean would suggest that the site will show improvement regardless of any intervention; in the context of a pre-post with non-equivalent control group study, however, this improvement would erroneously be attributed to the intervention itself (Type I error).

There are, however, various ways that implementation scientists can mitigate these weaknesses. First, as mentioned briefly above, it is important to select a control group that is as similar as possible to the intervention site(s), which can include matching at both the health care network and clinic level (e.g. Kirchner et al., 2014 ). Second, propensity score weighting (e.g. Morgan, 2018 ) can statistically mitigate internal validity concerns, although this approach may be of limited utility when comparing secular trends between different study cohorts ( Dimick and Ryan, 2014 ). More broadly, qualitative methods (e.g. periodic interviews with staff at intervention and control sites) can help uncover key contextual factors that may be affecting study results above and beyond the intervention itself.

2.2. Interrupted Time Series

Interrupted time series (ITS; Shadish, 2002 ; Taljaard et al., 2014 ; Wagner et al., 2002 ) designs represent one of the most robust categories of quasi-experimental designs. Rather than relying on a non-equivalent control group, ITS designs rely on repeated data collections from intervention sites to determine whether a particular intervention is associated with improvement on a given metric relative to the pre-intervention secular trend. They are particularly useful in cases where a comparable control group cannot be identified—for example, following widespread implementation of policy mandates, quality improvement initiatives, or dissemination campaigns ( Eccles et al., 2003 ). In ITS designs, data are collected at multiple time points both before and after an intervention (e.g., policy change, implementation effort), and analyses explore whether the intervention was associated with the outcome beyond any pre-existing secular trend. More formally, ITS evaluations focus on identifying whether there is discontinuity in the trend (change in slope or level) after the intervention relative to before the intervention, using segmented regression to model pre- and post-intervention trends ( Gebski et al., 2012 ; Penfold and Zhang, 2013 ; Taljaard et al., 2014 ; Wagner et al., 2002 ). A number of recent implementation studies have used ITS designs, including an evaluation of implementation of a comprehensive smoke-free policy in a large UK mental health organization to reduce physical assaults ( Robson et al., 2017 ); the impact of a national policy limiting alcohol availability on suicide mortality in Slovenia ( Pridemore and Snowden, 2009 ); and the effect of delivery of a tailored intervention for primary care providers to increase psychological referrals for women with mild to moderate postnatal depression ( Hanbury et al., 2013 ).

ITS designs are appealing in implementation work for several reasons. Relative to uncontrolled pre-post analyses, ITS analyses reduce the chances that intervention effects are confounded by secular trends ( Bernal et al., 2017 ; Eccles et al., 2003 ). Time-varying confounders, such as seasonality, can also be adjusted for, provided adequate data ( Bernal et al., 2017 ). Indeed, recent work has confirmed that ITS designs can yield effect estimates similar to those derived from cluster-randomized RCTs ( Fretheim et al., 2013 ; Fretheim et al., 2015 ). Relative to an RCT, ITS designs can also allow for a more comprehensive assessment of the longitudinal effects of an intervention (positive or negative), as effects can be traced over all included time points ( Bernal et al., 2017 ; Penfold and Zhang, 2013 ).

ITS designs also present a number of challenges. First, the segmented regression approach requires clear delineation between pre- and post-intervention periods; interventions with indeterminate implementation periods are likely not good candidates for ITS. While ITS designs that include multiple ‘interruptions’ (e.g. introductions of new treatment components) are possible, they will require collection of enough time points between interruptions to ensure that each intervention’s effects can be ascertained individually ( Bernal et al., 2017 ). Second, collecting data from sufficient time points across all sites of interest, especially for the pre-intervention period, can be challenging ( Eccles et al., 2003 ): a common recommendation is at least eight time points both pre- and post-intervention ( Penfold and Zhang, 2013 ). This may be onerous, particularly if the data are not routinely collected by the health system(s) under study. Third, ITS cannot protect against confounding effects from other interventions that begin contemporaneously and may impact similar outcomes ( Eccles et al., 2003 ).

2.3. Stepped Wedge Designs

Stepped wedge trials are another type of quasi-experimental design. In a stepped wedge, all participants receive the intervention, but are assigned to the timing of the intervention in a staggered fashion ( Betran et al., 2018 ; Brown and Lilford, 2006 ; Hussey and Hughes, 2007 ), typically at the site or cluster level. Stepped wedge designs have their analytic roots in balanced incomplete block designs, in which all pairs of treatments occur an equal number of times within each block ( Hanani, 1961 ). Traditionally, all sites in stepped wedge trials have outcome measures assessed at all time points, thus allowing sites that receive the intervention later in the trial to essentially serve as controls for early intervention sites. A recent special issue of the journal Trials includes more detail on these designs ( Davey et al., 2015 ), which may be ideal for situations in which it is important for all participating patients or sites to receive the intervention during the trial. Stepped wedge trials may also be useful when resources are scarce enough that intervening at all sites at once (or even half of the sites as in a standard treatment-versus-control RCT) would not be feasible. If desired, the administration of the intervention to sites in waves allows for lessons learned in early sites to be applied to later sites (via formative evaluation; see Elwy et al., this issue).

The Behavioral Health Interdisciplinary Program (BHIP) Enhancement Project is a recent example of a stepped-wedge implementation trial ( Bauer et al., 2016 ; Bauer et al., 2019 ). This study involved using blended facilitation (including internal and external facilitators; Kirchner et al., 2014 ) to implement care practices consistent with the collaborative chronic care model (CCM; Bodenheimer et al., 2002a , b ; Wagner et al., 1996 ) in nine outpatient mental health teams in VA medical centers. Figure 2 illustrates the implementation and stepdown periods for that trial, with black dots representing primary data collection points.

BHIP Enhancement Project stepped wedge (adapted form Bauer et al., 2019).

The BHIP Enhancement Project was conducted as a stepped wedge for several reasons. First, the stepped wedge design allowed the trial to reach nine sites despite limited implementation resources (i.e. intervening at all nine sites simultaneously would not have been feasible given study funding). Second, the stepped wedge design aided in recruitment and retention, as all participating sites were certain to receive implementation support during the trial: at worst, sites that were randomized to later- phase implementation had to endure waiting periods totaling about eight months before implementation began. This was seen as a major strength of the design by its operational partner, the VA Office of Mental Health and Suicide Prevention. To keep sites engaged during the waiting period, the BHIP Enhancement Project offered a guiding workbook and monthly technical support conference calls.

Three additional features of the BHIP Enhancement Project deserve special attention. First, data collection for late-implementing sites did not begin until immediately before the onset of implementation support (see Figure 2 ). While this reduced statistical power, it also significantly reduced data collection burden on the study team. Second, onset of implementation support was staggered such that wave 2 began at the end of month 4 rather than month 6. This had two benefits: first, this compressed the overall amount of time required for implementation during the trial. Second, it meant that the study team only had to collect data from one site at a time, with data collection periods coming every 2–4 months. More traditional stepped wedge approaches typically have data collection across sites temporally aligned (e.g. Betran et al., 2018 ). Third, the BHIP Enhancement Project used a balancing algorithm ( Lew et al., 2019 ) to assign sites to waves, retaining some of the benefits of randomization while ensuring balance on key site characteristics (e.g. size, geographic region).

Despite their utility, stepped wedges have some important limitations. First, because they feature delayed implementation at some sites, stepped wedges typically take longer than similarly-sized parallel group RCTs. This increases the chances that secular trends, policy changes, or other external forces impact study results. Second, as with RCTs, imbalanced site assignment can confound results. This may occur deliberately in some cases—for example, if sites that develop their implementation plans first are assigned to earlier waves. Even if sites are randomized, however, early and late wave sites may still differ on important characteristics such as size, rurality, and case mix. The resulting confounding between site assignment and time can threaten the internal validity of the study—although, as above, balancing algorithms can reduce this risk. Third, the use of formative evaluation (Elwy, this issue), while useful for maximizing the utility of implementation efforts in a stepped wedge, can mean that late-wave sites receive different implementation strategies than early-wave sites. Similarly, formative evaluation may inform midstream adaptations to the clinical innovation being implemented. In either case, these changes may again threaten internal validity. Overall, then, stepped wedges represent useful tools for evaluating the impact of health interventions that (as with all designs) are subject to certain weaknesses and limitations.

3. Conclusions and Future Directions

Implementation science is focused on maximizing the extent to which effective healthcare practices are adopted, used, and sustained by clinicians, hospitals, and systems. Answering questions in these domains frequently requires different research methods than those employed in traditional efficacy- or effectiveness-oriented randomized clinical trials (RCTs). Implementation-oriented RCTs typically feature cluster or site-level randomization, and emphasize implementation outcomes (e.g. the number of patients receiving the new treatment as intended) rather than traditional clinical outcomes. Hybrid implementation-effectiveness designs incorporate both types of outcomes; more details on these approaches can be found elsewhere in this special issue (Landes, this issue). Other methodological innovations, such as factorial designs or sequential, multiple-assignment randomized trials (SMARTs), can address questions about multi-component or adaptive interventions, still under the umbrella of experimental designs. These types of trials may be especially important for demystifying the “black box” of implementation—that is, determining what components of an implementation strategy are most strongly associated with implementation success. In contrast, pre-post designs with non-equivalent control groups, interrupted time series (ITS), and stepped wedge designs are all examples of quasiexperimental designs that may serve implementation researchers when experimental designs would be inappropriate. A major theme cutting across each of these designs is that there are relative strengths and weaknesses associated with any study design decision. Determining what design to use ultimately will need to be informed by the primary research question to be answered, while simultaneously balancing the need for internal validity, external validity, feasibility, and ethics.

New innovations in study design are constantly being developed and refined. Several such innovations are covered in other articles within this special issue (e.g. Kim et al., this issue). One future direction relevant to the study designs presented in this article is the potential for adaptive trial designs, which allow information gleaned during the trial to inform the adaptation of components like treatment allocation, sample size, or study recruitment in the later phases of the same trial ( Pallmann et al., 2018 ). These designs are becoming increasingly popular in clinical treatment ( Bhatt and Mehta, 2016 ) but could also hold promise for implementation scientists, especially as interest grows in rapid-cycle testing of implementation strategies or efforts. Adaptive designs could potentially be incorporated into both SMART designs and stepped wedge studies, as well as traditional RCTs to further advance implementation science ( Cheung et al., 2015 ). Ideally, these and other innovations will provide researchers with increasingly robust and useful methodologies for answering timely implementation science questions.

• Many implementation science questions can be addressed by fully experimental designs (e.g. randomized controlled trials [RCTs]).
• Implementation trials differ in important ways, however, from more traditional efficacy- or effectiveness-oriented RCTs.
• Adaptive designs represent a recent innovation to determine optimal implementation strategies within a fully experimental framework.
• Quasi-experimental designs can be used to answer implementation science questions in the absence of randomization.
• The choice of study designs in implementation science requires careful consideration of scientific, pragmatic, and ethical issues.

Acknowledgments

This work was supported by Department of Veterans Affairs grants QUE 15–289 (PI: Bauer) and CIN 13403 and National Institutes of Health grant RO1 MH 099898 (PI: Kilbourne).

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

• Almirall D, Compton SN, Gunlicks-Stoessel M, Duan N, Murphy SA, 2012. Designing a pilot sequential multiple assignment randomized trial for developing an adaptive treatment strategy . Stat Med 31 ( 17 ), 1887–1902. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Bauer MS, McBride L, Williford WO, Glick H, Kinosian B, Altshuler L, Beresford T, Kilbourne AM, Sajatovic M, Cooperative Studies Program 430 Study, T., 2006. Collaborative care for bipolar disorder: Part II. Impact on clinical outcome, function, and costs . Psychiatr Serv 57 ( 7 ), 937–945. [ PubMed ] [ Google Scholar ]
• Bauer MS, Miller C, Kim B, Lew R, Weaver K, Coldwell C, Henderson K, Holmes S, Seibert MN, Stolzmann K, Elwy AR, Kirchner J, 2016. Partnering with health system operations leadership to develop a controlled implementation trial . Implement Sci 11 , 22. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Bauer MS, Miller CJ, Kim B, Lew R, Stolzmann K, Sullivan J, Riendeau R, Pitcock J, Williamson A, Connolly S, Elwy AR, Weaver K, 2019. Effectiveness of Implementing a Collaborative Chronic Care Model for Clinician Teams on Patient Outcomes and Health Status in Mental Health: A Randomized Clinical Trial . JAMA Netw Open 2 ( 3 ), e190230. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Bernal JL, Cummins S, Gasparrini A, 2017. Interrupted time series regression for the evaluation of public health interventions: a tutorial . Int J Epidemiol 46 ( 1 ), 348–355. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Betran AP, Bergel E, Griffin S, Melo A, Nguyen MH, Carbonell A, Mondlane S, Merialdi M, Temmerman M, Gulmezoglu AM, 2018. Provision of medical supply kits to improve quality of antenatal care in Mozambique: a stepped-wedge cluster randomised trial . Lancet Glob Health 6 ( 1 ), e57–e65. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Bhatt DL, Mehta C, 2016. Adaptive Designs for Clinical Trials . N Engl J Med 375 ( 1 ), 65–74. [ PubMed ] [ Google Scholar ]
• Bodenheimer T, Wagner EH, Grumbach K, 2002a. Improving primary care for patients with chronic illness . JAMA 288 ( 14 ), 1775–1779. [ PubMed ] [ Google Scholar ]
• Bodenheimer T, Wagner EH, Grumbach K, 2002b. Improving primary care for patients with chronic illness: the chronic care model, Part 2 . JAMA 288 ( 15 ), 1909–1914. [ PubMed ] [ Google Scholar ]
• Brown CA, Lilford RJ, 2006. The stepped wedge trial design: a systematic review . BMC medical research methodology 6 ( 1 ), 54. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Byiers BJ, Reichle J, Symons FJ, 2012. Single-subject experimental design for evidence-based practice . Am J Speech Lang Pathol 21 ( 4 ), 397–414. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Cheung YK, Chakraborty B, Davidson KW, 2015. Sequential multiple assignment randomized trial (SMART) with adaptive randomization for quality improvement in depression treatment program . Biometrics 71 ( 2 ), 450–459. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Collins LM, Dziak JJ, Kugler KC, Trail JB, 2014a. Factorial experiments: efficient tools for evaluation of intervention components . Am J Prev Med 47 ( 4 ), 498–504. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Collins LM, Dziak JJ, Li R, 2009. Design of experiments with multiple independent variables: a resource management perspective on complete and reduced factorial designs . Psychol Methods 14 ( 3 ), 202–224. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Collins LM, Murphy SA, Bierman KL, 2004. A conceptual framework for adaptive preventive interventions . Prev Sci 5 ( 3 ), 185–196. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Collins LM, Murphy SA, Nair VN, Strecher VJ, 2005. A strategy for optimizing and evaluating behavioral interventions . Ann Behav Med 30 ( 1 ), 65–73. [ PubMed ] [ Google Scholar ]
• Collins LM, Murphy SA, Strecher V, 2007. The multiphase optimization strategy (MOST) and the sequential multiple assignment randomized trial (SMART): new methods for more potent eHealth interventions . Am J Prev Med 32 ( 5 Suppl ), S112–118. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Collins LM, Nahum-Shani I, Almirall D, 2014b. Optimization of behavioral dynamic treatment regimens based on the sequential, multiple assignment, randomized trial (SMART) . Clin Trials 11 ( 4 ), 426–434. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Coulton S, Perryman K, Bland M, Cassidy P, Crawford M, Deluca P, Drummond C, Gilvarry E, Godfrey C, Heather N, Kaner E, Myles J, Newbury-Birch D, Oyefeso A, Parrott S, Phillips T, Shenker D, Shepherd J, 2009. Screening and brief interventions for hazardous alcohol use in accident and emergency departments: a randomised controlled trial protocol . BMC Health Serv Res 9 , 114. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Cousins K, Connor JL, Kypri K, 2014. Effects of the Campus Watch intervention on alcohol consumption and related harm in a university population . Drug Alcohol Depend 143 , 120–126. [ PubMed ] [ Google Scholar ]
• Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C, 2012. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact . Med Care 50 ( 3 ), 217–226. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Davey C, Hargreaves J, Thompson JA, Copas AJ, Beard E, Lewis JJ, Fielding KL, 2015. Analysis and reporting of stepped wedge randomised controlled trials: synthesis and critical appraisal of published studies, 2010 to 2014 . Trials 16 ( 1 ), 358. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Dimick JB, Ryan AM, 2014. Methods for evaluating changes in health care policy: the difference-in- differences approach . JAMA 312 ( 22 ), 2401–2402. [ PubMed ] [ Google Scholar ]
• Eccles M, Grimshaw J, Campbell M, Ramsay C, 2003. Research designs for studies evaluating the effectiveness of change and improvement strategies . Qual Saf Health Care 12 ( 1 ), 47–52. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Fisher RA, 1925, July Theory of statistical estimation In Mathematical Proceedings of the Cambridge Philosophical Society (Vol. 22, No. 5, pp. 700–725). Cambridge University Press. [ Google Scholar ]
• Fisher RA, 1935. The design of experiments . Oliver and Boyd, Edinburgh. [ Google Scholar ]
• Fretheim A, Soumerai SB, Zhang F, Oxman AD, Ross-Degnan D, 2013. Interrupted time-series analysis yielded an effect estimate concordant with the cluster-randomized controlled trial result . Journal of Clinical Epidemiology 66 ( 8 ), 883–887. [ PubMed ] [ Google Scholar ]
• Fretheim A, Zhang F, Ross-Degnan D, Oxman AD, Cheyne H, Foy R, Goodacre S, Herrin J, Kerse N, McKinlay RJ, Wright A, Soumerai SB, 2015. A reanalysis of cluster randomized trials showed interrupted time-series studies were valuable in health system evaluation . J Clin Epidemiol 68 ( 3 ), 324–333. [ PubMed ] [ Google Scholar ]
• Gaglio B, Shoup JA, Glasgow RE, 2013. The RE-AIM framework: a systematic review of use over time . Am J Public Health 103 ( 6 ), e38–46. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Gebski V, Ellingson K, Edwards J, Jernigan J, Kleinbaum D, 2012. Modelling interrupted time series to evaluate prevention and control of infection in healthcare . Epidemiol Infect 140 ( 12 ), 2131–2141. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Glasgow RE, Vogt TM, Boles SM, 1999. Evaluating the public health impact of health promotion interventions: the RE-AIM framework . Am J Public Health 89 ( 9 ), 1322–1327. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Hanani H, 1961. The existence and construction of balanced incomplete block designs . The Annals of Mathematical Statistics 32 ( 2 ), 361–386. [ Google Scholar ]
• Hanbury A, Farley K, Thompson C, Wilson PM, Chambers D, Holmes H, 2013. Immediate versus sustained effects: interrupted time series analysis of a tailored intervention . Implement Sci 8 , 130. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Handley MA, Lyles CR, McCulloch C, Cattamanchi A, 2018. Selecting and Improving Quasi-Experimental Designs in Effectiveness and Implementation Research . Annu Rev Public Health 39 , 5–25. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Hussey MA, Hughes JP, 2007. Design and analysis of stepped wedge cluster randomized trials . Contemp Clin Trials 28 ( 2 ), 182–191. [ PubMed ] [ Google Scholar ]
• Kilbourne AM, Almirall D, Eisenberg D, Waxmonsky J, Goodrich DE, Fortney JC, Kirchner JE, Solberg LI, Main D, Bauer MS, Kyle J, Murphy SA, Nord KM, Thomas MR, 2014a. Protocol: Adaptive Implementation of Effective Programs Trial (ADEPT): cluster randomized SMART trial comparing a standard versus enhanced implementation strategy to improve outcomes of a mood disorders program . Implement Sci 9 , 132. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Kilbourne AM, Almirall D, Goodrich DE, Lai Z, Abraham KM, Nord KM, Bowersox NW, 2014b. Enhancing outreach for persons with serious mental illness: 12-month results from a cluster randomized trial of an adaptive implementation strategy . Implement Sci 9 , 163. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Kilbourne AM, Bramlet M, Barbaresso MM, Nord KM, Goodrich DE, Lai Z, Post EP, Almirall D, Verchinina L, Duffy SA, Bauer MS, 2014c. SMI life goals: description of a randomized trial of a collaborative care model to improve outcomes for persons with serious mental illness . Contemp Clin Trials 39 ( 1 ), 74–85. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Kilbourne AM, Goodrich DE, Lai Z, Clogston J, Waxmonsky J, Bauer MS, 2012a. Life Goals Collaborative Care for patients with bipolar disorder and cardiovascular disease risk . Psychiatr Serv 63 ( 12 ), 1234–1238. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Kilbourne AM, Goodrich DE, Nord KM, Van Poppelen C, Kyle J, Bauer MS, Waxmonsky JA, Lai Z, Kim HM, Eisenberg D, Thomas MR, 2015. Long-Term Clinical Outcomes from a Randomized Controlled Trial of Two Implementation Strategies to Promote Collaborative Care Attendance in Community Practices . Adm Policy Ment Health 42 ( 5 ), 642–653. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Kilbourne AM, Neumann MS, Pincus HA, Bauer MS, Stall R, 2007. Implementing evidence-based interventions in health care: application of the replicating effective programs framework . Implement Sci 2 , 42. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Kilbourne AM, Neumann MS, Waxmonsky J, Bauer MS, Kim HM, Pincus HA, Thomas M, 2012b. Public-academic partnerships: evidence-based implementation: the role of sustained community-based practice and research partnerships . Psychiatr Serv 63 ( 3 ), 205–207. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Kilbourne AM, Post EP, Nossek A, Drill L, Cooley S, Bauer MS, 2008. Improving medical and psychiatric outcomes among individuals with bipolar disorder: a randomized controlled trial . Psychiatr Serv 59 ( 7 ), 760–768. [ PubMed ] [ Google Scholar ]
• Kirchner JE, Ritchie MJ, Pitcock JA, Parker LE, Curran GM, Fortney JC, 2014. Outcomes of a partnered facilitation strategy to implement primary care-mental health . J Gen Intern Med 29 Suppl 4 , 904–912. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Lei H, Nahum-Shani I, Lynch K, Oslin D, Murphy SA, 2012. A “SMART” design for building individualized treatment sequences . Annu Rev Clin Psychol 8 , 21–48. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Lew RA, Miller CJ, Kim B, Wu H, Stolzmann K, Bauer MS, 2019. A robust method to reduce imbalance for site-level randomized controlled implementation trial designs . Implementation Sci , 14 , 46. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Morgan CJ, 2018. Reducing bias using propensity score matching . J Nucl Cardiol 25 ( 2 ), 404–406. [ PubMed ] [ Google Scholar ]
• Morton V, Torgerson DJ, 2003. Effect of regression to the mean on decision making in health care . BMJ 326 ( 7398 ), 1083–1084. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Nahum-Shani I, Qian M, Almirall D, Pelham WE, Gnagy B, Fabiano GA, Waxmonsky JG, Yu J, Murphy SA, 2012. Experimental design and primary data analysis methods for comparing adaptive interventions . Psychol Methods 17 ( 4 ), 457–477. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• NeCamp T, Kilbourne A, Almirall D, 2017. Comparing cluster-level dynamic treatment regimens using sequential, multiple assignment, randomized trials: Regression estimation and sample size considerations . Stat Methods Med Res 26 ( 4 ), 1572–1589. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Neumann MS, Sogolow ED, 2000. Replicating effective programs: HIV/AIDS prevention technology transfer . AIDS Educ Prev 12 ( 5 Suppl ), 35–48. [ PubMed ] [ Google Scholar ]
• Pallmann P, Bedding AW, Choodari-Oskooei B, Dimairo M, Flight L, Hampson LV, Holmes J, Mander AP, Odondi L.o., Sydes MR, Villar SS, Wason JMS, Weir CJ, Wheeler GM, Yap C, Jaki T, 2018. Adaptive designs in clinical trials: why use them, and how to run and report them . BMC medicine 16 ( 1 ), 29–29. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Pape UJ, Millett C, Lee JT, Car J, Majeed A, 2013. Disentangling secular trends and policy impacts in health studies: use of interrupted time series analysis . J R Soc Med 106 ( 4 ), 124–129. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Pellegrini CA, Hoffman SA, Collins LM, Spring B, 2014. Optimization of remotely delivered intensive lifestyle treatment for obesity using the Multiphase Optimization Strategy: Opt-IN study protocol . Contemp Clin Trials 38 ( 2 ), 251–259. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Penfold RB, Zhang F, 2013. Use of Interrupted Time Series Analysis in Evaluating Health Care Quality Improvements . Academic Pediatrics 13 ( 6, Supplement ), S38–S44. [ PubMed ] [ Google Scholar ]
• Pridemore WA, Snowden AJ, 2009. Reduction in suicide mortality following a new national alcohol policy in Slovenia: an interrupted time-series analysis . Am J Public Health 99 ( 5 ), 915–920. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, Griffey R, Hensley M, 2011. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda . Adm Policy Ment Health 38 ( 2 ), 65–76. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Robson D, Spaducci G, McNeill A, Stewart D, Craig TJK, Yates M, Szatkowski L, 2017. Effect of implementation of a smoke-free policy on physical violence in a psychiatric inpatient setting: an interrupted time series analysis . Lancet Psychiatry 4 ( 7 ), 540–546. [ PubMed ] [ Google Scholar ]
• Schildcrout JS, Schisterman EF, Mercaldo ND, Rathouz PJ, Heagerty PJ, 2018. Extending the Case-Control Design to Longitudinal Data: Stratified Sampling Based on Repeated Binary Outcomes . Epidemiology 29 ( 1 ), 67–75. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Shadish WR, Cook Thomas D., Campbell Donald T, 2002. Experimental and quasi-experimental designs for generalized causal inference . Houghton Miffflin Company, Boston, MA. [ Google Scholar ]
• Simon GE, Ludman EJ, Bauer MS, Unutzer J, Operskalski B, 2006. Long-term effectiveness and cost of a systematic care program for bipolar disorder . Arch Gen Psychiatry 63 ( 5 ), 500–508. [ PubMed ] [ Google Scholar ]
• Stetler CB, Legro MW, Rycroft-Malone J, Bowman C, Curran G, Guihan M, Hagedorn H, Pineros S, Wallace CM, 2006. Role of “external facilitation” in implementation of research findings: a qualitative evaluation of facilitation experiences in the Veterans Health Administration . Implement Sci 1 , 23. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Taljaard M, McKenzie JE, Ramsay CR, Grimshaw JM, 2014. The use of segmented regression in analysing interrupted time series studies: an example in pre-hospital ambulance care . Implement Sci 9 , 77. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Wagner AK, Soumerai SB, Zhang F, Ross-Degnan D, 2002. Segmented regression analysis of interrupted time series studies in medication use research . J Clin Pharm Ther 27 ( 4 ), 299–309. [ PubMed ] [ Google Scholar ]
• Wagner EH, Austin BT, Von Korff M, 1996. Organizing care for patients with chronic illness . Milbank Q 74 ( 4 ), 511–544. [ PubMed ] [ Google Scholar ]
• Wyrick DL, Rulison KL, Fearnow-Kenney M, Milroy JJ, Collins LM, 2014. Moving beyond the treatment package approach to developing behavioral interventions: addressing questions that arose during an application of the Multiphase Optimization Strategy (MOST) . Transl Behav Med 4 ( 3 ), 252–259. [ PMC free article ] [ PubMed ] [ Google Scholar ]

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• Control Groups and Treatment Groups | Uses & Examples

Control Groups and Treatment Groups | Uses & Examples

Published on July 3, 2020 by Lauren Thomas . Revised on June 22, 2023.

In a scientific study, a control group is used to establish causality by isolating the effect of an independent variable .

Here, researchers change the independent variable in the treatment group and keep it constant in the control group. Then they compare the results of these groups.

Using a control group means that any change in the dependent variable can be attributed to the independent variable. This helps avoid extraneous variables or confounding variables from impacting your work, as well as a few types of research bias , like omitted variable bias .

Table of contents

Control groups in experiments, control groups in non-experimental research, importance of control groups, other interesting articles, frequently asked questions about control groups.

Control groups are essential to experimental design . When researchers are interested in the impact of a new treatment, they randomly divide their study participants into at least two groups:

• The treatment group (also called the experimental group ) receives the treatment whose effect the researcher is interested in.
• The control group receives either no treatment, a standard treatment whose effect is already known, or a placebo (a fake treatment to control for placebo effect ).

The treatment is any independent variable manipulated by the experimenters, and its exact form depends on the type of research being performed. In a medical trial, it might be a new drug or therapy. In public policy studies, it could be a new social policy that some receive and not others.

In a well-designed experiment, all variables apart from the treatment should be kept constant between the two groups. This means researchers can correctly measure the entire effect of the treatment without interference from confounding variables .

• You pay the students in the treatment group for achieving high grades.
• Students in the control group do not receive any money.

Studies can also include more than one treatment or control group. Researchers might want to examine the impact of multiple treatments at once, or compare a new treatment to several alternatives currently available.

• The treatment group gets the new pill.
• Control group 1 gets an identical-looking sugar pill (a placebo)
• Control group 2 gets a pill already approved to treat high blood pressure

Since the only variable that differs between the three groups is the type of pill, any differences in average blood pressure between the three groups can be credited to the type of pill they received.

• The difference between the treatment group and control group 1 demonstrates the effectiveness of the pill as compared to no treatment.
• The difference between the treatment group and control group 2 shows whether the new pill improves on treatments already available on the market.

Prevent plagiarism. Run a free check.

Although control groups are more common in experimental research, they can be used in other types of research too. Researchers generally rely on non-experimental control groups in two cases: quasi-experimental or matching design.

Control groups in quasi-experimental design

While true experiments rely on random assignment to the treatment or control groups, quasi-experimental design uses some criterion other than randomization to assign people.

Often, these assignments are not controlled by researchers, but are pre-existing groups that have received different treatments. For example, researchers could study the effects of a new teaching method that was applied in some classes in a school but not others, or study the impact of a new policy that is implemented in one state but not in the neighboring state.

In these cases, the classes that did not use the new teaching method, or the state that did not implement the new policy, is the control group.

Control groups in matching design

In correlational research , matching represents a potential alternate option when you cannot use either true or quasi-experimental designs.

In matching designs, the researcher matches individuals who received the “treatment”, or independent variable under study, to others who did not–the control group.

Each member of the treatment group thus has a counterpart in the control group identical in every way possible outside of the treatment. This ensures that the treatment is the only source of potential differences in outcomes between the two groups.

Control groups help ensure the internal validity of your research. You might see a difference over time in your dependent variable in your treatment group. However, without a control group, it is difficult to know whether the change has arisen from the treatment. It is possible that the change is due to some other variables.

If you use a control group that is identical in every other way to the treatment group, you know that the treatment–the only difference between the two groups–must be what has caused the change.

For example, people often recover from illnesses or injuries over time regardless of whether they’ve received effective treatment or not. Thus, without a control group, it’s difficult to determine whether improvements in medical conditions come from a treatment or just the natural progression of time.

Risks from invalid control groups

If your control group differs from the treatment group in ways that you haven’t accounted for, your results may reflect the interference of confounding variables instead of your independent variable.

Minimizing this risk

A few methods can aid you in minimizing the risk from invalid control groups.

• Ensure that all potential confounding variables are accounted for , preferably through an experimental design if possible, since it is difficult to control for all the possible confounders outside of an experimental environment.
• Use double-blinding . This will prevent the members of each group from modifying their behavior based on whether they were placed in the treatment or control group, which could then lead to biased outcomes.
• Randomly assign your subjects into control and treatment groups. This method will allow you to not only minimize the differences between the two groups on confounding variables that you can directly observe, but also those you cannot.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Student’s  t -distribution
• Normal distribution
• Null and Alternative Hypotheses
• Chi square tests
• Confidence interval
• Quartiles & Quantiles
• Cluster sampling
• Stratified sampling
• Data cleansing
• Reproducibility vs Replicability
• Peer review
• Prospective cohort study

Research bias

• Implicit bias
• Cognitive bias
• Placebo effect
• Hawthorne effect
• Hindsight bias
• Affect heuristic
• Social desirability bias

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

An experimental group, also known as a treatment group, receives the treatment whose effect researchers wish to study, whereas a control group does not. They should be identical in all other ways.

A true experiment (a.k.a. a controlled experiment) always includes at least one control group that doesn’t receive the experimental treatment.

However, some experiments use a within-subjects design to test treatments without a control group. In these designs, you usually compare one group’s outcomes before and after a treatment (instead of comparing outcomes between different groups).

For strong internal validity , it’s usually best to include a control group if possible. Without a control group, it’s harder to be certain that the outcome was caused by the experimental treatment and not by other variables.

A confounding variable , also called a confounder or confounding factor, is a third variable in a study examining a potential cause-and-effect relationship.

A confounding variable is related to both the supposed cause and the supposed effect of the study. It can be difficult to separate the true effect of the independent variable from the effect of the confounding variable.

In your research design , it’s important to identify potential confounding variables and plan how you will reduce their impact.

There are several methods you can use to decrease the impact of confounding variables on your research: restriction, matching, statistical control and randomization.

In restriction , you restrict your sample by only including certain subjects that have the same values of potential confounding variables.

In matching , you match each of the subjects in your treatment group with a counterpart in the comparison group. The matched subjects have the same values on any potential confounding variables, and only differ in the independent variable .

In statistical control , you include potential confounders as variables in your regression .

In randomization , you randomly assign the treatment (or independent variable) in your study to a sufficiently large number of subjects, which allows you to control for all potential confounding variables.

Experimental design means planning a set of procedures to investigate a relationship between variables . To design a controlled experiment, you need:

• A testable hypothesis
• At least one independent variable that can be precisely manipulated
• At least one dependent variable that can be precisely measured

When designing the experiment, you decide:

• How you will manipulate the variable(s)
• How you will control for any potential confounding variables
• How many subjects or samples will be included in the study
• How subjects will be assigned to treatment levels

Experimental design is essential to the internal and external validity of your experiment.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Thomas, L. (2023, June 22). Control Groups and Treatment Groups | Uses & Examples. Scribbr. Retrieved August 12, 2024, from https://www.scribbr.com/methodology/control-group/

Is this article helpful?

Lauren Thomas

Other students also liked, what is a controlled experiment | definitions & examples, random assignment in experiments | introduction & examples, single, double, & triple blind study | definition & examples, "i thought ai proofreading was useless but..".

I've been using Scribbr for years now and I know it's a service that won't disappoint. It does a good job spotting mistakes”

12th August 2024: digital purchasing is currently unavailable on Cambridge Core. Due to recent technical disruption affecting our publishing operation, we are experiencing some delays to publication. We are working hard to restore services as soon as possible and apologise for the inconvenience. For further updates please visit our website: https://www.cambridge.org/news-and-insights/technical-incident

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings .

Login Alert

• > The Cambridge Handbook of Research Methods and Statistics for the Social and Behavioral Sciences
• > Quasi-Experimental Research

Book contents

• The Cambridge Handbook of Research Methods and Statistics for the Social and Behavioral Sciences
• Cambridge Handbooks in Psychology
• Copyright page
• Contributors
• Part I From Idea to Reality: The Basics of Research
• Part II The Building Blocks of a Study
• Part III Data Collection
• 13 Cross-Sectional Studies
• 14 Quasi-Experimental Research
• 15 Non-equivalent Control Group Pretest–Posttest Design in Social and Behavioral Research
• 16 Experimental Methods
• 17 Longitudinal Research: A World to Explore
• 18 Online Research Methods
• 19 Archival Data
• 20 Qualitative Research Design
• Part IV Statistical Approaches
• Part V Tips for a Successful Research Career

14 - Quasi-Experimental Research

from Part III - Data Collection

Published online by Cambridge University Press:  25 May 2023

In this chapter, we discuss the logic and practice of quasi-experimentation. Specifically, we describe four quasi-experimental designs – one-group pretest–posttest designs, non-equivalent group designs, regression discontinuity designs, and interrupted time-series designs – and their statistical analyses in detail. Both simple quasi-experimental designs and embellishments of these simple designs are presented. Potential threats to internal validity are illustrated along with means of addressing their potentially biasing effects so that these effects can be minimized. In contrast to quasi-experiments, randomized experiments are often thought to be the gold standard when estimating the effects of treatment interventions. However, circumstances frequently arise where quasi-experiments can usefully supplement randomized experiments or when quasi-experiments can fruitfully be used in place of randomized experiments. Researchers need to appreciate the relative strengths and weaknesses of the various quasi-experiments so they can choose among pre-specified designs or craft their own unique quasi-experiments.

Access options

Save book to kindle.

To save this book to your Kindle, first ensure [email protected] is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle .

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service .

• Quasi-Experimental Research
• By Charles S. Reichardt , Daniel Storage , Damon Abraham
• Edited by Austin Lee Nichols , Central European University, Vienna , John Edlund , Rochester Institute of Technology, New York
• Book: The Cambridge Handbook of Research Methods and Statistics for the Social and Behavioral Sciences
• Online publication: 25 May 2023
• Chapter DOI: https://doi.org/10.1017/9781009010054.015

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox .

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive .

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

Chapter 7: Nonexperimental Research

Quasi-Experimental Research

Learning Objectives

• Explain what quasi-experimental research is and distinguish it clearly from both experimental and correlational research.
• Describe three different types of quasi-experimental research designs (nonequivalent groups, pretest-posttest, and interrupted time series) and identify examples of each one.

The prefix  quasi  means “resembling.” Thus quasi-experimental research is research that resembles experimental research but is not true experimental research. Although the independent variable is manipulated, participants are not randomly assigned to conditions or orders of conditions (Cook & Campbell, 1979). [1] Because the independent variable is manipulated before the dependent variable is measured, quasi-experimental research eliminates the directionality problem. But because participants are not randomly assigned—making it likely that there are other differences between conditions—quasi-experimental research does not eliminate the problem of confounding variables. In terms of internal validity, therefore, quasi-experiments are generally somewhere between correlational studies and true experiments.

Quasi-experiments are most likely to be conducted in field settings in which random assignment is difficult or impossible. They are often conducted to evaluate the effectiveness of a treatment—perhaps a type of psychotherapy or an educational intervention. There are many different kinds of quasi-experiments, but we will discuss just a few of the most common ones here.

Nonequivalent Groups Design

Recall that when participants in a between-subjects experiment are randomly assigned to conditions, the resulting groups are likely to be quite similar. In fact, researchers consider them to be equivalent. When participants are not randomly assigned to conditions, however, the resulting groups are likely to be dissimilar in some ways. For this reason, researchers consider them to be nonequivalent. A  nonequivalent groups design , then, is a between-subjects design in which participants have not been randomly assigned to conditions.

Imagine, for example, a researcher who wants to evaluate a new method of teaching fractions to third graders. One way would be to conduct a study with a treatment group consisting of one class of third-grade students and a control group consisting of another class of third-grade students. This design would be a nonequivalent groups design because the students are not randomly assigned to classes by the researcher, which means there could be important differences between them. For example, the parents of higher achieving or more motivated students might have been more likely to request that their children be assigned to Ms. Williams’s class. Or the principal might have assigned the “troublemakers” to Mr. Jones’s class because he is a stronger disciplinarian. Of course, the teachers’ styles, and even the classroom environments, might be very different and might cause different levels of achievement or motivation among the students. If at the end of the study there was a difference in the two classes’ knowledge of fractions, it might have been caused by the difference between the teaching methods—but it might have been caused by any of these confounding variables.

Of course, researchers using a nonequivalent groups design can take steps to ensure that their groups are as similar as possible. In the present example, the researcher could try to select two classes at the same school, where the students in the two classes have similar scores on a standardized math test and the teachers are the same sex, are close in age, and have similar teaching styles. Taking such steps would increase the internal validity of the study because it would eliminate some of the most important confounding variables. But without true random assignment of the students to conditions, there remains the possibility of other important confounding variables that the researcher was not able to control.

Pretest-Posttest Design

In a  pretest-posttest design , the dependent variable is measured once before the treatment is implemented and once after it is implemented. Imagine, for example, a researcher who is interested in the effectiveness of an antidrug education program on elementary school students’ attitudes toward illegal drugs. The researcher could measure the attitudes of students at a particular elementary school during one week, implement the antidrug program during the next week, and finally, measure their attitudes again the following week. The pretest-posttest design is much like a within-subjects experiment in which each participant is tested first under the control condition and then under the treatment condition. It is unlike a within-subjects experiment, however, in that the order of conditions is not counterbalanced because it typically is not possible for a participant to be tested in the treatment condition first and then in an “untreated” control condition.

If the average posttest score is better than the average pretest score, then it makes sense to conclude that the treatment might be responsible for the improvement. Unfortunately, one often cannot conclude this with a high degree of certainty because there may be other explanations for why the posttest scores are better. One category of alternative explanations goes under the name of  history . Other things might have happened between the pretest and the posttest. Perhaps an antidrug program aired on television and many of the students watched it, or perhaps a celebrity died of a drug overdose and many of the students heard about it. Another category of alternative explanations goes under the name of  maturation . Participants might have changed between the pretest and the posttest in ways that they were going to anyway because they are growing and learning. If it were a yearlong program, participants might become less impulsive or better reasoners and this might be responsible for the change.

Another alternative explanation for a change in the dependent variable in a pretest-posttest design is  regression to the mean . This refers to the statistical fact that an individual who scores extremely on a variable on one occasion will tend to score less extremely on the next occasion. For example, a bowler with a long-term average of 150 who suddenly bowls a 220 will almost certainly score lower in the next game. Her score will “regress” toward her mean score of 150. Regression to the mean can be a problem when participants are selected for further study  because  of their extreme scores. Imagine, for example, that only students who scored especially low on a test of fractions are given a special training program and then retested. Regression to the mean all but guarantees that their scores will be higher even if the training program has no effect. A closely related concept—and an extremely important one in psychological research—is  spontaneous remission . This is the tendency for many medical and psychological problems to improve over time without any form of treatment. The common cold is a good example. If one were to measure symptom severity in 100 common cold sufferers today, give them a bowl of chicken soup every day, and then measure their symptom severity again in a week, they would probably be much improved. This does not mean that the chicken soup was responsible for the improvement, however, because they would have been much improved without any treatment at all. The same is true of many psychological problems. A group of severely depressed people today is likely to be less depressed on average in 6 months. In reviewing the results of several studies of treatments for depression, researchers Michael Posternak and Ivan Miller found that participants in waitlist control conditions improved an average of 10 to 15% before they received any treatment at all (Posternak & Miller, 2001) [2] . Thus one must generally be very cautious about inferring causality from pretest-posttest designs.

Does Psychotherapy Work?

Early studies on the effectiveness of psychotherapy tended to use pretest-posttest designs. In a classic 1952 article, researcher Hans Eysenck summarized the results of 24 such studies showing that about two thirds of patients improved between the pretest and the posttest (Eysenck, 1952) [3] . But Eysenck also compared these results with archival data from state hospital and insurance company records showing that similar patients recovered at about the same rate  without  receiving psychotherapy. This parallel suggested to Eysenck that the improvement that patients showed in the pretest-posttest studies might be no more than spontaneous remission. Note that Eysenck did not conclude that psychotherapy was ineffective. He merely concluded that there was no evidence that it was, and he wrote of “the necessity of properly planned and executed experimental studies into this important field” (p. 323). You can read the entire article here: Classics in the History of Psychology .

Fortunately, many other researchers took up Eysenck’s challenge, and by 1980 hundreds of experiments had been conducted in which participants were randomly assigned to treatment and control conditions, and the results were summarized in a classic book by Mary Lee Smith, Gene Glass, and Thomas Miller (Smith, Glass, & Miller, 1980) [4] . They found that overall psychotherapy was quite effective, with about 80% of treatment participants improving more than the average control participant. Subsequent research has focused more on the conditions under which different types of psychotherapy are more or less effective.

Interrupted Time Series Design

A variant of the pretest-posttest design is the  interrupted time-series design . A time series is a set of measurements taken at intervals over a period of time. For example, a manufacturing company might measure its workers’ productivity each week for a year. In an interrupted time series-design, a time series like this one is “interrupted” by a treatment. In one classic example, the treatment was the reduction of the work shifts in a factory from 10 hours to 8 hours (Cook & Campbell, 1979) [5] . Because productivity increased rather quickly after the shortening of the work shifts, and because it remained elevated for many months afterward, the researcher concluded that the shortening of the shifts caused the increase in productivity. Notice that the interrupted time-series design is like a pretest-posttest design in that it includes measurements of the dependent variable both before and after the treatment. It is unlike the pretest-posttest design, however, in that it includes multiple pretest and posttest measurements.

Figure 7.3 shows data from a hypothetical interrupted time-series study. The dependent variable is the number of student absences per week in a research methods course. The treatment is that the instructor begins publicly taking attendance each day so that students know that the instructor is aware of who is present and who is absent. The top panel of  Figure 7.3 shows how the data might look if this treatment worked. There is a consistently high number of absences before the treatment, and there is an immediate and sustained drop in absences after the treatment. The bottom panel of  Figure 7.3 shows how the data might look if this treatment did not work. On average, the number of absences after the treatment is about the same as the number before. This figure also illustrates an advantage of the interrupted time-series design over a simpler pretest-posttest design. If there had been only one measurement of absences before the treatment at Week 7 and one afterward at Week 8, then it would have looked as though the treatment were responsible for the reduction. The multiple measurements both before and after the treatment suggest that the reduction between Weeks 7 and 8 is nothing more than normal week-to-week variation.

Combination Designs

A type of quasi-experimental design that is generally better than either the nonequivalent groups design or the pretest-posttest design is one that combines elements of both. There is a treatment group that is given a pretest, receives a treatment, and then is given a posttest. But at the same time there is a control group that is given a pretest, does  not  receive the treatment, and then is given a posttest. The question, then, is not simply whether participants who receive the treatment improve but whether they improve  more  than participants who do not receive the treatment.

Imagine, for example, that students in one school are given a pretest on their attitudes toward drugs, then are exposed to an antidrug program, and finally are given a posttest. Students in a similar school are given the pretest, not exposed to an antidrug program, and finally are given a posttest. Again, if students in the treatment condition become more negative toward drugs, this change in attitude could be an effect of the treatment, but it could also be a matter of history or maturation. If it really is an effect of the treatment, then students in the treatment condition should become more negative than students in the control condition. But if it is a matter of history (e.g., news of a celebrity drug overdose) or maturation (e.g., improved reasoning), then students in the two conditions would be likely to show similar amounts of change. This type of design does not completely eliminate the possibility of confounding variables, however. Something could occur at one of the schools but not the other (e.g., a student drug overdose), so students at the first school would be affected by it while students at the other school would not.

Finally, if participants in this kind of design are randomly assigned to conditions, it becomes a true experiment rather than a quasi experiment. In fact, it is the kind of experiment that Eysenck called for—and that has now been conducted many times—to demonstrate the effectiveness of psychotherapy.

Key Takeaways

• Quasi-experimental research involves the manipulation of an independent variable without the random assignment of participants to conditions or orders of conditions. Among the important types are nonequivalent groups designs, pretest-posttest, and interrupted time-series designs.
• Quasi-experimental research eliminates the directionality problem because it involves the manipulation of the independent variable. It does not eliminate the problem of confounding variables, however, because it does not involve random assignment to conditions. For these reasons, quasi-experimental research is generally higher in internal validity than correlational studies but lower than true experiments.
• Practice: Imagine that two professors decide to test the effect of giving daily quizzes on student performance in a statistics course. They decide that Professor A will give quizzes but Professor B will not. They will then compare the performance of students in their two sections on a common final exam. List five other variables that might differ between the two sections that could affect the results.
• regression to the mean
• spontaneous remission

Image Descriptions

Figure 7.3 image description: Two line graphs charting the number of absences per week over 14 weeks. The first 7 weeks are without treatment and the last 7 weeks are with treatment. In the first line graph, there are between 4 to 8 absences each week. After the treatment, the absences drop to 0 to 3 each week, which suggests the treatment worked. In the second line graph, there is no noticeable change in the number of absences per week after the treatment, which suggests the treatment did not work. [Return to Figure 7.3]

• Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design & analysis issues in field settings . Boston, MA: Houghton Mifflin. ↵
• Posternak, M. A., & Miller, I. (2001). Untreated short-term course of major depression: A meta-analysis of studies using outcomes from studies using wait-list control groups. Journal of Affective Disorders, 66 , 139–146. ↵
• Eysenck, H. J. (1952). The effects of psychotherapy: An evaluation. Journal of Consulting Psychology, 16 , 319–324. ↵
• Smith, M. L., Glass, G. V., & Miller, T. I. (1980). The benefits of psychotherapy . Baltimore, MD: Johns Hopkins University Press. ↵

A between-subjects design in which participants have not been randomly assigned to conditions.

The dependent variable is measured once before the treatment is implemented and once after it is implemented.

A category of alternative explanations for differences between scores such as events that happened between the pretest and posttest, unrelated to the study.

An alternative explanation that refers to how the participants might have changed between the pretest and posttest in ways that they were going to anyway because they are growing and learning.

The statistical fact that an individual who scores extremely on a variable on one occasion will tend to score less extremely on the next occasion.

The tendency for many medical and psychological problems to improve over time without any form of treatment.

A set of measurements taken at intervals over a period of time that are interrupted by a treatment.

Research Methods in Psychology - 2nd Canadian Edition Copyright © 2015 by Paul C. Price, Rajiv Jhangiani, & I-Chant A. Chiang is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License , except where otherwise noted.

Share This Book

Research Methodologies Guide

• Action Research
• Bibliometrics
• Case Studies
• Content Analysis
• Digital Scholarship This link opens in a new window
• Documentary
• Ethnography
• Focus Groups
• Grounded Theory
• Life Histories/Autobiographies
• Longitudinal
• Participant Observation
• Qualitative Research (General)

Quasi-Experimental Design

• Usability Studies

Quasi-Experimental Design is a unique research methodology because it is characterized by what is lacks. For example, Abraham & MacDonald (2011) state:

" Quasi-experimental research is similar to experimental research in that there is manipulation of an independent variable. It differs from experimental research because either there is no control group, no random selection, no random assignment, and/or no active manipulation. "

This type of research is often performed in cases where a control group cannot be created or random selection cannot be performed. This is often the case in certain medical and psychological studies. 

For more information on quasi-experimental design, review the resources below: 

Where to Start

Below are listed a few tools and online guides that can help you start your Quasi-experimental research. These include free online resources and resources available only through ISU Library.

• Quasi-Experimental Research Designs by Bruce A. Thyer This pocket guide describes the logic, design, and conduct of the range of quasi-experimental designs, encompassing pre-experiments, quasi-experiments making use of a control or comparison group, and time-series designs. An introductory chapter describes the valuable role these types of studies have played in social work, from the 1930s to the present. Subsequent chapters delve into each design type's major features, the kinds of questions it is capable of answering, and its strengths and limitations.
• Experimental and Quasi-Experimental Designs for Research by Donald T. Campbell; Julian C. Stanley. Call Number: Q175 C152e Written 1967 but still used heavily today, this book examines research designs for experimental and quasi-experimental research, with examples and judgments about each design's validity.

Online Resources

• Quasi-Experimental Design From the Web Center for Social Research Methods, this is a very good overview of quasi-experimental design.
• Experimental and Quasi-Experimental Research From Colorado State University.
• Quasi-experimental design--Wikipedia, the free encyclopedia Wikipedia can be a useful place to start your research- check the citations at the bottom of the article for more information.
• << Previous: Qualitative Research (General)
• Next: Sampling >>
• Last Updated: Aug 12, 2024 4:07 PM
• URL: https://instr.iastate.libguides.com/researchmethods

• Search Menu
• Sign in through your institution
• Advance articles
• Editor's Choice
• Supplement Archive
• Cover Archive
• IDSA Guidelines
• IDSA Journals
• The Journal of Infectious Diseases
• Open Forum Infectious Diseases
• Photo Quizzes
• State-of-the-Art Reviews
• Voices of ID
• Author Guidelines
• Open Access
• Why Publish
• IDSA Journals Calls for Papers
• Advertising and Corporate Services
• Advertising
• Journals Career Network
• Reprints and ePrints
• Sponsored Supplements
• Branded Books
• About Clinical Infectious Diseases
• About the Infectious Diseases Society of America
• About the HIV Medicine Association
• IDSA COI Policy
• Editorial Board
• Self-Archiving Policy
• For Reviewers
• For Press Offices
• Journals on Oxford Academic
• Books on Oxford Academic

Article Contents

Two-group tests, regression analysis, time-series analysis, adding a control group, acknowledgments.

• < Previous

Statistical Analysis and Application of Quasi Experiments to Antimicrobial Resistance Intervention Studies

• Article contents
• Figures & tables
• Supplementary Data

George M. Eliopoulos, Michelle Shardell, Anthony D. Harris, Samer S. El-Kamary, Jon P. Furuno, Ram R. Miller, Eli N. Perencevich, Statistical Analysis and Application of Quasi Experiments to Antimicrobial Resistance Intervention Studies, Clinical Infectious Diseases , Volume 45, Issue 7, 1 October 2007, Pages 901–907, https://doi.org/10.1086/521255

• Permissions Icon Permissions

Quasi-experimental study designs are frequently used to assess interventions that aim to limit the emergence of antimicrobial-resistant pathogens. However, previous studies using these designs have often used suboptimal statistical methods, which may result in researchers making spurious conclusions. Methods used to analyze quasi-experimental data include 2-group tests, regression analysis, and time-series analysis, and they all have specific assumptions, data requirements, strengths, and limitations. An example of a hospital-based intervention to reduce methicillin-resistant Staphylococcus aureus infection rates and reduce overall length of stay is used to explore these methods.

Choosing the appropriate study design is critical when performing antimicrobial resistance intervention studies. When randomized studies in single hospitals or multihospital cluster-randomized trials are infeasible, investigators often choose before-and-after quasi-experimental designs [ 1 , 2 ]. Quasi-experimental studies can assess interventions applied at the hospital or unit level (e.g., hygiene education program in the medical intensive care unit [MICU] [ 3 ]) or individual level (e.g., methicillin-resistant Staphylococcus aureus [MRSA] decolonization programs [ 4 ]), in which data are collected at equally spaced time intervals (e.g., monthly) before and after the intervention.

Nonrandomization and the resulting data structure of quasi experiments impart several methodological challenges for analysis. First, common statistical methods, including 2-group Student's t tests and linear regression, were developed to analyze independent, individual-level observations, whereas quasi-experimental data are typically correlated unit-level observations; for example, MRSA counts (defined as the number of MRSA infections at multiple time intervals) collected 1 month apart are likely more similar than MRSA counts collected 2 months apart. Second, nonrandom assignment of the intervention often necessitates analytical control for potential confounders.

Unfortunately, application of statistical techniques to quasi experiments is rarely described in introductory biostatistics texts and courses. We aim to provide a resource for bridging the gap between clinician researchers and biostatisticians by introducing clinicians to statistical analysis of quasi experiments while guiding biostatisticians regarding design-related challenges of intervention studies for controlling antimicrobial resistance, thereby improving conduct and reporting of these studies, as recently outlined [ 5 , 6 ]. Strength of evidence from quasi-experimental data depends on the study design [ 1 , 2 , 7 ]. Studies with a concurrent nonequivalent control group provide stronger evidence about effectiveness of an intervention than do studies without a control group. Also, studies with several preintervention observations provide stronger evidence than do studies with few or no preintervention observations. As discussed below, the internal validity of quasi experiments is partially related to study design elements that affect researchers' ability to control for correlation, confounding, and time trends. Thus, before a study is initiated, hypotheses should be clearly stated, and design and analysis plans should be carefully developed.

We discuss several statistical techniques using the following example (motivated by a study by Pittet et al. [ 3 ]). After several months of abnormally high MRSA infection rates in the MICU, a hospital epidemiologist launches an education-based intervention to increase compliance with hand-disinfection procedures. The epidemiologist aims to compare rates of positivity for MRSA in clinical cultures before and after implementing the intervention. A secondary aim is to assess whether the intervention decreases overall length of stay (LOS) in the MICU. For both aims, data from 36 months before the intervention (2003–2005) are compared with data from 12 months after the intervention (2006). For ease of explanation, we first describe statistical methods for this example without a control group. We then discuss adaptations of methods for studies with a nonequivalent control group.

We discuss 2-group tests (e.g., Student's t test and χ 2 test), regression analysis (including segmented models), and time-series analysis in application to quasi-experimental studies of interventions to control antibiotic-resistant bacterial pathogens. We use simulated data for illustration and review data requirements, software, strengths, and limitations for each statistical method (tables 1 and 2 ). Persons seeking additional resources on statistics or quasi experiments are urged to consult a statistics primer [ 8 ] and literature regarding quasi-experimental studies, respectively [ 1 , 2 , 7 ].

Statistical method and software commands by outcome type.

Characteristics of each statistical method.

Two-group (i.e., bivariate) tests make crude comparisons (i.e., unadjusted for confounders) of MRSA infection rates and mean LOS in pre- and postintervention periods. We specifically discuss Student's t tests for continuous outcomes (e.g., LOS) and 2-rate χ 2 tests for count outcomes (e.g., number of MRSA infections).

Continuous outcomes. For continuous outcomes, 2 mean values are compared using Student's t test. In our example, we test the equality of the mean LOS before and after the education-based hand disinfection intervention. When data from several preintervention and postintervention periods are collected, as in interrupted time-series study designs [ 1 , 2 , 7 ], data from multiple periods before and after implementation of the intervention are pooled to produce 2 grand mean values. For example, 2300 patients per year (6900 total) with a mean LOS of 3.0 days during 2003–2005 (preintervention period) and 2800 patients with a mean LOS of 2.5 days in 2006 (postintervention period) can be compared. However, Student's t tests are sensitive to outlying values. If some patients have atypically long LOS, the median value is the preferred measurement of central tendency. Transformation (e.g., natural logarithm) of individual patients' LOS or a nonparametric test to compare median values (e.g., Wilcoxon rank-sum test) can be used.

Count outcomes. Crude comparisons can be made for count outcomes (e.g., number of MRSA infections) by performing a 2-rate χ 2 test. In our example, because the number of hospital admissions varies over time, comparing numbers of pre- and postintervention MRSA infections may produce invalid results. Summarizing data as a proportion, with the number of MRSA infections divided by the number of hospital admissions (e.g., 150 infections/6900 hospital admissions [2.2%], compared with 40 infections/2800 hospital admissions [1.4%]; P = .009), is appropriate if all patients are observed for the same duration of follow-up, when the proportion is interpreted as risk of infection for that particular follow-up period (e.g., 3-day risk of MRSA infection). However, observation of patients in infection-control studies is typically limited to hospital stays that vary in duration. The 2-rate χ 2 test accommodates this difference by comparing rates (number of infections per unit of person-time) between pre- and postintervention periods [ 6 ]. Given 150 and 40 infections before and after the intervention, respectively, if 6700 preintervention person-days per year (20,100 total) and 6600 postintervention person-days are observed, then the rates are 7.5 and 6.1 infections per 1000 person-days before and after the intervention, respectively ( P = .21). Thus, correcting for person-time using rates may produce conclusions different from those using proportions.

The 2-rate χ 2 test assumes that infection counts follow a Poisson distribution [ 9 , 10–11 ]. The Poisson assumption implies that the mean infection count per person-time equals the variance in the infection count for that person-time. If this assumption is violated, then incorrect SE estimates are calculated, resulting in incorrect confidence intervals and P values.

In interrupted time-series study designs, rates are collected at several periods, allowing the variance of infection counts per person per unit of time to be empirically estimated and compared with the mean value. If the “mean equals variance” assumption is not valid, a test using “robust” SEs on the basis of empirically estimated variances is recommended [ 12 , 13 ]. Consider 12 months of data on MRSA infection rates with a mean rate of 2.8 cases per 1000 person-days and a variance of 2.2. Thus, the Poisson assumption appears valid. In contrast, consider MRSA infection rates with a mean rate of 4.4 cases per 1000 person-days and a variance of 6.6. This latter example is typical such that satisfying the Poisson assumption is rare in practical applications. Therefore, researchers should perform both 2-rate χ 2 tests (with and without robust SEs) to evaluate whether confidence intervals and P values vary across assumptions. If conclusions differ between the 2 methods, test results using the more conservative robust SEs should be reported.

Strengths and limitations. Strengths of 2-group tests include simplicity, interpretability of results, and minimal data requirements (2 observation periods) ( table 2 ). These tests can accommodate >2 groups (e.g., before intervention, after intervention, and after intervention plus change in antimicrobial prescribing), using analysis of variance for continuous outcomes and χ 2 tests for count outcomes.

Two-group tests are limited by several assumptions. One assumption, independence between patients admitted to the hospital in the same period, is implausible because infectious organisms are transmissible. Independence of observations between periods is also implausible, because patients admitted to the hospital in different months may be exposed to constant antibiotic prescribing patterns. Also, without multiple levels of stratification, the ability to adjust for potential confounders (e.g., differences in severity of illness) is limited. Last, 2-group tests can detect changes in outcome levels but not changes in trends (e.g., monthly increases or decreases in the MRSA infection rate). If we use the 2-rate χ 2 test with data in figure 1 , the MRSA infection rates for 36 months before and 12 months after an intervention are 6.8 and 6.6 cases per 1000 person-days, respectively ( P = .87). However, figure 1 shows rates increasing by 0.25 cases per 1000 person-days per month until implementation of the intervention, then decreasing by 0.75 cases per 1000 person-days per month. By pooling counts into single pre- and postintervention rates, the 2-rate χ 2 test cannot detect this change in slope or trend, incorrectly finding no evidence of effectiveness of the intervention. To detect changes in slopes, a different statistical method, such as segmented regression, is needed.

Changes in rate of infection with methicillin-resistant Staphylococcus aureus (MRSA) over time before and after an intervention implemented at month 36, showing a change in slope that would not be detected by 2-group tests. Preintervention and postintervention rates are 6.8 and 6.6 infections per 1000 person-days, respectively ( P = .87, by 2-rate χ 2 test). Preintervention and postintervention slopes are 0.25 and -0.75 infections per 1000 person-days per month, respectively.

Regression analysis quantifies the relationship between an outcome (e.g., LOS or MRSA infection) and an intervention, allowing for statistical control of known confounders. Linear regression is used for continuous normally distributed outcomes (e.g., average monthly LOS or log-transformed individual LOS). Other outcome types, including MRSA counts, require analysis using generalized linear models [ 14 ]. In our example, MRSA infections are considered as MRSA counts per time period with an assumed Poisson distribution; thus, the appropriate method is Poisson regression.

Unlike in statistical literature, in clinical literature, “segmented regression” means regression analysis in which changes in mean outcome levels and trends before and after an intervention are estimated [ 15 ]. If changes in slopes are not estimated (e.g., nonsegmented regression model is fit), then estimates of the slopes may be biased, and changes in time trends attributable to the intervention would be undetected. Segmented regression models can be fit to estimate changes in levels and trends. In our example below, we estimate pre- and postintervention changes in LOS and MRSA levels and trends.

Continuous outcomes. Although individual LOS is usually skewed, mean monthly LOS is approximately normally distributed for large sample sizes (i.e., >30 patients per month). If LOS increases over time secondary to a steady increase in MRSA infection rates, regression analysis can model this pattern and estimate the effect of an intervention controlling for potential confounders (e.g., age and reasons for hospitalization). Given intervention status and potential confounders, the outcome variable (in this case, LOS) must satisfy the assumption of having constant variance.

Using the same data, we estimate changes in mean LOS, controlling for trends, using 2 different models ( figure 2 ). Figure 2A shows the results of nonsegmented linear regression, which cannot assess a change in time trend (i.e., slope). Figure 2B shows the results of segmented linear regression, which allows the slopes to differ before and after the intervention. Compared with the model in figure 2 A , the estimated time trend using segmented linear regression in figure 2 B is flatter after the intervention. Forcing equal slopes before and after the intervention when they are unequal can lead to spurious conclusions about an intervention's effectiveness.

Interrupted time-series data regarding length of hospital stay (LOS) simulated from a segmented linear regression model with a change in slope (before vs. after the intervention), fit with a nonsegmented linear regression model that cannot estimate a change in slope (A) and a segmented linear regression model that can estimate a change in slope (B). The intervention was implemented at month 36.

Count outcomes. Poisson regression is preferred over linear regression for estimating the association between the intervention and monthly MRSA infection rates, controlling for time trend, because counts are not normally distributed ( figure 3 ). Differences estimated from this model are summarized as incident rate ratios of MRSA infections.

Figure 3. Interrupted time-series methicillin-resistant Staphylococcus aureus (MRSA) infection data simulated from a segmented Poisson regression model with a change in slope (before vs. after the intervention), fit with a nonsegmented Poisson regression model that cannot estimate a change in slope (A) and a segmented Poisson regression model that can estimate a change in slope (B). The intervention was implemented at month 36.

Using the same data, we estimate changes in MRSA infection rates, controlling for trends, using 2 models ( figure 3 ). Figure 3A shows the results of nonsegmented Poisson regression, which precludes estimation of changes in time trend (i.e., slope), whereas figure 3 B shows the results of segmented Poisson regression, which allows different slopes before and after the intervention.

SE estimates of Poisson regression models are constrained by the “mean equals variance” assumption. This assumption is relaxed by fitting an overdispersed Poisson regression model [ 14 , 16 ]. Allowing overdispersion can affect SE estimates if the Poisson assumption is false without changing estimated regression parameters, producing more valid inferences. Poisson regression and overdispersed Poisson regression result in equal incident rate ratio estimates but different confidence intervals.

Strengths and limitations. Regression allows estimation of associations between the intervention and outcome while controlling for potential confounders, which is particularly important in nonrandomized quasi-experimental studies ( table 2 ). Segmented regression models estimate changes in mean outcome levels (i.e., intercepts) and trends (i.e., slopes), unlike standard regression models. However, some limitations previously discussed with 2-group tests remain. Specifically, independence between individuals and time periods is assumed. Additionally, regression analysis, in contrast to 2-group tests, requires data from multiple pre- and postintervention time intervals to estimate the slope. General guidelines suggest the use of at least 10 observations per model parameter to avoid overfitting [ 17 ]. The models in figures 2B and 3B contained 5 parameters; thus, they should be used only for studies with at least 50 total observations (in our example, months). For intervention studies, data from at least 10 observations before and after the intervention should be used. However, using at least 24 observations (in our example, 12 months before and after the intervention) would capture potential seasonal changes. Data from shorter intervals can be used (e.g., biweekly); however, choice of time interval is a compromise between maximizing the number of observations and maintaining sufficient data within each interval to provide interpretable summary measures [ 15 , 18 ]. In SAS, the command PROC GENMOD can estimate Poisson and linear regression models ( table 1 ) [ 19 ].

Time-series analysis consists of advanced statistical techniques that require understanding of regression and correlation. Whereas “interrupted time-series design” refers to studies consisting of equally spaced pre- and postintervention observations, “time-series analysis” refers to statistical methods for analyzing time-series design data. Two-group tests and regression analysis assume that monthly LOS and MRSA infection rates are independent over time. In contrast, time-series analysis estimates regression models while relaxing the independence assumption by estimating the autocorrelation between observations collected at different times (e.g., MRSA infection counts among MICU patients across different periods). To estimate autocorrelation, a correlation model is specified along with the regression model, resulting in more accurate SE estimates and improved statistical inference.

Continuous outcomes. Time-series analysis accommodates the previously discussed regression models; however, the challenge is how to correctly model correlation. In linear regression, monthly LOS measurements are assumed to be independent. However, autocorrelation may take one of several forms. For example, if correlation between 2 observations gradually decreases as time between them increases (e.g., correlation between months 1 and 2 is 0.5, correlation between months 1 and 3 is 0.25, and correlation between months 1 and 4 is 0.12), autocorrelation is likely autoregressive. However, if autocorrelation between 2 observations is initially strong but abruptly decreases to ∼0 (e.g., correlation between months 1 and 2 is 0.5 and correlation between months 1 and 3 is 0.05), a moving-average model is more appropriate. Occasionally, autocorrelation is strong for observations close in time and then sharply decreases to a nonzero level after some time threshold. In this case, autoregressive or moving-average models would be inadequate, and autoregressive moving-average (ARMA) models should be used. When correlation between observations does not decrease with duration of time, autoregressive, integrated, moving-average (ARIMA) models may be appropriate. In SAS, PROC AUTOREG estimates autoregressive models, and PROC ARIMA estimates autoregressive, moving-average, ARMA, and ARIMA models.

Count outcomes. Although most time-series software assume that outcomes are normally distributed, methods for Poisson counts are available [ 20 , 21 , 22–23 ]. One approach is to transform counts into monthly rates and use time-series methods for normal data (rates are approximately normally distributed if they are based on large numbers). In addition, Autoregressive [ 22 , 23 ], moving-average [ 21 ], and ARMA [ 20 ] models have been extended for generalized linear models (including Poisson models), called generalized ARMA models. The “garma” command in the R software library VGAM estimates generalized ARMA models [ 24 ].

Strengths and limitations. Time-series methods estimate dependence (i.e., correlation) between observations over time, lessening a common threat to valid inferences. They also accommodate segmented models. Thus, time-series methods generalize regression by relaxing the assumption of independent observations. However, the large data requirements often preclude its use. A general guideline is having ∼50 time points (e.g., 3 years of monthly preintervention data and 1 year of monthly postintervention data) to estimate complex correlation structures [ 25 ]. If fewer observations are available, only simple correlation structures can be reliably estimated [ 15 ].

Another limitation of time-series analysis is difficulty in building and interpreting correlation models. Several technical resources are available to guide analysts [ 26 , 27–28 ]. Review articles [ 25 , 29 , 30 ] and biomedical examples are also available [ 18 , 31 , 32 ]. Bootstrapping circumvents the problem of specifying and estimating an autocorrelation model. Bootstrap SEs can be calculated by estimating regression parameters assuming independence (i.e., linear or Poisson regression). Resulting SEs account for autocorrelation by sampling the data multiple (e.g., 1000 times) with replacement and estimating the parameters with each sample [ 33 ]. Thus, the bootstrap with regression is an alternative to time-series analysis when too few time intervals are observed.

Each method can easily accommodate comparison with a nonequivalent control group, a preferred epidemiological quasi-experimental design, because regression to the mean and maturation effects are common threats in these studies [ 1 , 7 ]. In our example, the intervention could be implemented in the MICU, and the nonequivalent control group could be the surgical intensive care unit. A 2-group t test would then compare changes in the mean LOS in the MICU and surgical intensive care unit (mean LOS after the intervention minus mean LOS before the intervention). Regression analysis (e.g., linear and Poisson) controlling for confounding variables can be performed by fitting separate trends for the MICU and surgical intensive care unit and comparing differences in changes in levels (i.e., intercepts) and trends (i.e., slopes) between the 2 units ( figure 4 ). In our example, the MRSA infection rate in the MICU decreases by 0.8 cases per 1000 person-days immediately on implementation of the intervention, suggesting a large impact of the intervention. However, the MRSA infection rate in the surgical intensive care unit decreases by 0.6 cases per 1000 person-days, suggesting that the decrease in the MRSA infection rate is partially attributable to nonintervention factors, which could not have been identified without a control group. Hence, including a control group is recommended to identify the true impact of an intervention.

Segmented Poisson regression analysis of interrupted time-series methicillin-resistant Staphylococcus aureus (MRSA) infection data, comparing infection rates in the medical intensive care unit (MICU; intervention group) and surgical intensive care unit (SICU; control group) before and after the intervention (implemented at month 36). The reduction of 0.6 infections per 1000 person-days in the SICU suggests that the reduction of 0.8 infections per 1000 person-days in the MICU was not solely due to the intervention.

In summary, 2-group tests, regression analysis, and time-series analysis can accommodate interrupted time-series quasi-experimental data. However, statistical validity depends on using appropriate methods for the study question, meeting data requirements, and verifying modeling assumptions. This last step requires premodeling exploratory data analysis and postmodeling diagnostics not addressed here [ 14 , 17 , 26 , 27 ].

Obtaining high-quality results depends on performing a well-designed study, because statistics cannot correct for a poor initial design [ 1 , 7 , 34 ], nor can they compensate for poor reporting of methods [ 5 , 6 ]. Results from analyses can only provide valid inference on the level of intervention. We provide guidelines of minimal data requirements for using each statistical method ( table 2 ). However, larger sample sizes may be needed to obtain a desired precision for estimating measures of association (e.g., mean difference or rate ratio) or power for statistical tests. A simulation study can determine required sample size using model-generated data analyzed with an appropriate method [ 35 ]. Investigators are encouraged to report sample size calculations in addition to statistical analysis methods [ 5 , 6 ]. Analyzing quasi-experimental data is challenging; therefore, we recommend collaboration between investigators, epidemiologists, and statisticians.

Financial support. National Institute of Health (grants R37 AG09901, 1 R01 AI6085901A1, and P30 AG028747-01 to M.S.; P60 AG12583 to R.R.M.; and institutional grant 1K12RR023250-01 to J.P.F.), Centers for Disease Control and Prevention (grant 1 R01 CI000369-01 to A.D.H. and E.N.P.), and Department of Veterans Affairs Health Services Research and Development Service (grants IIR 04-123-2 and Level 2 Advanced Career Development Award to E.N.P.).

Potential conflicts of interest. All authors: no conflicts.

Google Scholar

Google Preview

• drug resistance, microbial
• length of stay
• pathogenic organism
• antimicrobials
• methicillin-resistant staphylococcus aureus infections

Email alerts

More on this topic, related articles in pubmed, citing articles via, looking for your next opportunity.

• Recommend to your Library

Affiliations

• Online ISSN 1537-6591
• Print ISSN 1058-4838
• Copyright © 2024 Infectious Diseases Society of America
• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

7.3 Quasi-Experimental Research

Learning objectives.

• Explain what quasi-experimental research is and distinguish it clearly from both experimental and correlational research.
• Describe three different types of quasi-experimental research designs (nonequivalent groups, pretest-posttest, and interrupted time series) and identify examples of each one.

The prefix quasi means “resembling.” Thus quasi-experimental research is research that resembles experimental research but is not true experimental research. Although the independent variable is manipulated, participants are not randomly assigned to conditions or orders of conditions (Cook & Campbell, 1979). Because the independent variable is manipulated before the dependent variable is measured, quasi-experimental research eliminates the directionality problem. But because participants are not randomly assigned—making it likely that there are other differences between conditions—quasi-experimental research does not eliminate the problem of confounding variables. In terms of internal validity, therefore, quasi-experiments are generally somewhere between correlational studies and true experiments.

Quasi-experiments are most likely to be conducted in field settings in which random assignment is difficult or impossible. They are often conducted to evaluate the effectiveness of a treatment—perhaps a type of psychotherapy or an educational intervention. There are many different kinds of quasi-experiments, but we will discuss just a few of the most common ones here.

Nonequivalent Groups Design

Recall that when participants in a between-subjects experiment are randomly assigned to conditions, the resulting groups are likely to be quite similar. In fact, researchers consider them to be equivalent. When participants are not randomly assigned to conditions, however, the resulting groups are likely to be dissimilar in some ways. For this reason, researchers consider them to be nonequivalent. A nonequivalent groups design , then, is a between-subjects design in which participants have not been randomly assigned to conditions.

Imagine, for example, a researcher who wants to evaluate a new method of teaching fractions to third graders. One way would be to conduct a study with a treatment group consisting of one class of third-grade students and a control group consisting of another class of third-grade students. This would be a nonequivalent groups design because the students are not randomly assigned to classes by the researcher, which means there could be important differences between them. For example, the parents of higher achieving or more motivated students might have been more likely to request that their children be assigned to Ms. Williams’s class. Or the principal might have assigned the “troublemakers” to Mr. Jones’s class because he is a stronger disciplinarian. Of course, the teachers’ styles, and even the classroom environments, might be very different and might cause different levels of achievement or motivation among the students. If at the end of the study there was a difference in the two classes’ knowledge of fractions, it might have been caused by the difference between the teaching methods—but it might have been caused by any of these confounding variables.

Of course, researchers using a nonequivalent groups design can take steps to ensure that their groups are as similar as possible. In the present example, the researcher could try to select two classes at the same school, where the students in the two classes have similar scores on a standardized math test and the teachers are the same sex, are close in age, and have similar teaching styles. Taking such steps would increase the internal validity of the study because it would eliminate some of the most important confounding variables. But without true random assignment of the students to conditions, there remains the possibility of other important confounding variables that the researcher was not able to control.

Pretest-Posttest Design

In a pretest-posttest design , the dependent variable is measured once before the treatment is implemented and once after it is implemented. Imagine, for example, a researcher who is interested in the effectiveness of an antidrug education program on elementary school students’ attitudes toward illegal drugs. The researcher could measure the attitudes of students at a particular elementary school during one week, implement the antidrug program during the next week, and finally, measure their attitudes again the following week. The pretest-posttest design is much like a within-subjects experiment in which each participant is tested first under the control condition and then under the treatment condition. It is unlike a within-subjects experiment, however, in that the order of conditions is not counterbalanced because it typically is not possible for a participant to be tested in the treatment condition first and then in an “untreated” control condition.

If the average posttest score is better than the average pretest score, then it makes sense to conclude that the treatment might be responsible for the improvement. Unfortunately, one often cannot conclude this with a high degree of certainty because there may be other explanations for why the posttest scores are better. One category of alternative explanations goes under the name of history . Other things might have happened between the pretest and the posttest. Perhaps an antidrug program aired on television and many of the students watched it, or perhaps a celebrity died of a drug overdose and many of the students heard about it. Another category of alternative explanations goes under the name of maturation . Participants might have changed between the pretest and the posttest in ways that they were going to anyway because they are growing and learning. If it were a yearlong program, participants might become less impulsive or better reasoners and this might be responsible for the change.

Another alternative explanation for a change in the dependent variable in a pretest-posttest design is regression to the mean . This refers to the statistical fact that an individual who scores extremely on a variable on one occasion will tend to score less extremely on the next occasion. For example, a bowler with a long-term average of 150 who suddenly bowls a 220 will almost certainly score lower in the next game. Her score will “regress” toward her mean score of 150. Regression to the mean can be a problem when participants are selected for further study because of their extreme scores. Imagine, for example, that only students who scored especially low on a test of fractions are given a special training program and then retested. Regression to the mean all but guarantees that their scores will be higher even if the training program has no effect. A closely related concept—and an extremely important one in psychological research—is spontaneous remission . This is the tendency for many medical and psychological problems to improve over time without any form of treatment. The common cold is a good example. If one were to measure symptom severity in 100 common cold sufferers today, give them a bowl of chicken soup every day, and then measure their symptom severity again in a week, they would probably be much improved. This does not mean that the chicken soup was responsible for the improvement, however, because they would have been much improved without any treatment at all. The same is true of many psychological problems. A group of severely depressed people today is likely to be less depressed on average in 6 months. In reviewing the results of several studies of treatments for depression, researchers Michael Posternak and Ivan Miller found that participants in waitlist control conditions improved an average of 10 to 15% before they received any treatment at all (Posternak & Miller, 2001). Thus one must generally be very cautious about inferring causality from pretest-posttest designs.

Does Psychotherapy Work?

Early studies on the effectiveness of psychotherapy tended to use pretest-posttest designs. In a classic 1952 article, researcher Hans Eysenck summarized the results of 24 such studies showing that about two thirds of patients improved between the pretest and the posttest (Eysenck, 1952). But Eysenck also compared these results with archival data from state hospital and insurance company records showing that similar patients recovered at about the same rate without receiving psychotherapy. This suggested to Eysenck that the improvement that patients showed in the pretest-posttest studies might be no more than spontaneous remission. Note that Eysenck did not conclude that psychotherapy was ineffective. He merely concluded that there was no evidence that it was, and he wrote of “the necessity of properly planned and executed experimental studies into this important field” (p. 323). You can read the entire article here:

http://psychclassics.yorku.ca/Eysenck/psychotherapy.htm

Fortunately, many other researchers took up Eysenck’s challenge, and by 1980 hundreds of experiments had been conducted in which participants were randomly assigned to treatment and control conditions, and the results were summarized in a classic book by Mary Lee Smith, Gene Glass, and Thomas Miller (Smith, Glass, & Miller, 1980). They found that overall psychotherapy was quite effective, with about 80% of treatment participants improving more than the average control participant. Subsequent research has focused more on the conditions under which different types of psychotherapy are more or less effective.

In a classic 1952 article, researcher Hans Eysenck pointed out the shortcomings of the simple pretest-posttest design for evaluating the effectiveness of psychotherapy.

Wikimedia Commons – CC BY-SA 3.0.

Interrupted Time Series Design

A variant of the pretest-posttest design is the interrupted time-series design . A time series is a set of measurements taken at intervals over a period of time. For example, a manufacturing company might measure its workers’ productivity each week for a year. In an interrupted time series-design, a time series like this is “interrupted” by a treatment. In one classic example, the treatment was the reduction of the work shifts in a factory from 10 hours to 8 hours (Cook & Campbell, 1979). Because productivity increased rather quickly after the shortening of the work shifts, and because it remained elevated for many months afterward, the researcher concluded that the shortening of the shifts caused the increase in productivity. Notice that the interrupted time-series design is like a pretest-posttest design in that it includes measurements of the dependent variable both before and after the treatment. It is unlike the pretest-posttest design, however, in that it includes multiple pretest and posttest measurements.

Figure 7.5 “A Hypothetical Interrupted Time-Series Design” shows data from a hypothetical interrupted time-series study. The dependent variable is the number of student absences per week in a research methods course. The treatment is that the instructor begins publicly taking attendance each day so that students know that the instructor is aware of who is present and who is absent. The top panel of Figure 7.5 “A Hypothetical Interrupted Time-Series Design” shows how the data might look if this treatment worked. There is a consistently high number of absences before the treatment, and there is an immediate and sustained drop in absences after the treatment. The bottom panel of Figure 7.5 “A Hypothetical Interrupted Time-Series Design” shows how the data might look if this treatment did not work. On average, the number of absences after the treatment is about the same as the number before. This figure also illustrates an advantage of the interrupted time-series design over a simpler pretest-posttest design. If there had been only one measurement of absences before the treatment at Week 7 and one afterward at Week 8, then it would have looked as though the treatment were responsible for the reduction. The multiple measurements both before and after the treatment suggest that the reduction between Weeks 7 and 8 is nothing more than normal week-to-week variation.

Figure 7.5 A Hypothetical Interrupted Time-Series Design

The top panel shows data that suggest that the treatment caused a reduction in absences. The bottom panel shows data that suggest that it did not.

Combination Designs

A type of quasi-experimental design that is generally better than either the nonequivalent groups design or the pretest-posttest design is one that combines elements of both. There is a treatment group that is given a pretest, receives a treatment, and then is given a posttest. But at the same time there is a control group that is given a pretest, does not receive the treatment, and then is given a posttest. The question, then, is not simply whether participants who receive the treatment improve but whether they improve more than participants who do not receive the treatment.

Imagine, for example, that students in one school are given a pretest on their attitudes toward drugs, then are exposed to an antidrug program, and finally are given a posttest. Students in a similar school are given the pretest, not exposed to an antidrug program, and finally are given a posttest. Again, if students in the treatment condition become more negative toward drugs, this could be an effect of the treatment, but it could also be a matter of history or maturation. If it really is an effect of the treatment, then students in the treatment condition should become more negative than students in the control condition. But if it is a matter of history (e.g., news of a celebrity drug overdose) or maturation (e.g., improved reasoning), then students in the two conditions would be likely to show similar amounts of change. This type of design does not completely eliminate the possibility of confounding variables, however. Something could occur at one of the schools but not the other (e.g., a student drug overdose), so students at the first school would be affected by it while students at the other school would not.

Finally, if participants in this kind of design are randomly assigned to conditions, it becomes a true experiment rather than a quasi experiment. In fact, it is the kind of experiment that Eysenck called for—and that has now been conducted many times—to demonstrate the effectiveness of psychotherapy.

Key Takeaways

• Quasi-experimental research involves the manipulation of an independent variable without the random assignment of participants to conditions or orders of conditions. Among the important types are nonequivalent groups designs, pretest-posttest, and interrupted time-series designs.
• Quasi-experimental research eliminates the directionality problem because it involves the manipulation of the independent variable. It does not eliminate the problem of confounding variables, however, because it does not involve random assignment to conditions. For these reasons, quasi-experimental research is generally higher in internal validity than correlational studies but lower than true experiments.
• Practice: Imagine that two college professors decide to test the effect of giving daily quizzes on student performance in a statistics course. They decide that Professor A will give quizzes but Professor B will not. They will then compare the performance of students in their two sections on a common final exam. List five other variables that might differ between the two sections that could affect the results.

Discussion: Imagine that a group of obese children is recruited for a study in which their weight is measured, then they participate for 3 months in a program that encourages them to be more active, and finally their weight is measured again. Explain how each of the following might affect the results:

• regression to the mean
• spontaneous remission

Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design & analysis issues in field settings . Boston, MA: Houghton Mifflin.

Eysenck, H. J. (1952). The effects of psychotherapy: An evaluation. Journal of Consulting Psychology, 16 , 319–324.

Posternak, M. A., & Miller, I. (2001). Untreated short-term course of major depression: A meta-analysis of studies using outcomes from studies using wait-list control groups. Journal of Affective Disorders, 66 , 139–146.

Smith, M. L., Glass, G. V., & Miller, T. I. (1980). The benefits of psychotherapy . Baltimore, MD: Johns Hopkins University Press.

Research Methods in Psychology Copyright © 2016 by University of Minnesota is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License , except where otherwise noted.

Introduction to Experimental and Quasi-Experimental Design

• First Online: 26 April 2022

Cite this chapter

• Melissa Whatley   ORCID: orcid.org/0000-0002-7073-6772 2  

Part of the book series: Springer Texts in Education ((SPTE))

678 Accesses

This chapter introduces readers to main concepts in experimental and quasi-experimental design. First, randomized control trials are introduced as the primary example of experimental design. Next, nonexperimental contexts, and particularly the use of propensity score matching to approximate the conditions of randomized control trials, are described. Finally, this chapter introduces two quasi-experimental design that are particularly useful in international education research: regression discontinuity and difference-in-differences.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime
• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

The extent to which randomization is possible in these scenarios is an important consideration that we will discuss later in this chapter.

Note that here I report only a subset of Meegan and Kashima’s (2010) results as I focus on only one of their two experimental conditions.

Advanced readers who are interested in these alternatives can read more about exact matching (Rubin, 1973 ), genetic matching (Diamond & Sekhon, 2013 ), or coarsened exact matching (Iacus, King, & Porro, 2011 ).

Sometimes, researchers are especially interested in estimating the impact of the treatment on the treated units (the average treatment effect on the treated [ATT]) rather than the average treatment effect for all units in a dataset (ATE). In this case, no treatment units are dropped from the dataset, even though some control units are.

A discussion of propensity score matching approaches is beyond the scope of this book, but several of the suggested readings at the end of this chapter delve deeper into matching techniques and their benefits and drawbacks.

Note that Iriondo (2020) estimates the impact of Erasmus participation on employment and salary using two different datasets. The results presented in this section correspond to his results for the Labor Insertion Survey.

Regression discontinuity and difference-in-differences are not the only two quasi-experimental designs available to researchers (other quasi-experimental designs include instrumental variable and time-series analyses). However, they are two of the more common designs and are also possibly the most useful to individuals conducting research in international education.

Recommended Reading

A deeper dive.

Cunningham, S. (2021). Causal inference: The mixtape . Yale University Press.

Google Scholar  

DesJardins, S. L., & Flaster, A. (2013). Nonexperimental designs and causal analyses of college access, persistence, and completion. In L. W. Perna & A. Jones (Eds.), The state of college access and completion: Improving college success for students from underrepresented groups (pp. 190–207). Routledge.

Diamond, A., & Sekhon, J. S. (2013). Genetic matching for estimating causal effects: A general multivariate matching method for achieving balance in observational studies. Review of Economics and Statistics, 95 (3), 932–945.

Article   Google Scholar  

Furquim, F., Corral, D., & Hillman, N. (2020). A primer for interpreting and designing difference-in-difference studies in higher education research. In L. Perna (Ed.), Higher education: Handbook of theory and research (Vol. 35, pp. 2–53).

Iacus, S. M., King, G., & Porro, G. (2011). Multivariate matching methods that are monotonic imbalance bounding. Journal of the American Statistical Association, 106 (493), 345–361.

Murnane, R. J., & Willett, J. B. (2011). Methods matter: Improving causal inference in educational and social science research. Oxford University Press.

Reynolds, C. L., & DesJardins, S. L. (2009). The use of matching methods in higher education research: Answering whether attendance at a 2-year institution results in differences in educational attainment. In  Higher education: Handbook of theory and research  (pp. 47–97). Springer.

Rosenbaum, P. R., & Rubin, D. B. (1985). Constructing a control group using multivariate matched sampling methods that incorporate the propensity score. The American Statistician, 39 (1), 33–38.

Rubin, D. B. (1973). The use of matched sampling and regression adjustment to remove bias in observational studies.  Biometrics , (Vol. 29, pp.185–203).

Additional Examples

d’Hombres, B., & Schnepf, S. V. (2021). International mobility of students in Italy and the UK: Does it pay off and for whom? Higher Education . https://doi.org/10.1007/s10734-020-00631-1

Dicks, A., & Lancee, B. (2018). Double disadvantage in school? Children of immigrants and the relative age effect: A regression discontinuity design based on the month of birth. European Sociological Review, 34 (3), 319–333.

Marini, G., & Yang, L. (2021). Globally bred Chinese talents returning home: An analysis of a reverse brain-drain flagship policy. Science and Public Policy . https://doi.org/10.1093/scipol/scab021

Monogan, J. E., & Doctor, A. C. (2017). Immigration politics and partisan realignment: California, Texas, and the 1994 election. State Politics & Policy Quarterly, 17 (1), 3–23.

Download references

Author information

Authors and affiliations.

School for International Training, Brattleboro, VT, USA

Melissa Whatley

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Melissa Whatley .

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Whatley, M. (2022). Introduction to Experimental and Quasi-Experimental Design. In: Introduction to Quantitative Analysis for International Educators. Springer Texts in Education. Springer, Cham. https://doi.org/10.1007/978-3-030-93831-4_9

Download citation

DOI : https://doi.org/10.1007/978-3-030-93831-4_9

Published : 26 April 2022

Publisher Name : Springer, Cham

Print ISBN : 978-3-030-93830-7

Online ISBN : 978-3-030-93831-4

eBook Packages : Education Education (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

Experimental vs Quasi-Experimental Design: Which to Choose?

Here’s a table that summarizes the similarities and differences between an experimental and a quasi-experimental study design:

What is a quasi-experimental design?

A quasi-experimental design is a non-randomized study design used to evaluate the effect of an intervention. The intervention can be a training program, a policy change or a medical treatment.

Unlike a true experiment, in a quasi-experimental study the choice of who gets the intervention and who doesn’t is not randomized. Instead, the intervention can be assigned to participants according to their choosing or that of the researcher, or by using any method other than randomness.

Having a control group is not required, but if present, it provides a higher level of evidence for the relationship between the intervention and the outcome.

(for more information, I recommend my other article: Understand Quasi-Experimental Design Through an Example ) .

Examples of quasi-experimental designs include:

• One-Group Posttest Only Design
• Static-Group Comparison Design
• One-Group Pretest-Posttest Design
• Separate-Sample Pretest-Posttest Design

What is an experimental design?

An experimental design is a randomized study design used to evaluate the effect of an intervention. In its simplest form, the participants will be randomly divided into 2 groups:

• A treatment group: where participants receive the new intervention which effect we want to study.
• A control or comparison group: where participants do not receive any intervention at all (or receive some standard intervention).

Randomization ensures that each participant has the same chance of receiving the intervention. Its objective is to equalize the 2 groups, and therefore, any observed difference in the study outcome afterwards will only be attributed to the intervention – i.e. it removes confounding.

(for more information, I recommend my other article: Purpose and Limitations of Random Assignment ).

Examples of experimental designs include:

• Posttest-Only Control Group Design
• Pretest-Posttest Control Group Design
• Solomon Four-Group Design
• Matched Pairs Design
• Randomized Block Design

When to choose an experimental design over a quasi-experimental design?

Although many statistical techniques can be used to deal with confounding in a quasi-experimental study, in practice, randomization is still the best tool we have to study causal relationships.

Another problem with quasi-experiments is the natural progression of the disease or the condition under study — When studying the effect of an intervention over time, one should consider natural changes because these can be mistaken with changes in outcome that are caused by the intervention. Having a well-chosen control group helps dealing with this issue.

So, if losing the element of randomness seems like an unwise step down in the hierarchy of evidence, why would we ever want to do it?

This is what we’re going to discuss next.

When to choose a quasi-experimental design over a true experiment?

The issue with randomness is that it cannot be always achievable.

So here are some cases where using a quasi-experimental design makes more sense than using an experimental one:

• If being in one group is believed to be harmful for the participants , either because the intervention is harmful (ex. randomizing people to smoking), or the intervention has a questionable efficacy, or on the contrary it is believed to be so beneficial that it would be malevolent to put people in the control group (ex. randomizing people to receiving an operation).
• In cases where interventions act on a group of people in a given location , it becomes difficult to adequately randomize subjects (ex. an intervention that reduces pollution in a given area).
• When working with small sample sizes , as randomized controlled trials require a large sample size to account for heterogeneity among subjects (i.e. to evenly distribute confounding variables between the intervention and control groups).

Further reading

• Statistical Software Popularity in 40,582 Research Papers
• Checking the Popularity of 125 Statistical Tests and Models
• Objectives of Epidemiology (With Examples)
• 12 Famous Epidemiologists and Why

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

Part 3: Using quantitative methods

13. Experimental design

Chapter outline.

• What is an experiment and when should you use one? (8 minute read)
• True experimental designs (7 minute read)
• Quasi-experimental designs (8 minute read)
• Non-experimental designs (5 minute read)
• Critical, ethical, and critical considerations  (5 minute read)

Content warning : examples in this chapter contain references to non-consensual research in Western history, including experiments conducted during the Holocaust and on African Americans (section 13.6).

13.1 What is an experiment and when should you use one?

Learning objectives.

Learners will be able to…

• Identify the characteristics of a basic experiment
• Describe causality in experimental design
• Discuss the relationship between dependent and independent variables in experiments
• Explain the links between experiments and generalizability of results
• Describe advantages and disadvantages of experimental designs

The basics of experiments

The first experiment I can remember using was for my fourth grade science fair. I wondered if latex- or oil-based paint would hold up to sunlight better. So, I went to the hardware store and got a few small cans of paint and two sets of wooden paint sticks. I painted one with oil-based paint and the other with latex-based paint of different colors and put them in a sunny spot in the back yard. My hypothesis was that the oil-based paint would fade the most and that more fading would happen the longer I left the paint sticks out. (I know, it’s obvious, but I was only 10.)

I checked in on the paint sticks every few days for a month and wrote down my observations. The first part of my hypothesis ended up being wrong—it was actually the latex-based paint that faded the most. But the second part was right, and the paint faded more and more over time. This is a simple example, of course—experiments get a heck of a lot more complex than this when we’re talking about real research.

Merriam-Webster defines an experiment   as “an operation or procedure carried out under controlled conditions in order to discover an unknown effect or law, to test or establish a hypothesis, or to illustrate a known law.” Each of these three components of the definition will come in handy as we go through the different types of experimental design in this chapter. Most of us probably think of the physical sciences when we think of experiments, and for good reason—these experiments can be pretty flashy! But social science and psychological research follow the same scientific methods, as we’ve discussed in this book.

As the video discusses, experiments can be used in social sciences just like they can in physical sciences. It makes sense to use an experiment when you want to determine the cause of a phenomenon with as much accuracy as possible. Some types of experimental designs do this more precisely than others, as we’ll see throughout the chapter. If you’ll remember back to Chapter 11  and the discussion of validity, experiments are the best way to ensure internal validity, or the extent to which a change in your independent variable causes a change in your dependent variable.

Experimental designs for research projects are most appropriate when trying to uncover or test a hypothesis about the cause of a phenomenon, so they are best for explanatory research questions. As we’ll learn throughout this chapter, different circumstances are appropriate for different types of experimental designs. Each type of experimental design has advantages and disadvantages, and some are better at controlling the effect of extraneous variables —those variables and characteristics that have an effect on your dependent variable, but aren’t the primary variable whose influence you’re interested in testing. For example, in a study that tries to determine whether aspirin lowers a person’s risk of a fatal heart attack, a person’s race would likely be an extraneous variable because you primarily want to know the effect of aspirin.

In practice, many types of experimental designs can be logistically challenging and resource-intensive. As practitioners, the likelihood that we will be involved in some of the types of experimental designs discussed in this chapter is fairly low. However, it’s important to learn about these methods, even if we might not ever use them, so that we can be thoughtful consumers of research that uses experimental designs.

While we might not use all of these types of experimental designs, many of us will engage in evidence-based practice during our time as social workers. A lot of research developing evidence-based practice, which has a strong emphasis on generalizability, will use experimental designs. You’ve undoubtedly seen one or two in your literature search so far.

The logic of experimental design

How do we know that one phenomenon causes another? The complexity of the social world in which we practice and conduct research means that causes of social problems are rarely cut and dry. Uncovering explanations for social problems is key to helping clients address them, and experimental research designs are one road to finding answers.

As you read about in Chapter 8 (and as we’ll discuss again in Chapter 15 ), just because two phenomena are related in some way doesn’t mean that one causes the other. Ice cream sales increase in the summer, and so does the rate of violent crime; does that mean that eating ice cream is going to make me murder someone? Obviously not, because ice cream is great. The reality of that relationship is far more complex—it could be that hot weather makes people more irritable and, at times, violent, while also making people want ice cream. More likely, though, there are other social factors not accounted for in the way we just described this relationship.

Experimental designs can help clear up at least some of this fog by allowing researchers to isolate the effect of interventions on dependent variables by controlling extraneous variables . In true experimental design (discussed in the next section) and some quasi-experimental designs, researchers accomplish this w ith the control group and the experimental group . (The experimental group is sometimes called the “treatment group,” but we will call it the experimental group in this chapter.) The control group does not receive the intervention you are testing (they may receive no intervention or what is known as “treatment as usual”), while the experimental group does. (You will hopefully remember our earlier discussion of control variables in Chapter 8 —conceptually, the use of the word “control” here is the same.)

In a well-designed experiment, your control group should look almost identical to your experimental group in terms of demographics and other relevant factors. What if we want to know the effect of CBT on social anxiety, but we have learned in prior research that men tend to have a more difficult time overcoming social anxiety? We would want our control and experimental groups to have a similar gender mix because it would limit the effect of gender on our results, since ostensibly, both groups’ results would be affected by gender in the same way. If your control group has 5 women, 6 men, and 4 non-binary people, then your experimental group should be made up of roughly the same gender balance to help control for the influence of gender on the outcome of your intervention. (In reality, the groups should be similar along other dimensions, as well, and your group will likely be much larger.) The researcher will use the same outcome measures for both groups and compare them, and assuming the experiment was designed correctly, get a pretty good answer about whether the intervention had an effect on social anxiety.

You will also hear people talk about comparison groups , which are similar to control groups. The primary difference between the two is that a control group is populated using random assignment, but a comparison group is not. Random assignment entails using a random process to decide which participants are put into the control or experimental group (which participants receive an intervention and which do not). By randomly assigning participants to a group, you can reduce the effect of extraneous variables on your research because there won’t be a systematic difference between the groups.

Do not confuse random assignment with random sampling. Random sampling is a method for selecting a sample from a population, and is rarely used in psychological research. Random assignment is a method for assigning participants in a sample to the different conditions, and it is an important element of all experimental research in psychology and other related fields. Random sampling also helps a great deal with generalizability , whereas random assignment increases internal validity .

We have already learned about internal validity in Chapter 11 . The use of an experimental design will bolster internal validity since it works to isolate causal relationships. As we will see in the coming sections, some types of experimental design do this more effectively than others. It’s also worth considering that true experiments, which most effectively show causality , are often difficult and expensive to implement. Although other experimental designs aren’t perfect, they still produce useful, valid evidence and may be more feasible to carry out.

Key Takeaways

• Experimental designs are useful for establishing causality, but some types of experimental design do this better than others.
• Experiments help researchers isolate the effect of the independent variable on the dependent variable by controlling for the effect of extraneous variables .
• Experiments use a control/comparison group and an experimental group to test the effects of interventions. These groups should be as similar to each other as possible in terms of demographics and other relevant factors.
• True experiments have control groups with randomly assigned participants, while other types of experiments have comparison groups to which participants are not randomly assigned.
• Think about the research project you’ve been designing so far. How might you use a basic experiment to answer your question? If your question isn’t explanatory, try to formulate a new explanatory question and consider the usefulness of an experiment.
• Why is establishing a simple relationship between two variables not indicative of one causing the other?

13.2 True experimental design

• Describe a true experimental design in social work research
• Understand the different types of true experimental designs
• Determine what kinds of research questions true experimental designs are suited for
• Discuss advantages and disadvantages of true experimental designs

True experimental design , often considered to be the “gold standard” in research designs, is thought of as one of the most rigorous of all research designs. In this design, one or more independent variables are manipulated by the researcher (as treatments), subjects are randomly assigned to different treatment levels (random assignment), and the results of the treatments on outcomes (dependent variables) are observed. The unique strength of experimental research is its internal validity and its ability to establish ( causality ) through treatment manipulation, while controlling for the effects of extraneous variable. Sometimes the treatment level is no treatment, while other times it is simply a different treatment than that which we are trying to evaluate. For example, we might have a control group that is made up of people who will not receive any treatment for a particular condition. Or, a control group could consist of people who consent to treatment with DBT when we are testing the effectiveness of CBT.

As we discussed in the previous section, a true experiment has a control group with participants randomly assigned , and an experimental group . This is the most basic element of a true experiment. The next decision a researcher must make is when they need to gather data during their experiment. Do they take a baseline measurement and then a measurement after treatment, or just a measurement after treatment, or do they handle measurement another way? Below, we’ll discuss the three main types of true experimental designs. There are sub-types of each of these designs, but here, we just want to get you started with some of the basics.

Using a true experiment in social work research is often pretty difficult, since as I mentioned earlier, true experiments can be quite resource intensive. True experiments work best with relatively large sample sizes, and random assignment, a key criterion for a true experimental design, is hard (and unethical) to execute in practice when you have people in dire need of an intervention. Nonetheless, some of the strongest evidence bases are built on true experiments.

For the purposes of this section, let’s bring back the example of CBT for the treatment of social anxiety. We have a group of 500 individuals who have agreed to participate in our study, and we have randomly assigned them to the control and experimental groups. The folks in the experimental group will receive CBT, while the folks in the control group will receive more unstructured, basic talk therapy. These designs, as we talked about above, are best suited for explanatory research questions.

Before we get started, take a look at the table below. When explaining experimental research designs, we often use diagrams with abbreviations to visually represent the experiment. Table 13.1 starts us off by laying out what each of the abbreviations mean.

Pretest and post-test control group design

In pretest and post-test control group design , participants are given a pretest of some kind to measure their baseline state before their participation in an intervention. In our social anxiety experiment, we would have participants in both the experimental and control groups complete some measure of social anxiety—most likely an established scale and/or a structured interview—before they start their treatment. As part of the experiment, we would have a defined time period during which the treatment would take place (let’s say 12 weeks, just for illustration). At the end of 12 weeks, we would give both groups the same measure as a post-test .

In the diagram, RA (random assignment group A) is the experimental group and RB is the control group. O 1 denotes the pre-test, X e denotes the experimental intervention, and O 2 denotes the post-test. Let’s look at this diagram another way, using the example of CBT for social anxiety that we’ve been talking about.

In a situation where the control group received treatment as usual instead of no intervention, the diagram would look this way, with X i denoting treatment as usual (Figure 13.3).

Hopefully, these diagrams provide you a visualization of how this type of experiment establishes time order , a key component of a causal relationship. Did the change occur after the intervention? Assuming there is a change in the scores between the pretest and post-test, we would be able to say that yes, the change did occur after the intervention. Causality can’t exist if the change happened before the intervention—this would mean that something else led to the change, not our intervention.

Post-test only control group design

Post-test only control group design involves only giving participants a post-test, just like it sounds (Figure 13.4).

But why would you use this design instead of using a pretest/post-test design? One reason could be the testing effect that can happen when research participants take a pretest. In research, the testing effect refers to “measurement error related to how a test is given; the conditions of the testing, including environmental conditions; and acclimation to the test itself” (Engel & Schutt, 2017, p. 444) [1] (When we say “measurement error,” all we mean is the accuracy of the way we measure the dependent variable.) Figure 13.4 is a visualization of this type of experiment. The testing effect isn’t always bad in practice—our initial assessments might help clients identify or put into words feelings or experiences they are having when they haven’t been able to do that before. In research, however, we might want to control its effects to isolate a cleaner causal relationship between intervention and outcome.

Going back to our CBT for social anxiety example, we might be concerned that participants would learn about social anxiety symptoms by virtue of taking a pretest. They might then identify that they have those symptoms on the post-test, even though they are not new symptoms for them. That could make our intervention look less effective than it actually is.

However, without a baseline measurement establishing causality can be more difficult. If we don’t know someone’s state of mind before our intervention, how do we know our intervention did anything at all? Establishing time order is thus a little more difficult. You must balance this consideration with the benefits of this type of design.

Solomon four group design

One way we can possibly measure how much the testing effect might change the results of the experiment is with the Solomon four group design. Basically, as part of this experiment, you have two control groups and two experimental groups. The first pair of groups receives both a pretest and a post-test. The other pair of groups receives only a post-test (Figure 13.5). This design helps address the problem of establishing time order in post-test only control group designs.

For our CBT project, we would randomly assign people to four different groups instead of just two. Groups A and B would take our pretest measures and our post-test measures, and groups C and D would take only our post-test measures. We could then compare the results among these groups and see if they’re significantly different between the folks in A and B, and C and D. If they are, we may have identified some kind of testing effect, which enables us to put our results into full context. We don’t want to draw a strong causal conclusion about our intervention when we have major concerns about testing effects without trying to determine the extent of those effects.

Solomon four group designs are less common in social work research, primarily because of the logistics and resource needs involved. Nonetheless, this is an important experimental design to consider when we want to address major concerns about testing effects.

• True experimental design is best suited for explanatory research questions.
• True experiments require random assignment of participants to control and experimental groups.
• Pretest/post-test research design involves two points of measurement—one pre-intervention and one post-intervention.
• Post-test only research design involves only one point of measurement—post-intervention. It is a useful design to minimize the effect of testing effects on our results.
• Solomon four group research design involves both of the above types of designs, using 2 pairs of control and experimental groups. One group receives both a pretest and a post-test, while the other receives only a post-test. This can help uncover the influence of testing effects.
• Think about a true experiment you might conduct for your research project. Which design would be best for your research, and why?
• What challenges or limitations might make it unrealistic (or at least very complicated!) for you to carry your true experimental design in the real-world as a student researcher?
• What hypothesis(es) would you test using this true experiment?

13.4 Quasi-experimental designs

• Describe a quasi-experimental design in social work research
• Understand the different types of quasi-experimental designs
• Determine what kinds of research questions quasi-experimental designs are suited for
• Discuss advantages and disadvantages of quasi-experimental designs

Quasi-experimental designs are a lot more common in social work research than true experimental designs. Although quasi-experiments don’t do as good a job of giving us robust proof of causality , they still allow us to establish time order , which is a key element of causality. The prefix quasi means “resembling,” so quasi-experimental research is research that resembles experimental research, but is not true experimental research. Nonetheless, given proper research design, quasi-experiments can still provide extremely rigorous and useful results.

There are a few key differences between true experimental and quasi-experimental research. The primary difference between quasi-experimental research and true experimental research is that quasi-experimental research does not involve random assignment to control and experimental groups. Instead, we talk about comparison groups in quasi-experimental research instead. As a result, these types of experiments don’t control the effect of extraneous variables as well as a true experiment.

Quasi-experiments are most likely to be conducted in field settings in which random assignment is difficult or impossible. They are often conducted to evaluate the effectiveness of a treatment—perhaps a type of psychotherapy or an educational intervention.  We’re able to eliminate some threats to internal validity, but we can’t do this as effectively as we can with a true experiment.  Realistically, our CBT-social anxiety project is likely to be a quasi experiment, based on the resources and participant pool we’re likely to have available. 

It’s important to note that not all quasi-experimental designs have a comparison group.  There are many different kinds of quasi-experiments, but we will discuss the three main types below: nonequivalent comparison group designs, time series designs, and ex post facto comparison group designs.

Nonequivalent comparison group design

You will notice that this type of design looks extremely similar to the pretest/post-test design that we discussed in section 13.3. But instead of random assignment to control and experimental groups, researchers use other methods to construct their comparison and experimental groups. A diagram of this design will also look very similar to pretest/post-test design, but you’ll notice we’ve removed the “R” from our groups, since they are not randomly assigned (Figure 13.6).

Researchers using this design select a comparison group that’s as close as possible based on relevant factors to their experimental group. Engel and Schutt (2017) [2] identify two different selection methods:

• Individual matching : Researchers take the time to match individual cases in the experimental group to similar cases in the comparison group. It can be difficult, however, to match participants on all the variables you want to control for.
• Aggregate matching : Instead of trying to match individual participants to each other, researchers try to match the population profile of the comparison and experimental groups. For example, researchers would try to match the groups on average age, gender balance, or median income. This is a less resource-intensive matching method, but researchers have to ensure that participants aren’t choosing which group (comparison or experimental) they are a part of.

As we’ve already talked about, this kind of design provides weaker evidence that the intervention itself leads to a change in outcome. Nonetheless, we are still able to establish time order using this method, and can thereby show an association between the intervention and the outcome. Like true experimental designs, this type of quasi-experimental design is useful for explanatory research questions.

What might this look like in a practice setting? Let’s say you’re working at an agency that provides CBT and other types of interventions, and you have identified a group of clients who are seeking help for social anxiety, as in our earlier example. Once you’ve obtained consent from your clients, you can create a comparison group using one of the matching methods we just discussed. If the group is small, you might match using individual matching, but if it’s larger, you’ll probably sort people by demographics to try to get similar population profiles. (You can do aggregate matching more easily when your agency has some kind of electronic records or database, but it’s still possible to do manually.)

Time series design

Another type of quasi-experimental design is a time series design. Unlike other types of experimental design, time series designs do not have a comparison group. A time series is a set of measurements taken at intervals over a period of time (Figure 13.7). Proper time series design should include at least three pre- and post-intervention measurement points. While there are a few types of time series designs, we’re going to focus on the most common: interrupted time series design.

But why use this method? Here’s an example. Let’s think about elementary student behavior throughout the school year. As anyone with children or who is a teacher knows, kids get very excited and animated around holidays, days off, or even just on a Friday afternoon. This fact might mean that around those times of year, there are more reports of disruptive behavior in classrooms. What if we took our one and only measurement in mid-December? It’s possible we’d see a higher-than-average rate of disruptive behavior reports, which could bias our results if our next measurement is around a time of year students are in a different, less excitable frame of mind. When we take multiple measurements throughout the first half of the school year, we can establish a more accurate baseline for the rate of these reports by looking at the trend over time.

We may want to test the effect of extended recess times in elementary school on reports of disruptive behavior in classrooms. When students come back after the winter break, the school extends recess by 10 minutes each day (the intervention), and the researchers start tracking the monthly reports of disruptive behavior again. These reports could be subject to the same fluctuations as the pre-intervention reports, and so we once again take multiple measurements over time to try to control for those fluctuations.

This method improves the extent to which we can establish causality because we are accounting for a major extraneous variable in the equation—the passage of time. On its own, it does not allow us to account for other extraneous variables, but it does establish time order and association between the intervention and the trend in reports of disruptive behavior. Finding a stable condition before the treatment that changes after the treatment is evidence for causality between treatment and outcome.

Ex post facto comparison group design

Ex post facto (Latin for “after the fact”) designs are extremely similar to nonequivalent comparison group designs. There are still comparison and experimental groups, pretest and post-test measurements, and an intervention. But in ex post facto designs, participants are assigned to the comparison and experimental groups once the intervention has already happened. This type of design often occurs when interventions are already up and running at an agency and the agency wants to assess effectiveness based on people who have already completed treatment.

In most clinical agency environments, social workers conduct both initial and exit assessments, so there are usually some kind of pretest and post-test measures available. We also typically collect demographic information about our clients, which could allow us to try to use some kind of matching to construct comparison and experimental groups.

In terms of internal validity and establishing causality, ex post facto designs are a bit of a mixed bag. The ability to establish causality depends partially on the ability to construct comparison and experimental groups that are demographically similar so we can control for these extraneous variables .

Quasi-experimental designs are common in social work intervention research because, when designed correctly, they balance the intense resource needs of true experiments with the realities of research in practice. They still offer researchers tools to gather robust evidence about whether interventions are having positive effects for clients.

• Quasi-experimental designs are similar to true experiments, but do not require random assignment to experimental and control groups.
• In quasi-experimental projects, the group not receiving the treatment is called the comparison group, not the control group.
• Nonequivalent comparison group design is nearly identical to pretest/post-test experimental design, but participants are not randomly assigned to the experimental and control groups. As a result, this design provides slightly less robust evidence for causality.
• Nonequivalent groups can be constructed by individual matching or aggregate matching .
• Time series design does not have a control or experimental group, and instead compares the condition of participants before and after the intervention by measuring relevant factors at multiple points in time. This allows researchers to mitigate the error introduced by the passage of time.
• Ex post facto comparison group designs are also similar to true experiments, but experimental and comparison groups are constructed after the intervention is over. This makes it more difficult to control for the effect of extraneous variables, but still provides useful evidence for causality because it maintains the time order of the experiment.
• Think back to the experiment you considered for your research project in Section 13.3. Now that you know more about quasi-experimental designs, do you still think it’s a true experiment? Why or why not?
• What should you consider when deciding whether an experimental or quasi-experimental design would be more feasible or fit your research question better?

13.5 Non-experimental designs

• Describe non-experimental designs in social work research
• Discuss how non-experimental research differs from true and quasi-experimental research
• Demonstrate an understanding the different types of non-experimental designs
• Determine what kinds of research questions non-experimental designs are suited for
• Discuss advantages and disadvantages of non-experimental designs

The previous sections have laid out the basics of some rigorous approaches to establish that an intervention is responsible for changes we observe in research participants. This type of evidence is extremely important to build an evidence base for social work interventions, but it’s not the only type of evidence to consider. We will discuss qualitative methods, which provide us with rich, contextual information, in Part 4 of this text. The designs we’ll talk about in this section are sometimes used in qualitative research  but in keeping with our discussion of experimental design so far, we’re going to stay in the quantitative research realm for now. Non-experimental is also often a stepping stone for more rigorous experimental design in the future, as it can help test the feasibility of your research.

In general, non-experimental designs do not strongly support causality and don’t address threats to internal validity. However, that’s not really what they’re intended for. Non-experimental designs are useful for a few different types of research, including explanatory questions in program evaluation. Certain types of non-experimental design are also helpful for researchers when they are trying to develop a new assessment or scale. Other times, researchers or agency staff did not get a chance to gather any assessment information before an intervention began, so a pretest/post-test design is not possible.

A significant benefit of these types of designs is that they’re pretty easy to execute in a practice or agency setting. They don’t require a comparison or control group, and as Engel and Schutt (2017) [3] point out, they “flow from a typical practice model of assessment, intervention, and evaluating the impact of the intervention” (p. 177). Thus, these designs are fairly intuitive for social workers, even when they aren’t expert researchers. Below, we will go into some detail about the different types of non-experimental design.

One group pretest/post-test design

Also known as a before-after one-group design, this type of research design does not have a comparison group and everyone who participates in the research receives the intervention (Figure 13.8). This is a common type of design in program evaluation in the practice world. Controlling for extraneous variables is difficult or impossible in this design, but given that it is still possible to establish some measure of time order, it does provide weak support for causality.

Imagine, for example, a researcher who is interested in the effectiveness of an anti-drug education program on elementary school students’ attitudes toward illegal drugs. The researcher could assess students’ attitudes about illegal drugs (O 1 ), implement the anti-drug program (X), and then immediately after the program ends, the researcher could once again measure students’ attitudes toward illegal drugs (O 2 ). You can see how this would be relatively simple to do in practice, and have probably been involved in this type of research design yourself, even if informally. But hopefully, you can also see that this design would not provide us with much evidence for causality because we have no way of controlling for the effect of extraneous variables. A lot of things could have affected any change in students’ attitudes—maybe girls already had different attitudes about illegal drugs than children of other genders, and when we look at the class’s results as a whole, we couldn’t account for that influence using this design.

All of that doesn’t mean these results aren’t useful, however. If we find that children’s attitudes didn’t change at all after the drug education program, then we need to think seriously about how to make it more effective or whether we should be using it at all. (This immediate, practical application of our results highlights a key difference between program evaluation and research, which we will discuss in Chapter 23 .)

After-only design

As the name suggests, this type of non-experimental design involves measurement only after an intervention. There is no comparison or control group, and everyone receives the intervention. I have seen this design repeatedly in my time as a program evaluation consultant for nonprofit organizations, because often these organizations realize too late that they would like to or need to have some sort of measure of what effect their programs are having.

Because there is no pretest and no comparison group, this design is not useful for supporting causality since we can’t establish the time order and we can’t control for extraneous variables. However, that doesn’t mean it’s not useful at all! Sometimes, agencies need to gather information about how their programs are functioning. A classic example of this design is satisfaction surveys—realistically, these can only be administered after a program or intervention. Questions regarding satisfaction, ease of use or engagement, or other questions that don’t involve comparisons are best suited for this type of design.

Static-group design

A final type of non-experimental research is the static-group design. In this type of research, there are both comparison and experimental groups, which are not randomly assigned. There is no pretest, only a post-test, and the comparison group has to be constructed by the researcher. Sometimes, researchers will use matching techniques to construct the groups, but often, the groups are constructed by convenience of who is being served at the agency.

Non-experimental research designs are easy to execute in practice, but we must be cautious about drawing causal conclusions from the results. A positive result may still suggest that we should continue using a particular intervention (and no result or a negative result should make us reconsider whether we should use that intervention at all). You have likely seen non-experimental research in your daily life or at your agency, and knowing the basics of how to structure such a project will help you ensure you are providing clients with the best care possible.

• Non-experimental designs are useful for describing phenomena, but cannot demonstrate causality.
• After-only designs are often used in agency and practice settings because practitioners are often not able to set up pre-test/post-test designs.
• Non-experimental designs are useful for explanatory questions in program evaluation and are helpful for researchers when they are trying to develop a new assessment or scale.
• Non-experimental designs are well-suited to qualitative methods.
• If you were to use a non-experimental design for your research project, which would you choose? Why?
• Have you conducted non-experimental research in your practice or professional life? Which type of non-experimental design was it?

13.6 Critical, ethical, and cultural considerations

• Describe critiques of experimental design
• Identify ethical issues in the design and execution of experiments
• Identify cultural considerations in experimental design

As I said at the outset, experiments, and especially true experiments, have long been seen as the gold standard to gather scientific evidence. When it comes to research in the biomedical field and other physical sciences, true experiments are subject to far less nuance than experiments in the social world. This doesn’t mean they are easier—just subject to different forces. However, as a society, we have placed the most value on quantitative evidence obtained through empirical observation and especially experimentation.

Major critiques of experimental designs tend to focus on true experiments, especially randomized controlled trials (RCTs), but many of these critiques can be applied to quasi-experimental designs, too. Some researchers, even in the biomedical sciences, question the view that RCTs are inherently superior to other types of quantitative research designs. RCTs are far less flexible and have much more stringent requirements than other types of research. One seemingly small issue, like incorrect information about a research participant, can derail an entire RCT. RCTs also cost a great deal of money to implement and don’t reflect “real world” conditions. The cost of true experimental research or RCTs also means that some communities are unlikely to ever have access to these research methods. It is then easy for people to dismiss their research findings because their methods are seen as “not rigorous.”

Obviously, controlling outside influences is important for researchers to draw strong conclusions, but what if those outside influences are actually important for how an intervention works? Are we missing really important information by focusing solely on control in our research? Is a treatment going to work the same for white women as it does for indigenous women? With the myriad effects of our societal structures, you should be very careful ever assuming this will be the case. This doesn’t mean that cultural differences will negate the effect of an intervention; instead, it means that you should remember to practice cultural humility implementing all interventions, even when we “know” they work.

How we build evidence through experimental research reveals a lot about our values and biases, and historically, much experimental research has been conducted on white people, and especially white men. [4] This makes sense when we consider the extent to which the sciences and academia have historically been dominated by white patriarchy. This is especially important for marginalized groups that have long been ignored in research literature, meaning they have also been ignored in the development of interventions and treatments that are accepted as “effective.” There are examples of marginalized groups being experimented on without their consent, like the Tuskegee Experiment or Nazi experiments on Jewish people during World War II. We cannot ignore the collective consciousness situations like this can create about experimental research for marginalized groups.

None of this is to say that experimental research is inherently bad or that you shouldn’t use it. Quite the opposite—use it when you can, because there are a lot of benefits, as we learned throughout this chapter. As a social work researcher, you are uniquely positioned to conduct experimental research while applying social work values and ethics to the process and be a leader for others to conduct research in the same framework. It can conflict with our professional ethics, especially respect for persons and beneficence, if we do not engage in experimental research with our eyes wide open. We also have the benefit of a great deal of practice knowledge that researchers in other fields have not had the opportunity to get. As with all your research, always be sure you are fully exploring the limitations of the research.

• While true experimental research gathers strong evidence, it can also be inflexible, expensive, and overly simplistic in terms of important social forces that affect the resources.
• Marginalized communities’ past experiences with experimental research can affect how they respond to research participation.
• Social work researchers should use both their values and ethics, and their practice experiences, to inform research and push other researchers to do the same.
• Think back to the true experiment you sketched out in the exercises for Section 13.3. Are there cultural or historical considerations you hadn’t thought of with your participant group? What are they? Does this change the type of experiment you would want to do?
• How can you as a social work researcher encourage researchers in other fields to consider social work ethics and values in their experimental research?

Media Attributions

• Being kinder to yourself © Evgenia Makarova is licensed under a CC BY-NC-ND (Attribution NonCommercial NoDerivatives) license
• Original by author is licensed under a CC BY-NC-SA (Attribution NonCommercial ShareAlike) license
• Original by author. is licensed under a CC BY-NC-SA (Attribution NonCommercial ShareAlike) license
• Orginal by author. is licensed under a CC BY-NC-SA (Attribution NonCommercial ShareAlike) license
• therapist © Zackary Drucker is licensed under a CC BY-NC-ND (Attribution NonCommercial NoDerivatives) license
• nonexper-pretest-posttest is licensed under a CC BY-NC-SA (Attribution NonCommercial ShareAlike) license
• Engel, R. & Schutt, R. (2016). The practice of research in social work. Thousand Oaks, CA: SAGE Publications, Inc. ↵
• Sullivan, G. M. (2011). Getting off the “gold standard”: Randomized controlled trials and education research. Journal of Graduate Medical Education ,  3 (3), 285-289. ↵

an operation or procedure carried out under controlled conditions in order to discover an unknown effect or law, to test or establish a hypothesis, or to illustrate a known law.

explains why particular phenomena work in the way that they do; answers “why” questions

variables and characteristics that have an effect on your outcome, but aren't the primary variable whose influence you're interested in testing.

the group of participants in our study who do not receive the intervention we are researching in experiments with random assignment

in experimental design, the group of participants in our study who do receive the intervention we are researching

the group of participants in our study who do not receive the intervention we are researching in experiments without random assignment

using a random process to decide which participants are tested in which conditions

The ability to apply research findings beyond the study sample to some broader population,

Ability to say that one variable "causes" something to happen to another variable. Very important to assess when thinking about studies that examine causation such as experimental or quasi-experimental designs.

the idea that one event, behavior, or belief will result in the occurrence of another, subsequent event, behavior, or belief

An experimental design in which one or more independent variables are manipulated by the researcher (as treatments), subjects are randomly assigned to different treatment levels (random assignment), and the results of the treatments on outcomes (dependent variables) are observed

a type of experimental design in which participants are randomly assigned to control and experimental groups, one group receives an intervention, and both groups receive pre- and post-test assessments

A measure of a participant's condition before they receive an intervention or treatment.

A measure of a participant's condition after an intervention or, if they are part of the control/comparison group, at the end of an experiment.

A demonstration that a change occurred after an intervention. An important criterion for establishing causality.

an experimental design in which participants are randomly assigned to control and treatment groups, one group receives an intervention, and both groups receive only a post-test assessment

The measurement error related to how a test is given; the conditions of the testing, including environmental conditions; and acclimation to the test itself

a subtype of experimental design that is similar to a true experiment, but does not have randomly assigned control and treatment groups

In nonequivalent comparison group designs, the process by which researchers match individual cases in the experimental group to similar cases in the comparison group.

In nonequivalent comparison group designs, the process in which researchers match the population profile of the comparison and experimental groups.

a set of measurements taken at intervals over a period of time

Research that involves the use of data that represents human expression through words, pictures, movies, performance and other artifacts.

Graduate research methods in social work Copyright © 2021 by Matthew DeCarlo, Cory Cummings, Kate Agnelli is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License , except where otherwise noted.

Share This Book

• Liberty University
• Jerry Falwell Library
• Special Collections
• < Previous

Home > ETD > Doctoral > 5868

Doctoral Dissertations and Projects

A quasi-experimental study on the effects of small group learning on mathematical resilience in upper elementary students.

Joshua Adam Costelnock , Liberty University Follow

School of Education

Doctor of Philosophy in Education (PhD)

Janice Kooken

whole group instruction, small group learning, mathematical resilience, progressive classroom

Disciplines

Recommended citation.

Costelnock, Joshua Adam, "A Quasi-Experimental Study on the Effects of Small Group Learning on Mathematical Resilience in Upper Elementary Students" (2024). Doctoral Dissertations and Projects . 5868. https://digitalcommons.liberty.edu/doctoral/5868

The purpose of this quantitative, quasi-experimental study was to determine the effect of small group learning during the core mathematics block on 5th-grade students’ mathematical resilience, compared to a control group. Student collaboration and mathematical discourse decreased during the COVID-19 pandemic, leading to a drop in math proficiency levels in the United States. Approximately 80 5th-grade students from the southwest United States were divided into two sample groups of about 40 each. These groups were assessed using the Upper Elementary Mathematics Resilience Scale. One group primarily experienced teacher-centered whole group instruction, while the other group spent half of their daily core learning block in student-centered small group instruction. Differences between the two groups were analyzed using ANCOVA on the two measures of the Mathematical Resilience Scale: value and growth mindset. The ANCOVA tested for differences in the post-test, using the pre-test as the covariate. Data for the value subscale showed a statistically significant change between the groups, though the direction of the change was unexpected. Data for the growth subscale did not reach appropriate levels of significance. For future research, it is recommended that the scale be administered at the beginning of the school year instead of the end, and that the sample size be increased in both groups.

Since August 09, 2024

Included in

Education Commons

• Collections
• Faculty Expert Gallery
• Theses and Dissertations
• Conferences and Events
• Open Educational Resources (OER)
• Explore Disciplines

Advanced Search

• Notify me via email or RSS .

Faculty Authors

• Submit Research
• Expert Gallery Login

Student Authors

• Undergraduate Submissions
• Graduate Submissions
• Honors Submissions

Home | About | FAQ | My Account | Accessibility Statement

Privacy Copyright

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here .

Loading metrics

Open Access

Peer-reviewed

Research Article

Effect of an educational intervention based on self-efficacy theory and health literacy skills on preventive behaviors of urinary tract infection in pregnant women: A quasi-experimental study

Roles Writing – original draft, Writing – review & editing

Affiliations Social Determinants of Health Research Center, Mashhad University of Medical Sciences, Mashhad, Iran, Faculty of Health, Department of Health, Safety, and Environment, Mashhad University of Medical Sciences, Mashhad, Iran

Roles Data curation

Affiliation Faculty of Health, Department of Health Education and Health Promotion, Mashhad University of Medical Sciences, Mashhad, Iran

Roles Writing – review & editing

Affiliation Department of Health Education and Health Promotion, School of Health, Mashhad University of Medical Sciences, Mashhad, Iran

Roles Methodology

Affiliations Social Determinants of Health Research Center, Mashhad University of Medical Sciences, Mashhad, Iran, Faculty of Health Sciences, Department of Epidemiology and Biostatistics, Mashhad University of Medical Sciences, Mashhad, Iran

* E-mail: [email protected]

Affiliations Social Determinants of Health Research Center, Mashhad University of Medical Sciences, Mashhad, Iran, Department of Health Education and Health Promotion, School of Health, Mashhad University of Medical Sciences, Mashhad, Iran

• Seyedeh Belin Tavakoly Sany, 
• Vajieh Eslami, 
• Elaheh lael-Monfared, 
• Vahid Ghavami, 
• Nooshin Peyman

• Published: August 13, 2024
• https://doi.org/10.1371/journal.pone.0306558
• Reader Comments

The impact of self-efficacy and health literacy skills on pregnant women’s adherence to urinary tract infection (UTI) preventive behaviors is inadequately investigated. Thus, the present study explored whether an educational intervention based on self-efficacy and health literacy skills managed to improve UTI preventive behaviors among pregnant women.

A quasi-experimental study was conducted from January to July 2021 among pregnant women residing in Mashhad, Iran. To this aim, 110 pregnant women at a gestational age of 12–18 weeks were randomly assigned to a control (n = 55) and an intervention group (n = 55) and completed all questionnaires during the intervention and the 3-month follow-up. The intervention group received the full training program, comprising six 2-hourly training sessions.

Most women were from low-income families (69.1%), were housewives (74.5%) with high school education or lower (63.6%). The theory-based intervention had a significant effect (P < 0·05) on UTI preventive behavior outcomes (i.e., clothing habits, nutrition, urination, health, and sexual behaviors) in the intervention group compared with the control group after intervention, and in their variation from baseline to follow-up in all scores.

Conclusions

An educational intervention based on health literacy skills and self-efficacy could be an effective theory-based intervention to improve UTI preventive behaviors and reduce recurrent UTI and complications.

Citation: Tavakoly Sany SB, Eslami V, lael-Monfared E, Ghavami V, Peyman N (2024) Effect of an educational intervention based on self-efficacy theory and health literacy skills on preventive behaviors of urinary tract infection in pregnant women: A quasi-experimental study. PLoS ONE 19(8): e0306558. https://doi.org/10.1371/journal.pone.0306558

Editor: Kahsu Gebrekidan, University of Oulu: Oulun Yliopisto, FINLAND

Received: December 31, 2023; Accepted: June 18, 2024; Published: August 13, 2024

Copyright: © 2024 Tavakoly Sany et al. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: We confirm at this time Our submission contains all raw data required to replicate the results of our study. All relevant data are within the manuscript and its Supporting Information files.

Funding: This research was funded by the Mashhad University of Medical Sciences (Project number: 980582). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Competing interests: The authors have declared that no competing interests exist.

Abbreviations: UTI, urinary tract infection; WHO, World Health Organization

1. Background

Pregnancy is a natural physiological process in a woman’s life, accompanied by physiological and psychological changes. However, maternal comorbidities or unexpected diseases can complicate pregnancy and have adverse effects. Thus, a mother’s health before and during childbirth is very important for children’s health [ 1 , 2 ].

Urinary tract infection (UTI) is a common clinical disease marked by a continuous and active proliferation of bacteria inside the urinary tract [ 3 ], and involves the urinary tract, bladder and kidney infections. UTI may be symptomatic or asymptomatic [ 4 ] with the latter being of a particular importance due to the absence of any symptoms. Its complications account for about 150 million mortalities annually worldwide [ 5 ]. UTI is a common bacterial infection and the second main complication of pregnancy, after anemia. Anatomical and physiological changes of the urinary tracts during pregnancy increase the prevalence of UTI [ 6 ]. The prevalence of asymptomatic bacteriuria in the world is 2–15% [ 7 ]. In Iran, the prevalence of UTI in pregnant women is 8.7% [ 8 ].

In developing countries, pregnant women have a higher rate of UTI than counterparts in developed countries [ 9 ]. In a meta-analysis, the overall prevalence of UTI during pregnancy in Iran was estimated at 13%. In different parts of Iran, this rate varied greatly. For example, in Tehran and Arak, it is 2–13%, in Hamadan 10%, and in Torbat Heydarieh, it is reported to be 10% [ 10 ]. UTI is among the most widespread and costly medical complications in pregnancy which accounts for 10% of all hospitalizations during pregnancy [ 11 ]. As the existing literature shows, UTI in pregnant women begins at the 6 th week of pregnancy and reaches its peak in the 22 nd - 24 th week [ 2 , 8 ].

Besides the high cost of treatment and hospitalization, UTI during pregnancy has many lifelong maternal and fetal complications, including pyelonephritis, preeclampsia, shock, septicemia, anemia, and endometritis. Fetal complications of UTI during pregnancy include birth weight loss, premature birth, respiratory failure, fetal death, mental retardation, and lower intelligence quotient (IQ) [ 4 , 9 ]. The report of the World Health Organization (WHO) on premature birth shows that every year a million infants die due to premature birth [ 12 ] and that the probability of preeclampsia in pregnant women with UTI is 1.22 times as high as pregnant women without UTI [ 13 ]. Antibiotics are essential to fight UTIs during pregnancy [ 12 ], but an excessive use is a global health threat as it can develop antimicrobial resistance and increase the risk of spontaneous abortion and birth defects [ 14 , 15 ]. The consumption of safe antibiotics during pregnancy is limited due to their teratogenic potential [ 16 ]. In light of the aforementioned issues, several measures can be taken to prevent UTI during pregnancy, such as adherence to healthy behaviors in sexual activity, the clothing style, eating habits, urinary habits and cleaning, which are all among the predisposing factors for UTI [ 17 , 18 ].

Inadequate knowledge and skills can decrease the motivation to adopt preventive behaviors and can hinder a full prevention [ 17 ]. Health literacy skills and self-efficacy are effective factors to prevent infectious diseases [ 19 – 21 ]. In the related literature, health literacy is “a set of reading, listening, analysis and decision-making skills, and the ability to apply these skills in health-related conditions” [ 22 , 23 ]. American Center for Health Care Strategies reported that people with higher health literacy have more chances of using spoken and written information provided by professionals; therefore, they have a better state of health. Health literacy skills improve the acquisition of knowledge about health issues, correct decisions about health, and benefits of healthcare services [ 24 , 25 ]. Problems related to lifestyle changes require a high level of self-confidence. Achieving high self-efficacy, and improving self-efficacy and health literacy is possible through active education [ 26 ].

Choosing a behavior change model for health education is the first step to a planning process [ 18 , 27 ]. A prominent educational theory used to predict and describe behavior is the self-efficacy theory, commonly used in behavior changing programs[ 28 ]. According to Bandura, there are four main sources of self-efficacy including mastery experiences, vicarious experiences, verbal persuasion, and physiological and affective states” [ 29 ]. Self-efficacy is a major prerequisite for behavior change [ 30 ]. Individuals with inadequate self-efficacy are less likely to make efforts to show a new healthy behavior or to change the former unhealthy behavior [ 31 ].

There is research evidence that self-efficacy is an important psychological construct directly and indirectly affecting disease-controlling health behaviors. Self-efficacy can transform knowledge and information related to health promotion and educational interventions in behavioral performance [ 32 ]. Health literacy has been included as a predictor of self-efficacy [ 32 , 33 ]. Although a body of research in Iran shows that women’s awareness of UTI prevention is at a satisfactory level, the prevalence of UTI in pregnant women is still increasing [ 8 , 17 , 34 ]. It seems that only raising the level of knowledge cannot lead to the prevention of UTI [ 35 – 37 ], and there is a need for recognition of other factors affecting on UTI preventive behaviors [ 38 ]. Research evidence shows that self-efficacy and health literacy skills are effective in improving health behaviors. Yet, the relationship between UTI preventive behaviors and health literacy and self-efficacy in pregnant women has not been investigated. Considering the high prevalence of UTI during pregnancy and the serious risks that threaten the mother’s and fetus’ health, the present study aimed to investigate the effect of a health education intervention based on the self-efficacy theory and health literacy skills on pregnant women’s UTI preventive behaviors. The present findings can help decision-makers develop a comprehensive educational program to promote UTI preventive behaviors. Preventive behaviors against UTI helps reduce the excessive use of antibiotics, especially in pregnant women.

2. Materials and methods

2.1. participants and sampling.

λ : ratio of sample size in group 2 to group 1

v : number of measures before intervention

w : number of measures after intervention

p t : correlation coefficient of repeated measures

Δ plan : standardized expected effect size

Due to the lack of data required to estimate the sample size in the existing literature (e.g., the absence of standard deviation of scores after the intervention in the intervention group), the information about control group in the study by Tehrani et al. [ 26 ] was used. The equality of variance of two groups was assumed. Cohen’s standard effect size was 0.56. The first type error was 0.05 and the test power was 80%. λ , v, w and p t. were, respectively, 1, 1, 2 and 0. 5. The estimated sample size, with an attrition rate of 10%, for each group, was 55. The participants were randomly assigned to the intervention and control groups.

In the present study, the data collection was done based on a test of functional health literacy in adults [ 28 , 30 ], and the general self-efficacy questionnaire [ 39 ]. Also, a researcher-made questionnaire was developed to measure UTI preventive behaviors. This questionnaire included demographic information (occupation, age, education, husband’s education and occupation, body mass index (BMI), vomiting during pregnancy, and income) as well as the five domains of UTI prevention behaviors. The questionnaires were completed before, immediately after and three months after the educational intervention in the health centers. All participants were informed about the purpose of study and their demographic information was recorded. Having signed an informed letter of consent, the participants completed the questionnaire of UTI prevention behaviors, test of functional health literacy in adults (TOFHLA) and Schwarzer’s self-efficacy scale.

General self-efficacy questionnaire (GSE).

Schwarzer’s general self-efficacy questionnaire was used to measure the participants’ self-efficacy. This scale contained 17 questions rated on a 4-point Likert scale ranging from strongly disagree to strongly agree. It was scored from 17 to 85, and a high score showed stronger self-efficacy. Three aspects of behavior, including the desire to initiate the behavior, resistance to barriers, and efforts to complete the task were measured using this test (e.g., “I am a self-reliant person.”, and “I avoid facing difficulties.”) ( S1 Table ). The reliability of scale was estimated at 0.84 in the study by Woodruff and Kashman, and 0.83 in the study by Asgharanjad, and Ahmadi Qutb Al-Dini [ 27 , 39 , 40 ].

Test of functional health literacy in adults (TOFHLA).

This questionnaire consisted of two sections, calculations and reading comprehension. The calculations section assessed one’s ability to understand the doctor’s and health educators’ advice. This section required certain calculations, and the score could range between 0 and 50. The reading comprehension section assessed one’s ability to read and comprehend three passages entitled as instructions on preparing for imaging of the upper gastrointestinal tract, the patient responsibilities and rights about the standard hospital consent form and insurance forms. This score ranged from 0 to 50. Thus, the overall health literacy score obtained from these two sections could range between 0 and 100. There were three levels of interpretation of scores: insufficient (0–59), borderline (60–74) and sufficient (75–100) [ 31 ]. The validity and reliability of this questionnaire in Iran were measured by Raisi et al. The reliability was estimated at 0.79 for the calculations section and 0.88 for the reading comprehension section. Its content validity ratio (CVR) was higher than 0.56. and the content validity index (CVI) was estimated at 0.79 [ 28 – 30 ].

Urinary tract infection preventive behaviors questionnaire.

In this study, a researcher-made questionnaire was used to measure UTI preventive behaviors in pregnant women. This questionnaire includes demographic information and five dimensions of UTI preventive behaviors, including 25 questions on clothing style (4 questions), eating habits (6 questions), urinary habits (2 questions), cleanliness (7 questions) and sexual behavioral habits (6 questions) ( S2 Table ). In this instrument, the questions were rated based on a Likert scale ranging from never (0) to always (4), with a minimum score of 25 and a maximum score of 100. To check the content validity of the researcher-made questionnaire, it was provided to six eminent professors of health promotion, two distinguished professors of reproductive health, two gynecologists and five midwifery experts. Thus, the content validity (CVR) was measured and substantiated. For the overall instrument, the CVR was estimated at 0.94. Having made the suggested revisions, the content validity index (CVI) for all scales was increased to 0.94. To check internal consistency, Cronbach’s alpha test was used, and the estimated value was 0.72. Also, to check the reliability, a test-retest method was used for 20 pregnant mothers at a time interval of two weeks, based on which the intra-cluster correlation coefficient (ICC) was estimated at 0.97, indicating an acceptable reliability [ 41 ].

2.2. Intervention

The present quasi-experimental study involved two groups, an intervention, and a control. The intervention was made from January 2021 to July 2021 based on a consort checklist and the Template for Intervention Description and Replication checklist (TIDieR) ( Table 1 ) [ 42 ]. Four health centers were selected randomly from a list of centers, and were assigned to the intervention group (n = 2) and the control (n = 2). Then, a list was made of pregnant women’s names based on their demographic information and health history, and a number was assigned to each using a table of random numbers. Two hundred pregnant women were randomly selected, of whom 84 women were not included in the intervention because they did not meet the inclusion criteria. Six women failed to attend the training or follow-up because of travelling, COVID-19 lock-down, and work-related problems. Finally, 110 women completed all stages of study (before intervention, immediately after intervention, and three months after intervention) ( Fig 1 ).

• PPT PowerPoint slide
• PNG larger image
• TIFF original image

https://doi.org/10.1371/journal.pone.0306558.g001

https://doi.org/10.1371/journal.pone.0306558.t001

The educational intervention was conducted for the intervention group. All women underwent a training program of six two-hour sessions every 7 days. From two centers of the control group, 55 pregnant women with similar conditions were randomly selected and considered as the control group, and the educational content was provided to them after the completion of the intervention. Due to the COVID-19 pandemic, the sensitivity of pregnant mothers’ condition and the health protocols against face-to-face group meetings, four training sessions were held face to face, and the remaining sessions were held online on WhatsApp as the mothers requested. In this study, different oral and combined methods (e.g., lectures with Q&As, brainstorming, group discussions, poster presentation and pamphlets) were used along with online sources (e.g., telephone and social networks to share videos, photos and group discussions in real-time class held in audio-only mode) ( Table 1 ). The intervention training program was designed based on Bandura’s self-efficacy theory [ 27 ] and health literacy skills (spoken communication, promotion and written communication, empowerment, improvement of support systems) [ 40 ] ( Table 2 ).

https://doi.org/10.1371/journal.pone.0306558.t002

In this educational program, according to the participants’ age and literacy level and the objectives of the educational program, there were three cognitive, attitudinal, and functional domains to address, for which visual and auditory media were used such as educational slides, overhead projectors, whiteboards, and pamphlets in this program. Face-to-face training and phone-mediated training were used along with video, photo and voice records in non-face-to-face training. Trainings were conducted by a health education specialist and a gynecologist. During the study, the control group did not receive any special training from the researcher, and after the completion of the intervention, the training was provided as e-learning to the control group. Questionnaires in both groups were completed once before the intervention, and twice more immediately after and three months after the educational intervention. This was done face to face in the first session and online via sharing the questionnaire link in the group to complete.

2.3. Data statistical analysis

Having collected the data, to analyze the descriptive data, the questionnaires were coded and punched into SPSS21. After a careful checking and ensuring of the accuracy of data entry, descriptive statistics of the central tendency and variability indices, such as the mean and standard deviation of values related to the interval variables, and the distribution of frequency and percentage of non-parametric variables were used. To check the normality of distribution of interval variables in the treatment and control groups, Kolmogorov-Smirnov test was used. As the results showed, appropriate parametric tests were used for interval variables and appropriate non-parametric tests were used for non-interval variables. To test the relationship between interval variables, Spearman and Pearson correlation coefficients were used according to the abnormal distribution of data. Mann-Whitney U-test, and Kruskal-Wallis tests were used to test the relationship between interval and non-interval variables according to the number of classes of qualitative variables. Chi-square test was used to explore the relationship between non-interval variables. To compare the two groups before, immediately after and 3 months after intervention in terms of interval variables, repeated measure analysis of variance was used. Friedman’s test was used for non-interval variables. The significance level in all tests was 0.05 and SPSS 21 was used to describe and analyze the data.

Ethics approval and consent to participate.

The study protocol was approved by the Ethics Committee of Mashhad University of Medical Sciences (#IR.MUMS.REC.1398.268) after obtaining the required permit for the research. The participants provided a written informed consent and were assured of confidentiality of data. All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional research committee with the 1964 Helsinki declaration.

Before intervention, there were no significant differences (P>0.05) between the intervention and control groups in terms of demographic characteristics (i.e., age, gestational age, education, income, employment status, BMI, history of pre-pregnancy UTI, and vomiting during pregnancy). In this sense, the variables were homogeneous in both groups. The mean (±SD) of age, gestational age, and BMI were 24.80 (±4.92), 13.69 (± 3.82) and 24.93 (±3.18), respectively. Most eligible women were housekeepers (74.5%), low-income families (69.1%) with high school diploma or below (63.6%) ( Table 3 ).

https://doi.org/10.1371/journal.pone.0306558.t003

At the baseline, all UTI preventive behavioral constructs, total preventive behaviors, and self-efficacy were homogeneous in both groups. The results related to UTI preventive behaviors showed a significant improvement (P < 0.05) in all constructs (clothing habits, nutrition, urination, health, and sexual behaviors) in the intervention group at the follow-up, and in all scores changing from baseline to the follow-up. The results of testing self-efficacy showed a significant change (P < 0.05) in the intervention group compared to the control group in the follow-up, and in changes from the baseline to follow-up ( Table 4 ).

https://doi.org/10.1371/journal.pone.0306558.t004

The mean health literacy score immediately after the intervention and three months later was significantly different in the intervention group. The mean score of health literacy was significantly different within the intervention group (p < 0.001). There was no significant difference (P > 0.05) in the change of UTI preventive behavior constructs, total preventive behaviors, self-efficacy, and self-efficacy in the control group at the follow-up ( Table 4 ). The results presented in S3 Table showed that the incidence of UTI three months after the intervention in the control group was 25.4%. The control group significantly had more cases with UTI than the intervention group.

In this section, a generalized estimating equation ( GEE ) model was used to simultaneously measure the effect of intervention, time, self-efficacy, and health literacy on UTI preventive behaviors. The results of the GEE model were in line with the bivariate analysis that showed significant interactions between groups and time. S4 and S5 Tables showed the impact of intervention based on health literacy and self-efficacy on improving UTI preventive behaviors in different groups and times. Changes in UTI preventive behavior score within the intervention group were significantly higher than the control (P = 0 <0.001), and UTI preventive behaviors were increased considerably across time in the baseline through follow-up among participants in the intervention group compared with the control (P = 0 <0.001). As the results showed, changes in self-efficacy (p = 0.043) and health literacy (p = 0.042) were significantly associated with UTI preventive behaviors.

4. Discussion

Due to the prevalence of UTI in pregnant women, UTI is considered a major concern in public health public health [ 41 , 43 , 44 ]. The present finding suggests that conducting an educational intervention based on the self-efficacy theory and health literacy skills among pregnant women is an effective intervention to control and prevent UTI because women in the intervention group represented a lower risk of UTI and better preventive behaviors compared with participants in the control group.

The present findings showed a significant increase in the level of preventive behaviors in the intervention group. Before the intervention, there was no significant difference between the intervention and control groups. However, after the educational intervention, this difference was statistically significant. The present study showed that the educational intervention based self-efficacy and health literacy skills and the use of educational strategies and programs such as the mastery of alternative behavior and verbal persuasion, educational methods such as goal-setting and role-play were effective in improving preventive behaviors. As the present findings showed, the use of the self-efficacy theory can be effective in improving perceived self-efficacy in individuals. It seems that women with adequate self-efficacy and health literacy may well find and use health information and engage in their care [ 18 , 45 ].

In this study, pregnant women in the intervention group showed a significant change in the mean score of self-efficacy after the intervention. All women learned how to break complex tasks into smaller and simpler activities and set realistic goals to modify their action and commitment to conduct UTI preventive behaviors despite conflicting conditions. Likewise, the present researchers tried to improve mother’s self-confidence and self-monitoring to perform certain behaviors. The existing literature shows that individuals with low self-efficacy use fewer resources of health information and health literacy to improve their health or change the habitual behaviors [ 33 ]. The findings reported by Osborn et al., 2011 are consistent with the present findings, as individuals with higher perceived self-efficacy had a better understanding of their health state and used health information and health literacy to improve their health and show self-care behaviors [ 34 , 46 ]. The results of the present study are in line with a body of research by Hejazi et al. [ 47 ], Abdullahi et al. [ 48 ], which showed a significant effect of self-efficacy on adopting, initiating, and maintaining healthy behavior. They found that self-efficacy acted as a moderator to link healthy behaviors with motivation and knowledge [ 49 , 50 ].

The results of the present study showed statistically significant differences in the change of health literacy skills in the intervention group at 3-months follow-up, and in changes from baseline to follow-up in all scores. Health literacy is the main skill to influence one’s ability to use health information, make well-informed decisions, and maintain good health [ 38 , 40 , 42 , 43 ]. Before the intervention, women had difficulty finding and comprehending health information and healthcare services to make well-health decisions. Likewise, a significant improvement in the health literacy score was found in the intervention group in post-intervention and follow-up. This could be due to the improved women’s willingness and ability to involve in behaviors and care that improve their health. In the present study, a supportive and reliable environment was created to address health information and measures that contribute to a higher stage of well-health decisions and commitments among pregnant women to modify their UTI preventive behaviors [ 4 , 6 ]. Therefore, it is essential to promote health literacy skills in community, as high health literacy is associated with better health outcomes among patients.

Therefore, it is necessary to plan and implement model-based educational programs based on the self-efficacy theory and health literacy skills to increase pregnant women’s self-efficacy and health literacy. The results of the present study are in line with a body of research. In a descriptive study conducted on 140 pregnant women in Zahedan based on the Health Belief Model (HBM), Rahimi et al. showed that self-efficacy was the strongest predictor of preventive behaviors against UTI. It seems that the reasons for the greater effect of self-efficacy are women’s self-confidence and awareness of the effect of simple behaviors and measures to control UTI [ 45 , 46 ]. In a quasi-experimental study conducted on 60 mothers to children under 6 years of age, Hashemiparast et al. showed the mean self-efficacy score was increased in the intervention group after the intervention. In this study, self-efficacy implied confidence in one’s ability to perform UTI preventive behaviors [ 47 ]. Eshghi Mutlaq et al. (2016) found that their educational intervention had a significant effect on improving self-care behaviors in mothers with prediabetes during pregnancy, who felt more self-efficacious and capable of understanding their positive state of health. They also showed showed diabetes self-care behaviors in their daily life [ 48 ]. In line with the present study, Ebrahimipour et al. (1994) conducted some research on the effect of an educational intervention based on the self-efficacy theory on the adoption of HIV-AIDS preventive behaviors in high-risk women. This study showed that the educational intervention based on self-efficacy strategies could significantly increase the adoption of self-care behaviors in the intervention group (P<0.001) [ 49 ]. Ha et al. also showed that the educational intervention and the use of educational strategies and programs such as mastery of alternative behavior and verbal persuasion, educational methods such as goal-setting and role-play were effective in improving self-care [ 45 ]. As the present study showed, the use of this theory proved effective in improving perceived self-efficacy in individuals. The findings emphasized the importance of self-efficacy in preventive behaviors as a suitable educational alternative for UTI self-care and prevention in pregnant women. Therefore, it is necessary to plan and implement model-based educational programs to increase pregnant women’s self-efficacy.

It seems that women with adequate self-efficacy and health literacy may well find and use health information and well engage in their care [ 18 , 46 , 50 ]. Limited studies exist, investigating the role of self-efficacy and health literacy pandemic conditions influencing awareness and health behaviors among pregnant women. Therefore, further studies need to be conducted on enhancing women’s capability to improve health prevention behaviors toward the different diseases, and focusing on health literacy skills and self-efficacy strategies cause persistent and long-term health behaviors.

The strengths of our findings lie in determining the role of self-efficacy and health literacy using valid instrument among the pregnant women as the groups at risk. Our findings highlighted self-efficacy and health literacy skills as the main modifiable determinants to control and manage unborn child’s health and mother’s health because an adequate level of health literacy and self-efficacy improved individual’s healthy behaviors and health outcomes. Future research on intervention-based health literacy and self-efficacy skills will continue in Iran because this type of training for individuals empowers communities to engage in their self-care, improve the healthy behavior, and can increase valuable health outcome in strengthening healthcare delivery. Therefore, it would be worthwhile to study the modifying health literacy and self-efficacy as a long-term measure.

In this study, the data collection instrument was self-reporting, which can cause problems such as recall and distraction. Due to the COVID-19 pandemic, the questionnaires were not filled face to face. Instead, the questionnaire hyperlink was shared with the pregnant women to fill out the questionnaires in their convenience. In this type of questionnaire completion, errors occur more often, and the researcher has no control over the respondents, which reduces the number of visits by pregnant women to the health centers, as well as attendance to face-to-face training sessions.

Finally, the results of the present study showed that the educational intervention based on the self-efficacy theory and health literacy skills can be effective in improving UTI preventive behaviors. The promotion of UTI preventive behaviors in pregnant women after the intervention showed that holding training sessions based on the self-efficacy theory and health literacy was useful. Such a training can improve preventive behaviors. The results of this research can be used to increase UTI preventive behaviors in all sex and age groups and can reduce recurrent UTI complications. Also, the present findings can help health system managers formulate intervention programs specifically for employees to prevent office infection and increase health indices while maintaining the health of the mother and the fetus. Development of educational programs by managers for health workers aiming to raise the awareness of women visiting health centers can reduce economic and psychological costs imposed on society.

Supporting information

S1 table. scherer general self-efficacy questionnaire..

https://doi.org/10.1371/journal.pone.0306558.s001

S2 Table. Distribution of urinary tract infection prevention behaviors.

https://doi.org/10.1371/journal.pone.0306558.s002

S3 Table. UTI ratio in control and intervention groups at follow-up.

https://doi.org/10.1371/journal.pone.0306558.s003

S4 Table. Effectiveness of the intervention on improving the UTI preventive behaviors via self-efficacy in different group and time period.

https://doi.org/10.1371/journal.pone.0306558.s004

S5 Table. Effectiveness of the intervention on improving the UTI preventive behaviors via Health literacy in different group and time period.

https://doi.org/10.1371/journal.pone.0306558.s005

Acknowledgments

The authors wish to express their gratitude towards the vice president of research in Mashhad University of Medical Sciences, the chiefs and staffs of the health centers and the esteemed participants.

• View Article
• Google Scholar
• PubMed/NCBI
• 13. Ratzan S. and Parker R., Health literacy. National library of medicine current bibliographies in medicine. Bethesda: National Institutes of Health, US Department of Health and Human Services, 2000.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

• Search in PubMed
• Search in NLM Catalog
• Add to Search

Effectiveness of early cognitive exercise intervention on improvement cognitive function of stroke patients in the acute phase

Affiliations.

• 1 Latemmamala Hospital, Soppeng Jl. Malaka Raya, Soppeng, Indonesia.
• 2 Faculty of Nursing, Hasanuddin University, Makassar, Indonesia Perintis Kemerdekaan KM. 10, Tamalanrea Makassar 90245, Indonesia. Electronic address: [email protected].
• 3 Faculty of Nursing, Hasanuddin University, Makassar, Indonesia Perintis Kemerdekaan KM. 10, Tamalanrea Makassar 90245, Indonesia.
• PMID: 38555179
• DOI: 10.1016/j.jvn.2023.11.010

The purpose of this study was to assess the efficacy of early cognitive training in enhancement of cognitive function in stroke patients. This research used a quasi-experimental design, 45 patients were divided into two groups, and sequential sampling was employed. The experimental group (n = 22) received two weeks of early cognitive training six times per week, whereas the control group (n = 23) received regular hospital care. The Indonesian version of the Montreal Cognitive Assessment was used to evaluate cognitive function (MoCA-Ina). On the second day of therapy, pre-test data were taken, and post-test data were gathered after the intervention. Statistical test outcomes The MoCA-Ina score changed considerably between the intervention and control groups (p = 0.000 and p = 0.003, respectively). Several tests determined that the score was p = 0.017; the score after the intervention was substantially different between the two groups. It means cognitive function improves after exercise in the acute phase.

Keywords: Cognitive disorder; Cognitive function; Early cognitive training; Stroke.

Copyright © 2023. Published by Elsevier Inc.

PubMed Disclaimer

Similar articles

• Synergistic effects of aerobic exercise and cognitive training on cognition, physiological markers, daily function, and quality of life in stroke survivors with cognitive decline: study protocol for a randomized controlled trial. Yeh TT, Wu CY, Hsieh YW, Chang KC, Lee LC, Hung JW, Lin KC, Teng CH, Liao YH. Yeh TT, et al. Trials. 2017 Aug 31;18(1):405. doi: 10.1186/s13063-017-2153-7. Trials. 2017. PMID: 28859664 Free PMC article. Clinical Trial.
• The Active Ingredient of Cognitive Restoration: A Multicenter Randomized Controlled Trial of Sequential Combination of Aerobic Exercise and Computer-Based Cognitive Training in Stroke Survivors With Cognitive Decline. Yeh TT, Chang KC, Wu CY. Yeh TT, et al. Arch Phys Med Rehabil. 2019 May;100(5):821-827. doi: 10.1016/j.apmr.2018.12.020. Epub 2019 Jan 9. Arch Phys Med Rehabil. 2019. PMID: 30639273 Clinical Trial.
• Clinical efficacy of aerobic exercise combined with computer-based cognitive training in stroke: a multicenter randomized controlled trial. Yeh TT, Chang KC, Wu CY, Chen CJ, Chuang IC. Yeh TT, et al. Top Stroke Rehabil. 2022 May;29(4):255-264. doi: 10.1080/10749357.2021.1922045. Epub 2021 Aug 2. Top Stroke Rehabil. 2022. PMID: 34340637 Clinical Trial.
• Occupational therapy for cognitive impairment in stroke patients. Gibson E, Koh CL, Eames S, Bennett S, Scott AM, Hoffmann TC. Gibson E, et al. Cochrane Database Syst Rev. 2022 Mar 29;3(3):CD006430. doi: 10.1002/14651858.CD006430.pub3. Cochrane Database Syst Rev. 2022. PMID: 35349186 Free PMC article. Review.
• The effects of exercise intervention on cognition and motor function in stroke survivors: a systematic review and meta-analysis. Li W, Luo Z, Jiang J, Li K, Wu C. Li W, et al. Neurol Sci. 2023 Jun;44(6):1891-1903. doi: 10.1007/s10072-023-06636-9. Epub 2023 Feb 14. Neurol Sci. 2023. PMID: 36781567 Review.
• Search in MeSH

Related information

Linkout - more resources, full text sources.

• Elsevier Science
• MedlinePlus Health Information
• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.