Elsevier

Journal of Clinical Epidemiology

Volume 103, November 2018, Pages 82-91
Journal of Clinical Epidemiology

Original Article
A methodological framework for model selection in interrupted time series studies

https://doi.org/10.1016/j.jclinepi.2018.05.026Get rights and content

Abstract

Interrupted time series (ITS) is a powerful and increasingly popular design for evaluating public health and health service interventions. The design involves analyzing trends in the outcome of interest and estimating the change in trend following an intervention relative to the counterfactual (the expected ongoing trend if the intervention had not occurred). There are two key components to modeling this effect: first, defining the counterfactual; second, defining the type of effect that the intervention is expected to have on the outcome, known as the impact model. The counterfactual is defined by extrapolating the underlying trends observed before the intervention to the postintervention period. In doing this, authors must consider the preintervention period that will be included, any time-varying confounders, whether trends may vary within different subgroups of the population and whether trends are linear or nonlinear. Defining the impact model involves specifying the parameters that model the intervention, including for instance whether to allow for an abrupt level change or a gradual slope change, whether to allow for a lag before any effect on the outcome, whether to allow a transition period during which the intervention is being implemented, and whether a ceiling or floor effect might be expected. Inappropriate model specification can bias the results of an ITS analysis and using a model that is not closely tailored to the intervention or testing multiple models increases the risk of false positives being detected. It is important that authors use substantive knowledge to customize their ITS model a priori to the intervention and outcome under study. Where there is uncertainty in model specification, authors should consider using separate data sources to define the intervention, running limited sensitivity analyses or undertaking initial exploratory studies.

Introduction

Interrupted time series (ITS) has become a core study design for the evaluation of public health interventions and health policies [1]. The design takes advantage of natural experiments whereby an intervention is introduced at a known point in time and a series of observations on the outcome of interest exist both before and after the intervention. The effect of the intervention is estimated by examining any change following the intervention compared with the “counterfactual”, represented by the expected ongoing trend in the absence of the intervention (Figure 1) [2]. ITS involves a pre–post comparison, controlling for the counterfactual baseline trend, within the same population; therefore, it can be used in situations where no control population is available [3], [4]. This also has the advantage that selection bias and confounding due to group differences, which threaten the reliability of nonrandomized controlled designs, are rarely a problem in ITS studies [2], [3]. Furthermore, because ITS incorporates the underlying trend, it controls for short-term fluctuations, secular trends, and regression to the mean [3], [4]. The basic ITS design also has limitations; for example, there is the potential for history bias whereby other events concurrent to the intervention may be responsible for an observed effect. In addition, instrumentation effects can occur if there are changes in the way the outcome is measured over time [3]. Previous studies have described these strengths and limitations of ITS in more detail and have provided guidance on its application [2], [4], [5]. Furthermore, methodological publications have discussed effective approaches for limiting the risk of history bias, including controlled ITS designs and multiple baseline designs [6], [7], [8].

One area that has not been covered in detail in the existing literature is how researchers should approach specifying the ITS model used in the analysis. As discussed previously, the ITS design involves making a comparison between the outcome observed following the intervention and the counterfactual. This comparison reduces to two key questions that define the estimated effect of the intervention [2]. First, how is the counterfactual defined? This involves modeling the preintervention trend. Second, how is the impact model of the intervention defined? That is, what type of effect do we hypothesize that the intervention will have on the outcome (such as whether the effect is gradual or abrupt, immediate or lagged)? This involves parameterizing the effect of the intervention relative to the counterfactual. Multiple alternative approaches exist to defining the counterfactual and the intervention impact model and inappropriate model selection could bias results, yet ITS studies often fail to provide a clear justification for their choice of modeling approach [9].

In this article, we suggest approaches to ensure that model specification is objective and appropriate to the intervention and outcome under investigation. The first section discusses the factors that contribute to defining the counterfactual and the second, the factors that contribute to defining the impact model. For each of these sections, we use illustrative examples from a recent ITS study of the impact of major reforms to the English National Health Service on hospital activity (described in Box 1) [10] to highlight the pitfalls of incorrect model specification and then provide a framework for a suggested approach to select the model. Finally, we also discuss sensitivity analysis and other approaches to dealing with uncertainty in model specification.

Case study

To illustrate the strengths and limitations of different approaches to model specification we use data from a recent study evaluating the impact of the 2012 Health and Social Care Act in England on hospital admissions and outpatient specialist visits [10]. This policy aimed to involve general practitioners (GPs) in commissioning (planning and purchasing) secondary care through the establishment of GP-led Clinical Commissioning Groups. GP-led commissioning is expected to reduce health care costs by shifting care away from secondary care to primary and community settings [11]. We therefore hypothesized that the reforms would result in a relative reduction in secondary care activity (inpatient admissions and outpatient visits). The health and social care act was enacted in April 2012, there was then a 12 month period during which the Clinical Commissioning Groups worked alongside the existing health care administrative bodies before taking over fully independent commissioning in April 2013. We had quarterly data on all NHS hospital admissions and outpatient visits between the second quarter of 2007 and the final quarter of 2015. More details about the intervention and the data can be found in the original study [10].

Section snippets

Defining the counterfactual

A key step in ITS analysis is to predict how the outcome would have continued over time if no intervention had been implemented, referred to as the “counterfactual” scenario. It is not possible to observe the intervention both being implemented and not being implemented in the same population at the same time. The true counterfactual is therefore never known and therefore inferring causality is rarely possible. Evaluation design centers on creating the best approximation of the true

Defining the impact of the intervention

As described previously the effect estimate in an ITS study is a measure of the level and/or trend change in the outcome after an intervention. We have discussed how the trend is defined; the next step is to define how the intervention and its potential impacts are modeled. Different interventions can have different impacts on an outcome: for example, mandatory helmet legislation might be expected to have an abrupt effect on cycle head injuries, whereas an educational programme on cycle safety

Dealing with uncertainty in model selection

So far in this article, we have emphasized the need to carefully define the preintervention trend and the intervention impact model according to the specific intervention, outcome, and data being used in the study. Often, however, the single best approach is difficult to define, in particular for novel interventions that have not previously been studied and when analyzing the public health effects of unplanned events. Below we discuss some approaches to dealing with uncertainty in model

Conclusion

ITS is one of the most rigorous quasi-experimental designs and avoids many of the sources of bias and confounding of other observational studies [1], [3]. Nevertheless, we have demonstrated the risk that incorrect modeling of either the underlying trend or the impact of the intervention has for generating misleading results. The threat to validity is greatest when more flexible or data-driven models are chosen as this increases the likelihood of detecting false positive effects due to

References (47)

  • A. Linden

    Conducting interrupted time series analysis for single and multiple group comparisons

    Stata J

    (2015)
  • A. Biglan et al.

    The value of interrupted time-series experiments for community intervention research

    Prev Sci

    (2000)
  • J.A. Lopez Bernal et al.

    The use of controls in interrupted time series studies of public health interventions

    Int J Epidemiol

    (2018)
  • C.R. Ramsay et al.

    Interrupted time series designs in health technology assessment: lessons from two systematic reviews of behavior change strategies

    Int J Technol Assess Health Care

    (2003)
  • J.A. Lopez Bernal et al.

    Association between the 2012 health and social care act and specialist visits and hospitalisations in England: a controlled interrupted time series analysis

    PLoS Med

    (2017)
  • J.A. Smith et al.

    GP led commissioning: time for a cool appraisal

    BMJ

    (2012)
  • U. Högberg et al.

    Secular trends in maternal mortality in Sweden from 1750 to 1980

    Bull World Health Organ

    (1986)
  • A. Gasparrini et al.

    On the relationship between smoking bans and incidence of acute myocardial infarction

    Eur J Epidemiol

    (2009)
  • D. Coggon et al.

    Epidemiology for the Uninitiated

    (2009)
  • V. Hall et al.

    Olympics and Paralympics 2012 mass gathering in London: time-series analysis shows no increase in attendances at sexual health clinics

    Sex Transm Infect

    (2015)
  • F. Barone-Adesi et al.

    Effects of Italian smoking regulation on rates of hospital admission for acute coronary events: a country-wide study

    PLoS One

    (2011)
  • A.Y. Walley et al.

    Opioid overdose rates and implementation of overdose education and nasal naloxone distribution in Massachusetts: interrupted time series analysis

    BMJ

    (2013)
  • K. Bhaskaran et al.

    Time series regression studies in environmental epidemiology

    Int J Epidemiol

    (2013)
  • Cited by (0)

    Conflict of interest: none.

    Funding: This study was funded by a UK Medical Research Council Population Health Scientist Fellowship awarded to JLB—Grant Ref: MR/L011891/1.

    View full text