Power sample size

What?

“How large should my sample be?”

This is one of the most common questions a statistician is asked. Sample size and statistical power (the probability of correctly rejecting the null hypothesis when there truly is a difference in outcome between treatment and control groups) are commonplace topics of discussion when dealing with primary studies. IPD meta-analyses are often conducted to answer research questions that primary studies did not consider or were not powered to answer. However, sample size and power are often ignored when planning an IPD meta-analysis. This may be because it is assumed that by combining many primary studies, we will achieve increased power. While it is true that power will be increased, IPD meta-analysis does not guarantee sufficient statistical power to answer the research question of interest.

IPD meta-analyses are commonly undertaken to answer the question of whether a patient-level characteristic modifies a treatment effect, to identify subgroups of patients who may be at greater benefit (or harm) than others. Such stratified medicine is a major interest of clinical decision-makers and pharmaceutical companies, looking to identify those populations in whom treatment is more effective (or less harmful). A single trial is usually underpowered for this purpose. Brookes et al. show that if a single trial has 80% power to detect a particular treatment effect (across all patients), then its power to detect an interaction (with a binary covariate) with the same magnitude as the overall treatment effect will only be 29%. To ensure 80% power to detect the interaction, the sample size in a single trial needs to be increased by approximately four times. Furthermore, to have 80% power to detect an interaction term half the size of the overall treatment effect there needs to be an approximately 16-fold increase in sample size.

Why?

IPD meta-analyses are both time-consuming and expensive to perform, requiring significant resources to obtain, clean and harmonise the IPD from relevant trials before then synthesising them; a process that can take months or even years. Therefore, before embarking on an IPD project, researchers and funders should ensure that it is likely to be worth the effort. In particular, how many studies are likely to provide their IPD and, based on this, what is the potential power of the planned IPD meta-analysis? In our experience, power calculations and sample size justifications are rarely reported in IPD meta-analysis protocols or publications. Researchers are perhaps grateful for whatever IPD can be obtained, and appeal to any IPD meta-analysis adding value over a single trial. However, if it was known in advance that IPD from a particular number of studies would only increase power to 50%, then researchers and funders may think twice before undertaking the IPD project. Conversely, if a potential IPD meta-analysis increases the power to over 80%, then funders will be reassured that the IPD project is worth resourcing.

How?

Sample size determination is never easy, and formal power calculations for an IPD meta-analysis are particularly difficult as they depend on many factors, which perhaps explains why they are currently neglected. The IPD cannot be considered as coming from a single trial, and thus sample size calculations must account for the clustering of patients within trials and the potential heterogeneity (e.g. in baseline risk and treatment effects) between-trials. Also, the power depends on the choice and specification of analysis model (e.g. covariates to be included, number of parameters, magnitude of effects), and the parameter estimation method, amongst other factors.

Several approaches have been proposed to calculate the power to detect a subgroup effect in an IPD meta-analysis study. These approaches fall into two general categories: analytical solutions and simulation-based solutions. Simmonds et al., Kovalchik et al. and Riley et al. have each proposed algebraic solutions to estimate the power of an IPD meta-analysis. These simple algebraic solutions are computationally fast but are not straightforward unless simplifying assumptions are made. For this reason, Kontopantelis et al. and Ensor et al. have also proposed simulation-based approaches, where IPD meta-analysis datasets are simulated multiple times based on a chosen data-generating mechanism (including numbers of studies, effect sizes, and heterogeneity), and then a chosen IPD meta-analysis model is applied to each dataset, with subsequent results (e.g. estimates and confidence intervals) summarised over the multiple analyses. In particular, the proportion of all simulations that give a p-value < 0.05 can be calculated, to give an estimate of the power.

Information needed to inform power calculation for IPD meta-analysis & where to find it.

Typically, the aggregate information (summary statistics) available in trial publications and reports can be used to inform the values of parameters needed for power calculation using either analytical or simulation-based approaches. As with power calculation for single studies, a range of values may be used based on the available evidence and clinical knowledge. Below are some examples of inputs required for power calculations for an IPD meta-analysis of randomised trials, marked with A or S depending on whether they are required for analytical or simulation-based approaches, respectively.

When considering the power of a summary (overall) treatment effect:

• A/S: Significance level

• S: Number of IPD meta-analysis datasets to generate (recommend at least 1000)

• A/S: Number of trials in the IPD meta-analysis

• A/S: Number of patients in each trial, in treatment and control groups

• A: For each trial, events in each group for a binary or time-to-event outcome. Mean outcome in each group for a continuous outcome

• S: Method for estimating the treatment effect in each study separately

• S: Between-trial distribution and magnitude of treatment effects, e.g. normal with a particular mean (summary) effect and between-trial variance (plus any between-trial correlation of baseline risks and treatment effects, if considered relevant)

• A/S: Magnitude of residual variance in each trial

• S: IPD meta-analysis model to pool effect estimates: e.g. fixed effect model or random effects model

• S: Approach to derive confidence intervals and p-values (e.g. standard normal-based method, Hartung-Knapp Sidik-Jonkman, etc.)

• S: Expected treatment effect (if interest lies in power to detect overall treatment effect)

Additionally, when considering the power of a treatment-covariate interaction:

• A/S: Expected subgroup effect

• S: Analysis model and method for estimating the interaction effect in each study separately

• A/S: Either covariate means/proportions (and variances) for each group in each study. Or distribution and magnitude of covariate values in each trial, e.g. normal with chosen mean and variance for a continuous covariate, or Bernoulli for a binary covariate with a chosen probability of being a 1

• A/S: Either subgroup covariate means/proportions (and variances) for each group in each study. Or between-trial distribution and magnitude of treatment-covariate interaction effect, e.g. normal with a particular (summary) mean effect and between-trial variance

• S: Magnitude of any non-proportional hazards in interaction effect

Software packages available for power calculation for IPD meta-analysis.

Stata software

• ipdpower package
  • Author: E. Kontopantelis
  • Description: A simulation-based command that calculates power for complex mixed effects two-level data structures
  • Outcomes: Continuous, Binary or Count outcomes
  • Subgroup covariate: Continuous or Binary covariate
  • Inputs: Manual user-defined inputs

R software

• ipdmeta package which contains the package ipd.sep()
  • Author: S. Kovalchik
  • Description: The function estimates the power of an IPD meta-analysis to detect a specified subgroup effect (covariate-treatment interaction) based on summary statistics
  • Outcomes: Binary or Continuous outcomes
  • Subgroup covariate: Binary or Continuous covariate
  • Inputs: Manual user-defined inputs or aggregate dataset as input

• SIMR package
  • Author: P. Green, C. MacLeod, P. Alday
  • Description: A more general package, not specifically designed for IPD meta-analysis. Calculates power for generalised linear mixed models, using simulation. Designed to work with models fit using the 'lme4' package.
  • Outcomes: Binary, Continuous, Count
  • Subgroup covariate: Binary or Continuous
  • Inputs: Manual user-defined inputs, including user-written model objects and parameters

For more details, refer to the help functions using ??package in R or help(package) in R or Stata.

Aims

In this example, our aim is to reflect the process researchers go through when considering or planning an IPD meta-analysis project. We assume that a clinical question has been identified and an IPD meta-analysis project is desired to address it. Additionally, a set of trials has been identified (and potentially promised their IPD) and aggregate data (summary statistics) for these trials have been published. The researchers want to know, in advance of collecting IPD, whether an IPD meta-analysis of these trials is likely to be powered to answer the clinical question at hand.

Background

Here we walkthrough how to use the online power calculator. The example used is based on a systematic review performed by Thangaratinam et al. to investigate the effects of weight management interventions on maternal and fetal outcomes. One of the primary outcomes was maternal weight gain and their aggregate data meta-analysis of 30 randomised trials showed a significant average reduction in weight gain of 0.97kg (95% CI: 0.34kg to 1.60kg reduction) for lifestyle interventions compared with control. However, there was a large amount of between-study heterogeneity, with an I-squared statistic of 87% and tau-squared of 1.87. Therefore, a major recommendation of Thangaratinam et al. was that an “IPD meta-analysis is needed to provide robust evidence on the differential effect of intervention in various groups based on BMI, age, parity, socioeconomic status and medical conditions in pregnancy”. That is, IPD was needed to examine potential treatment-covariate interactions.

In response to this, in 2012 the Weight Management in Pregnancy International IPD Collaboration (i-WIP) was established to share IPD from multiple randomised trials, and the National Institute for Health Research (NIHR) Health Technology Assessment (HTA) programme subsequently funded the project. At the time of developing the grant application, 14 of the trials (containing 1183 patients) included in the aforementioned aggregate data meta-analysis had provisionally agreed to provide their IPD. Summary data for these 14 trials is provided in the link below, including information about the weight gain in each treatment group, and the distribution of baseline BMI values.

• Online calculator (hosted offsite)
• The aggregate dataset (CSV file) used in the below example walkthrough:

References:

• Thangaratinam S, Rogozinska E, Jolly K, Glinkowski S, Duda W. Interventions to reduce or prevent obesity in pregnant women: a systematic review. Health Technology Assessment. 2012;16(31):191.
• Ensor J, Burke DL, Snell KIE, Hemming K, Riley RD. Simulation-based power calculations for planning a two-stage individual participant data meta-analysis: with application to randomised trials with a continuous outcome. BMC Medical Research Methodology. 2018. 18:41.

Summary of key literature on power calculation for IPD meta-analysis

General:

• Chapter 12. Power Calculations for Planning an IPD Meta-analysis. Richard D. Riley, Joie Ensor. In: Riley RD, Tierney J, Stewart LA (Eds). Individual Participant Data Meta-Analysis: A Handbook for Healthcare Research. Wiley 2021 (in-press)
• Brookes ST, Whitely E, Egger M, Smith GD, Mulheran PA and Peters TJ. Subgroup analyses in randomized trials: risks of subgroup-specific analyses; power and sample size for the interaction test. J Clin Epidemiol. 2004; 57: 229-36.

Algebraic solutions:

• Simmonds MC and Higgins JP. Covariate heterogeneity in meta-analysis: criteria for deciding between meta-regression and individual patient data. Stat Med. 2007; 26: 2982-99.
• Kovalchik SA. Aggregate-data estimation of an individual patient data linear random effects meta-analysis with a patient covariate-treatment interaction term. Biostatistics. 2013; 14: 273-83.
• Kovalchik SA and Cumberland WG. Using aggregate data to estimate the standard error of a treatment-covariate interaction in an individual patient data meta-analysis. Biom J. 2012; 54: 370-84.
• Chapter 12. Power Calculations for Planning an IPD Meta-analysis. Richard D. Riley, Joie Ensor. In: Riley RD, Tierney J, Stewart LA (Eds). Individual Participant Data Meta-Analysis: A Handbook for Healthcare Research. Wiley 2021 (in-press)

Simulation approaches:

• Kontopantelis E, Springate DA, Parisi R and Reeves D. Simulation-Based Power Calculations for Mixed Effects Modeling: ipdpower in Stata. 2016. 2016; 74: 25.
• Ensor J, Burke DL, Snell KIE, Hemming K, Riley RD. Simulation-based power calculations for planning a two-stage individual participant data meta-analysis: with application to randomised trials with a continuous outcome. BMC Medical Research Methodology. 2018. 18:41.