Critical appraisal and reporting

Biases

• An important part of an IPD meta-analysis project is to examine the robustness of IPD meta-analysis results to potential biases that could occur

• Publication-related biases hide relevant trials and data, often those with ‘negative’ findings (e.g. statistically non-significant results). As for any type of review, this could lead to IPD meta-analysis results being biased toward favourable effects

• Availability bias is a concern if IPD are obtained from only a subset of the trials from which requested, and the provision of IPD is linked to trial findings. This may also make the IPD meta-analysis results biased, although the direction of bias is hard to predict

• These issues may lead to small-study effects in the IPD meta-analysis, where smaller trials exhibit different (often greater) effect estimates than larger trials. Small-study effects are revealed by asymmetry in a funnel plot, but they may also arise due to factors that cause between-trial heterogeneity

• The impact of availability bias can be investigated by utilising aggregate data from non-IPD trials in sensitivity analyses; however, obtaining suitable aggregate data may be problematic, and the process may simply reinforce why IPD was required in the first place

• Other sensitivity analyses may be needed to examine bias concerns. In particular, analyses restricted to trials at low risk of bias investigate whether the meta-analysis conclusions are influenced by trial quality

References:

• Ahmed I, Sutton AJ, Riley RD. Assessment of publication bias, selection bias, and unavailable data in meta-analyses using individual participant data: a database survey. BMJ 2012; 344: d7762.
• Sterne JAC, Sutton AJ, Ioannidis JPA, Terrin N, Jones DR, Lau J, et al. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ 2011; 342:d4002.
• Riley RD. Commentary: Like it and lump it? Meta-analysis using individual participant data. Int J Epidemiol 2010; 39(5):1359-1361.
• Stewart L, Tierney J, Burdett S. Do Systematic Reviews Based on Individual Patient Data Offer a Means of Circumventing Biases Associated with Trial Publications? In: Rothstein HR, Sutton AJ, Borenstein M, editors. Publication Bias in Meta-Analysis: Prevention, Assessment and Adjustments. Chichester, UK.: John Wiley & Sons, Ltd., 2006.
• Clarke MJ, Stewart LA. Obtaining data from randomised controlled trials: how much do we need for reliable and informative meta-analyses? In: Chalmers I, Altman DG, editors. Systematic reviews. London: BMJ Publishing, 1995:37-47.

 

Incorporate aggregate data

• Some eligible trials for the IPD meta-analysis project may not agree to provide their IPD. However, if suitable aggregate data (such as treatment effect estimates and their variances) can be extracted from publications or provided by trial investigators, this aggregate data can be included in the second stage of the two-stage meta-analysis

• The summary results can then be compared to those derived from the IPD-only meta-analysis, which should still be considered the main analysis

• The approach can be implemented within the ipdmetan package. Specifically, the ad() option allows the user to call a separate file of aggregate data results for non-IPD trials, which is utilised in the second stage alongside the aggregate data derived for IPD trials by ipdmetan in the first stage

• It is possible to incorporate aggregate data alongside IPD in a one-stage meta-analysis framework. This requires the specification of two (or more) regression models with shared parameters, also known as hierarchical related regression

• Sometimes it is possible to partially reconstruct the IPD using aggregate data available within trial publications, such as two by two tables, means and standard deviations, or even Kaplan-Meier curves. The general premise is that the reconstructed IPD should match the known summary characteristics of the IPD, in terms of the sample size, number of events, mean values, and correlation amongst covariates, and so forth

• A major concern is that the aggregate data (or recreated IPD) from non-IPD trials is not subject to the same data checking and quality standards as that from IPD trials. Similarly, non-IPD trials cannot be standardised to the same extent of IPD trials, and so may introduce more heterogeneity in the meta-analysis, and simply reinforce why IPD were desired in the first place

References:

• Riley RD, Steyerberg EW. Meta-analysis of a binary outcome using individual participant data and aggregate data. Research Synthesis Methods 2010; 1 (1): 2-19.
• Riley RD, et al. Evidence synthesis combining individual patient data and aggregate data: a systematic review identified current practice and possible methods. J Clin Epidemiol 2007, 60:431-439.
• Riley RD, et al. Meta-analysis of continuous outcomes combining individual patient data and aggregate data. Stat Med 2008; 27:1870-93.

Reporting

• As with all areas of research, systematic reviews and meta-analyses of IPD could be better reported, making it easier for readers to understand, critique, and implement findings. Standard PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines are geared toward systematic reviews based on aggregate data and so lack reference to some important aspects of the IPD approach

• PRISMA-IPD, and its associated checklist and flow diagram, were developed to provides guidelines for reporting systematic reviews and meta-analyses of IPD

• Compared with standard PRISMA, the PRISMA-IPD checklist includes 3 new items that address (1) methods of checking the integrity of the IPD (such as pattern of randomisation, data consistency, baseline imbalance, and missing data), (2) reporting any important issues that emerge, and (3) exploring variation (such as whether certain types of individual benefit more from the intervention than others). A further additional item was created by reorganisation of standard PRISMA items relating to interpreting results, and wording was modified in 23 items to reflect the IPD approach

Reference:

• Stewart LA, Clarke M, Rovers M, Riley RD, Simmonds M, Stewart G, et al. Preferred Reporting Items for Systematic Review and Meta-Analyses of individual participant data: the PRISMA-IPD Statement. Jama. 2015; 313:1657-1665.

Critical appraisal

• The process of collecting, checking, and analysing IPD is more complex than for aggregate data, and not all IPD meta-analyses are done to the same standard, making it difficult for researchers, clinicians, patients, policy makers, funders, and publishers to judge their quality

• Tierney et al. provide step-by-step guide to help reviewers and users of IPD meta-analyses to understand them better and recognise those that are well designed and conducted and so help ensure that policy, practice, and research are informed by robust evidence about the effects of interventions

Key questions to ask when critically appraising an IPD meta-analysis project are:

• Is it part of a systematic review?
• Were all eligible trials identified?
• Were IPD obtained from most trials?
• Was the integrity of the IPD checked?
• Were the analyses prespecified in detail?
• Was the risk of bias of included trials assessed?
• Were the methods of analysis appropriate?

Reference:

• Tierney JF, et al. Individual participant data (IPD) meta-analyses of randomised controlled trials: guidance on their use. PLOS Med 2015; 12(7):e1001855.