Naive pooling of RCTs and non-randomized studies in network meta-analyses is common and problematicImproving the Quality of Evidence Used in Medical Network Meta-Analyses

BackgroundNetwork meta-analysis (NMA) can synthesize direct and indirect evidence across multiple interventions and is increasingly used to inform comparative-effectiveness decisions. When randomized controlled trials (RCTs) and non-randomized studies of interventions (NRSI) are jointly analyzed, naïve pooling of design-dissimilar evidence without design-aware modelling may violate exchangeability and distort treatment effect estimates and rankings.Methods and analysisWe will conduct a meta-epidemiological study of published clinical NMAs identified through systematic searches of Ovid MEDLINE(R) ALL, Embase via Ovid, EBM Reviews-Cochrane Database of Systematic Reviews via Ovid, and Web of Science Core Collection. Eligible reports will be full-text articles published between 1 January 2020 and 31 December 2025 that included both RCTs and at least one comparative NRSI in the review evidence base. The primary outcome will be the prevalence of naïve pooling in the main analysis. We will apply a prespecified four-category decision framework to distinguish unacknowledged naïve pooling, acknowledged but unmitigated naïve pooling, naïve main analysis with post hoc quantitative mitigation, and no naïve pooling in the main analysis. We will estimate prevalence with exact binomial 95% confidence intervals, assess whether naïve pooling has become more or less common over time, explore characteristics associated with naïve pooling or with not using quantitative methods to address design-related bias, summarize reporting completeness, protocol or registration availability, prespecification of design-handling methods, and availability of extractable paired alternative results, and evaluate empirical changes in effect direction, null-value crossing or statistical significance, and treatment ranking in articles reporting paired mixed-design and RCT-restricted or design-aware results.DiscussionThis study will provide contemporary evidence on how published NMAs handle mixed randomized and non-randomized evidence, distinguish lack of risk recognition from failure to implement corrective analyses, and quantify how often naïve pooling may change conclusions. The protocol includes prespecified procedures to describe reporting completeness and to compare articles with and without extractable paired alternative analyses. The findings may inform methodological training, journal peer review, and evidence appraisal in guideline development.Systematic review registrationhttps://doi.org/10.17605/OSF.IO/AM2T6, Unique Identifier: 10.17605/OSF.IO/AM2T6.