BMC Medical Research Methodology · Published 2026-06-15 · DOI 10.1186/s12874-026-02868-3
Jiayi Huang, Xinxin Deng, Liying Zhou, Junliang Tao, Cui Liang, Kehu Yang, Xiuxia Li
Abstract Background Evidence quality assessment is critical for informed public health decision-making, but manual approaches are time-consuming and subject to variability. Automated support tools have been proposed to improve efficiency and consistency, yet their current status has not been comprehensively or systematically mapped. Objective To identify or map the characteristics, performance, and limitations of existing automated tools for evidence quality assessment. Methods Following the JBI methodology and PRISMA-ScR checklist, we searched 6 English and 4 Chinese databases from their inception to February 9, 2025, to identify studies evaluating automated tools for evidence quality assessment. Eligible studies included original research on tool development, application, or validation. Study characteristics (e.g., year, country, design, tool type, technical features, reliability, and validity) were extracted and summarized descriptively. Results Twenty studies were included, most from the United Kingdom (30%), Canada (25%), and Australia (15%). Observational designs predominated (75%), with only 10% randomized controlled trials (RCTs). Twelve distinct tools were identified, of which 65% were publicly available. 58% of the tools were developed for RCTs, while 50% remained experimental and required human oversight. Reported outcomes focused on sensitivity, specificity, precision, efficiency, and consistency. Despite promising results, external validity and scalability were limited. Conclusion Automated tools for evidence quality assessment show potential to enhance efficiency and consistency but remain restricted in applicability. Current tools are often tailored to clinical trials and require human supervision. Broader adaptation and rigorous validation are needed before such tools can be widely integrated into public health decision-making.
Abstract from DOAJ. Public domain (CC0 1.0).
Read the article at the publisher →
Huang, J., Deng, X., Zhou, L., et al. (2026). Automated tools for evidence quality assessment: a scoping review. BMC Medical Research Methodology. https://doi.org/10.1186/s12874-026-02868-3