Health Research Reporting Guidelines, Study Execution Manuals, Critical Appraisal, Risk of Bias, & Non-reporting Biases

Cochrane Handbook for Systematic Reviews of Interventions

Chapter 13: Assessing risk of bias due to missing results in a synthesis focuses on the causes of missing evidence, such as studies that aren't published or non-significant outcomes aren't reported. It then describes a framework for describing the assessment of missing results. 

The language on this topic has changed over the years.

• PRISMA (2009) referred to this as "risk of bias across studies"
• PRISMA-P (2015; Item 16) refers to this as "meta-biases"
• PRISMA 2020 (Item 14) refers to this as "reporting bias assessment"
• The Cochrane Handbook also refers to these as "non-reporting biases"

While the terminology has changed, the intent is the same:

• Determine what evidence is missing in the final analysis; and
• Determine how that impacts the summary of evidence.

Where are these reported in an SR?

Any systematic deviation that results in a potential skewing away from the truth of the results should be evaluated and reported. The PRISMA reporting guidelines for a protocol and for the manuscript all include items for assessing these types of biases.

• Item #16: Meta-biases (PRISMA-P) (See pages 17-18 of the PRISMA-P E&E)
  • Item #16: Specify any planned assessment of meta-bias(es) (sometimes called non-reporting biases across studies)
• Item #14: Reporting bias assessment (PRISMA 2020) (See page 17 of the PRISMA E&E)
  • Item #14: Describe any methods used to assess risk of bias due to missing results in a synthesis (arising from reporting biases) (See the grey box at the bottom of page 18 of the PRISMA E&E)
• Item #21: Reporting biases (PRISMA 2020) (See page 25 of the PRISMA E&E)
  • Item #21: Present assessments of risk of bias due to missing results (arising from reporting biases) for each synthesis assessed.

What are non-reporting biases? 

Non-reporting biases encompass a variety of categories. The Catalogue of Bias (Catalogue of Biases. Richards GC, Onakpoya IJ. Reporting biases. In: Catalog Of Bias 2019) refers to reporting bias as, "A systematic distortion that arises from the selective disclosure or withholding of information by parties involved in the design, conduct, analysis, or dissemination of a study or research findings". It lists the following types of reporting biases:

• Seven types of reporting biases (Catalogue of Biases)
  1. Publication bias
  2. Time-lag bias
  3. Multiple (duplicate) publication bias
  4. Location bias
  5. Citation bias
  6. Language bias 
  7. Outcome reporting bias

This page will be updated as new tools and resources become available.

Risk of Bias Tools (Cochrane)

• ROB ME (Risk Of Bias due to Missing Evidence in a synthesis) (October 2023)
  • A cribsheet summarizing the tool.
  • A template for completing the assessment.
  • The full guidance document (preliminary tool). (October 2020 version)
    • Full guidance for the updated version is not available yet.

Chiocchia V, Nikolakopoulou A, Higgins JPT, Page MJ, Papakonstantinou T, Cipriani A, Furukawa TA, Siontis GCM, Egger M, Salanti G. ROB-MEN: a tool to assess risk of bias due to missing evidence in network meta-analysis. BMC Med. 2021 Nov 23;19(1):304. doi: 10.1186/s12916-021-02166-3. PMID: 34809639; PMCID: PMC8609747.