Skip to Main Content

Health Research Reporting Guidelines, Study Execution Manuals, Critical Appraisal, Risk of Bias, & Non-reporting Biases

What is Risk of Bias (RoB)?

Cochrane Handbook for Systematic Reviews of Interventions

Risk of Bias (RoB) is a metric used when conducting systematic reviews. The Cochrane Collaboration considers 2 places in which RoB should be considered: during the assessment of individual studies (i.e. internal validity) and when assessing the results of studies during the meta-analysis process. Chapter 7 of the Cochrane Handbook explains more about RoB, regardless of study methodology. It defines a bias as "a systematic error, or deviation from the truth, in results).” Chapter 8 walks through the process of assessing RoB in a randomized trial. NOTE: RoB in this instance refers to individual STUDY risk of bias and not non-reporting bias(es)/meta-bias(es). 

"Methodological quality refers to critical appraisal of a study or systematic review and the extent to which study authors conducted and reported their research to the highest possible standard. Bias refers to systematic deviation of results or inferences from the truth. These deviations can occur as a result of flaws in design, conduct, analysis, and/or reporting. It is not always possible to know whether an estimate is biased even if there is a flaw in the study; further, it is difficult to quantify and at times to predict the direction of bias. For these reasons, reviewers refer to ‘risk of bias’ (Chapter 8)." Chapter V: Overviews of Reviews (below the table).

The 5 domains of study RoB as identified in the Cochrane Handbook are:

  1. Bias arising from the randomization process;
  2. Bias due to deviations from intended interventions;
  3. Bias due to missing outcome data;
  4. Bias in measurement of the outcome; and
  5. Bias in selection of the reported result.

Assessing the Risk of Bias in Systematic Reviews of Health Care Interventions

This 2017 online book from the Agency for Healthcare Research and Quality (AHRQ) provides recommendations for the use of RoB including: when determining focus and RoB scope of RoB; determining the domains that should be utilized (or not) when assessing RoB; determining appropriate tools for RoB; and conduct of RoB.

This is an excellent resource, as is the Cochrane Handbook, but this book looks at RoB without regard to a specific tool. Rather, it discusses the various approaches to RoB, why they did or did not work properly, and then provides a series of recommendations. Even if you choose the Cochrane RoB tool (see below), take a look at the AHRQ recommendations to ensure you don't go astray when performing the RoB.

Tables & links


Evaluating non-randomised intervention studies

While older and preceding many (or all) of the RoB tools available, this Health Technology Assessment systematic review (2003) provides a thoughtful and extensive approach to the topic. The objective of this review was "to consider methods and related evidence for evaluating bias in non-randomised intervention studies."

  • Deeks JJ, Dinnes J, D'Amico R, et al. Evaluating non-randomised intervention studies. Health Technol Assess. 2003;7(27):iii-173.

LATITUDES Network of validated tools

LATITUDES Network

“LATITUDES network is a library of validity assessment tools for use in evidence syntheses.  Researchers can register tools under development.  The website also provides access to training resources.”

P Whiting, R Wolff, J Savović, S Mallett, B Devine, the LATITUDES group. LATITUDES network. 2023. URL: https://www.latitudes-network.org/

"LATITUDES was established to increase the robustness of evidence synthesis by improving the process of validity (risk of bias) assessment.  The goals of LATITUDES are to:

  • Improve the overall quality of evidence synthesis
  • Increase use of key risk of bias tools
  • Help people to use tools more effectively
  • Improve incorporation of results of the validity (risk of bias) assessment into the evidence synthesis
  • Disseminate best practice in validity (risk of bias) assessment
  • Prevent duplication of efforts in developing risk of validity (risk of bias) assessment tools"

LATITUDES Library

Comprehensive collection of validated tools. It provides access to more tools on this page. and even provides a means of searching by study type and name.

Which tool should you use?

The format is a jpeg of a flow chart to help select the best tool for your study design. It is still undergoing changes.

Learn about study designs

 

RoB visualization tool

robvis: Risk of Bias Visualization Tool

Developed by: Luke McGuinness, National Institute of Health Research Doctoral Research Fellow in Evidence Synthesis, Bristol Medical School
About: robvis is compatible with several risk of bias assessment tools including: RoB 2, ROBINS-E, ROBINS-I, QUADAS, and other RoB assessment tools

Randomized trials

RoB 2 tool (revised tool for Risk of Bias in randomized trials)

Funded by: Medical Research Council (MRC), University of Bristol
About: The following tools were developed in collaboration with the Cochrane Collaboration. There are currently 3 RoB 2 tools:


Tool to Assess Risk of Bias in Randomized Trials

Produced by: CLARITY Group at McMaster University in conjunction with Evidence Partners (DistillerSR)
About: About CLARITY
Note: This instrument does not asses domains as is common with RoB assessment tools, but rather is a scale (more common among critical appraisal instruments) of 6 items. 


 

Evidence Project Risk of Bias Tool (Word doc)

Produced by: Evidence Project, John Hopkins University
About: The Evidence Project risk of bias tool: assessing study rigor for both randomized and non-randomized intervention studies
Note: This is a RoB tool that can be used with both randomized and non-randomized study designs. "Studies that involve more than one study arm or group include randomized trials, non-randomized trials, case-control studies, cross-sectional studies, serial cross-sectional studies, and cohort studies. Studies that involve only one study group include before-after and time series designs."

  • Kennedy CE, Fonner VA, Armstrong KA, Denison JA, Yeh PT, O'Reilly KR, Sweat MD. The Evidence Project risk of bias tool: assessing study rigor for both randomized and non-randomized intervention studies. Syst Rev. 2019 Jan 3;8(1):3. doi: 10.1186/s13643-018-0925-0. PMID: 30606262; PMCID: PMC6317181.

Non-randomized intervention studies

ROBINS-I tool (Risk Of Bias in Non-randomized Studies - of Interventions)

Funded by: Medical Research Council (MRC), University of Bristol
About: ROBINS-I was developed in collaboration with the Cochrane Collaboration.
Observational study type: Cohort studies


 

Evidence Project Risk of Bias Tool (Word doc)

Produced by: Evidence Project, John Hopkins University
About: The Evidence Project risk of bias tool: assessing study rigor for both randomized and non-randomized intervention studies
Note: This is a RoB tool that can be used with both randomized and non-randomized study designs. "Studies that involve more than one study arm or group include randomized trials, non-randomized trials, case-control studies, cross-sectional studies, serial cross-sectional studies, and cohort studies. Studies that involve only one study group include before-after and time series designs."

  • Kennedy CE, Fonner VA, Armstrong KA, Denison JA, Yeh PT, O'Reilly KR, Sweat MD. The Evidence Project risk of bias tool: assessing study rigor for both randomized and non-randomized intervention studies. Syst Rev. 2019 Jan 3;8(1):3. doi: 10.1186/s13643-018-0925-0. PMID: 30606262; PMCID: PMC6317181.

Exposure studies

ROBINS-E tool (Risk Of Bias in non-randomized Studies - of Exposures for cohort studies)

Funded by: Medical Research Council (MRC), University of Bristol
About: The following tools were developed in collaboration with the Cochrane Collaboration.


OHAT Risk of Bias Rating Tool for Human and Animal Studies

Produced by: US National Institutes of Health National Toxicology Program, Office of Health Assessment and Translation (OHAT)
About: This risk-of-bias tool evaluates internal validity – the assessment of whether the design and conduct of the study compromised the credibility of the link between exposure and outcome (Higgins and Green 2011, IOM 2011, Viswanathan et al. 2012). There are other aspects of a study that will impact its utility for addressing the research question such as external validity – indirectness or applicability, which are addressed elsewhere in the OHAT Approach. In other words, risk of bias addresses the question “Are the results of the study credible?” Whereas indirectness addresses the question “Did the study design address the topic of the evaluation?


RoB-SPEO (Risk of Bias in Studies estimating Prevalence of Exposure to Occupational risk factors)

Development supported by: World Health Organization/International Labour Organization
About: This tools "applies a parallel approach to the evaluation of study quality, or "risk of bias," for human and non-human animal studies, facilitating consideration of potential bias across evidence streams with common terminology and domains."

  • Pega F, Norris SL, Backes C, et al. RoB-SPEO: A tool for assessing risk of bias in studies estimating the prevalence of exposure to occupational risk factors from the WHO/ILO Joint Estimates of the Work-related Burden of Disease and Injury. Environ Int. 2020;135:105039.
  • A Quality of Evidence assessment tool was developed by the same group

Newcastle-Ottawa Scale: Case Control Studies & Cohort Studies

Produced by: University of Newcastle, Australia and University of Ottawa, Canada

  • ManualMS Word.PDF
    • Instructions for both case control and cohort studies
  • ScaleMS Word.PDF
    • Instruments for both case control and cohort studies

Evidence Project Risk of Bias Tool (Word doc)

Produced by: Evidence Project, John Hopkins University
About: The Evidence Project risk of bias tool: assessing study rigor for both randomized and non-randomized intervention studies
Note: This is a RoB tool that can be used with both randomized and non-randomized study designs. "Studies that involve more than one study arm or group include randomized trials, non-randomized trials, case-control studies, cross-sectional studies, serial cross-sectional studies, and cohort studies. Studies that involve only one study group include before-after and time series designs."

  • Kennedy CE, Fonner VA, Armstrong KA, Denison JA, Yeh PT, O'Reilly KR, Sweat MD. The Evidence Project risk of bias tool: assessing study rigor for both randomized and non-randomized intervention studies. Syst Rev. 2019 Jan 3;8(1):3. doi: 10.1186/s13643-018-0925-0. PMID: 30606262; PMCID: PMC6317181.

Tool to Assess Risk of Bias in Cohort Studies

Produced by: CLARITY Group at McMaster University in conjunction with Evidence Partners (DistillerSR)
About: About CLARITY
Note: This instrument does not asses domains as is common with RoB assessment tools, but rather is a scale (more common among critical appraisal instruments) of 8 items. 

Case-control studies

Evidence Project Risk of Bias Tool (Word doc)

Produced by: Evidence Project, John Hopkins University
About: The Evidence Project risk of bias tool: assessing study rigor for both randomized and non-randomized intervention studies
Note: This is a RoB tool that can be used with both randomized and non-randomized study designs. "Studies that involve more than one study arm or group include randomized trials, non-randomized trials, case-control studies, cross-sectional studies, serial cross-sectional studies, and cohort studies. Studies that involve only one study group include before-after and time series designs."

  • Kennedy CE, Fonner VA, Armstrong KA, Denison JA, Yeh PT, O'Reilly KR, Sweat MD. The Evidence Project risk of bias tool: assessing study rigor for both randomized and non-randomized intervention studies. Syst Rev. 2019 Jan 3;8(1):3. doi: 10.1186/s13643-018-0925-0. PMID: 30606262; PMCID: PMC6317181.

Newcastle-Ottawa Scale: Case Control Studies & Cohort Studies

Produced by: University of Newcastle, Australia and University of Ottawa, Canada

  • ManualMS Word.PDF
    • Instructions for both case control and cohort studies
  • ScaleMS Word.PDF
    • Instruments for both case control and cohort studies

Tool to Assess Risk of Bias in Case Control Studies

Produced by: CLARITY Group at McMaster University in conjunction with Evidence Partners (DistillerSR)
About: About CLARITY
Note: This instrument does not asses domains as is common with RoB assessment tools, but rather is a scale (more common among critical appraisal instruments) of 5 items. 

Cross-sectional studies

A scoping review shows that no single existing risk of bias assessment tool considers all sources of bias for cross-sectional studies

Kelly SE, Brooks SPJ, Benkhedda K, MacFarlane AJ, Greene-Finestone LS, Skidmore B, Clifford TJ, Wells GA. A scoping review shows that no single existing risk of bias assessment tool considers all sources of bias for cross-sectional studies. J Clin Epidemiol. 2024 Jun 4:111408. doi: 10.1016/j.jclinepi.2024.111408. Epub ahead of print. PMID: 38844117. (Requires Pitt login)

This review will be used to determine which cross-sectional study risk of bias assessment tools will be incorporated into the LibGuide.

Diagnostic accuracy studies

QUADAS: Tool for the Quality Assessment of Diagnostic Accuracy Studies

Produced by: University of Bristol

QUADAS is the home of two assessment tools.

  • QUADAS-2 is for use in systematic reviews to evaluate the risk of bias and applicability of primary diagnostic accuracy studies, consisting of four key domains:
    • patient selection
    • index test
    • reference standard
    • flow and timing
  • QUADAS-C (C for Comparative)
    • An extension of QUADAS-2
    • Assess RoB in comparative accuracy studies

Prognostication/prediction studies

QUAPAS: Quality Assessment of Prognostic Accuracy Studies (Requires Pitt login)

Lee J, Mulder F, Leeflang M, Wolff R, Whiting P, Bossuyt PM. QUAPAS: An Adaptation of the QUADAS-2 Tool to Assess Prognostic Accuracy Studies. Ann Intern Med. 2022 Jul;175(7):1010-1018. doi: 10.7326/M22-0276. Epub 2022 Jun 14. PMID: 35696685


"Quality In Prognosis Studies" (QUIPS) (Requires Pitt login)

Hayden JA, van der Windt DA, Cartwright JL, Côté P, Bombardier C. Assessing bias in studies of prognostic factors. Ann Intern Med. 2013 Feb 19;158(4):280-6. doi: 10.7326/0003-4819-158-4-201302190-00009. PMID: 23420236.

  • Developed for prognostic factor studies

PROBAST: Prediction model Risk Of Bias ASsessment Tool

Produced by: PROBAST Group
About: Risk of Bias assessment for studies developing, validating, or updating (for example, extending) prediction models, both diagnostic and prognostic

Animal studies

SYRCLE's risk of bias tool for animal studies (preclinical)

Produced by: SYstematic Review Centre for Laboratory animal Experimentation (SYRCLE)
About: SYRCLE’s risk of bias tool for animal studies (BMC Medical Research Methodology open access article)


OHAT Risk of Bias Rating Tool for Human and Animal Studies

Produced by: US National Institutes of Health National Toxicology Program, Office of Health Assessment and Translation (OHAT)
About: This risk-of-bias tool evaluates internal validity – the assessment of whether the design and conduct of the study compromised the credibility of the link between exposure and outcome (Higgins and Green 2011, IOM 2011, Viswanathan et al. 2012). There are other aspects of a study that will impact its utility for addressing the research question such as external validity – indirectness or applicability, which are addressed elsewhere in the OHAT Approach. In other words, risk of bias addresses the question “Are the results of the study credible?” Whereas indirectness addresses the question “Did the study design address the topic of the evaluation?

Other study types

RoBDEMAT: A risk of bias tool and guideline to support reporting of pre-clinical dental materials research and assessment of systematic reviews

Note: I am unable to locate a web site with the full instrument. This article describes the development of the instrument and includes some of the questions.

  • Delgado AH, Sauro S, Lima AF, Loguercio AD, Della Bona A, Mazzoni A, Collares FM, Staxrud F, Ferracane J, Tsoi J, Amato J, Neuhaus KW, Ceballos L, Breschi L, Hannig M, Melo MA, Özcan M, Scotti N, Opdam N, Yamaguchi S, Paris S, Turkun LS, Doméjean S, Rosa V, Palin W, Schwendicke F. RoBDEMAT: A risk of bias tool and guideline to support reporting of pre-clinical dental materials research and assessment of systematic reviews. J Dent. 2022 Dec;127:104350. doi: 10.1016/j.jdent.2022.104350. Epub 2022 Oct 28. PMID: 36341980.

Tool to Assess Risk of Bias in Longitudinal Symptom Research Studies Aimed at the General Population

Produced by: CLARITY Group at McMaster University in conjunction with Evidence Partners (DistillerSR)
About: About CLARITY
Note: This instrument does not asses domains as is common with other RoB assessment tools, but rather is a scale (more common among critical appraisal instruments) of 3 items. 


Risk of Bias Instrument for Cross-Sectional Surveys of Attitudes and Practices

Produced by: CLARITY Group at McMaster University in conjunction with Evidence Partners (DistillerSR)
About: About CLARITY
Note: This instrument does not asses domains as is common with RoB assessment tools, but rather is a scale (more common among critical appraisal instruments) of 5 items.