Risk of Bias (RoB) is a metric used when conducting systematic reviews. The Cochrane Collaboration considers 2 places in which RoB should be considered: during the assessment of individual studies (i.e. internal validity) and when assessing the results of studies during the meta-analysis process. Chapter 7 of the Cochrane Handbook explains more about RoB, regardless of study methodology. It defines a bias as "a systematic error, or deviation from the truth, in results).” Chapter 8 walks through the process of assessing RoB in a randomized trial. NOTE: RoB in this instance refers to individual STUDY risk of bias and not non-reporting bias(es)/meta-bias(es).
"Methodological quality refers to critical appraisal of a study or systematic review and the extent to which study authors conducted and reported their research to the highest possible standard. Bias refers to systematic deviation of results or inferences from the truth. These deviations can occur as a result of flaws in design, conduct, analysis, and/or reporting. It is not always possible to know whether an estimate is biased even if there is a flaw in the study; further, it is difficult to quantify and at times to predict the direction of bias. For these reasons, reviewers refer to ‘risk of bias’ (Chapter 8)." Chapter V: Overviews of Reviews (below the table).
The 5 domains of study RoB as identified in the Cochrane Handbook are:
This 2017 online book from the Agency for Healthcare Research and Quality (AHRQ) provides recommendations for the use of RoB including: when determining focus and RoB scope of RoB; determining the domains that should be utilized (or not) when assessing RoB; determining appropriate tools for RoB; and conduct of RoB.
This is an excellent resource, as is the Cochrane Handbook, but this book looks at RoB without regard to a specific tool. Rather, it discusses the various approaches to RoB, why they did or did not work properly, and then provides a series of recommendations. Even if you choose the Cochrane RoB tool (see below), take a look at the AHRQ recommendations to ensure you don't go astray when performing the RoB.
Tables & links
While older and preceding many (or all) of the RoB tools available, this Health Technology Assessment systematic review (2003) provides a thoughtful and extensive approach to the topic. The objective of this review was "to consider methods and related evidence for evaluating bias in non-randomised intervention studies."
Developed by: Luke McGuinness, National Institute of Health Research Doctoral Research Fellow in Evidence Synthesis, Bristol Medical School
About: robvis is compatible with several risk of bias assessment tools including: RoB 2, ROBINS-E, ROBINS-I, QUADAS, and other RoB assessment tools
Produced by: Bristol Medical School, University of Bristol
About: ROBIS Tool
Additional Resources: Resources including training
Funded by: Medical Research Council (MRC), University of Bristol
About: The following tools were developed in collaboration with the Cochrane Collaboration. There are currently 3 RoB 2 tools:
Produced by: CLARITY Group at McMaster University in conjunction with Evidence Partners (DistillerSR)
About: About CLARITY
Note: This instrument does not asses domains as is common with RoB assessment tools, but rather is a scale (more common among critical appraisal instruments) of 6 items.
Produced by: Evidence Project, John Hopkins University
About: The Evidence Project risk of bias tool: assessing study rigor for both randomized and non-randomized intervention studies
Note: This is a RoB tool that can be used with both randomized and non-randomized study designs. "Studies that involve more than one study arm or group include randomized trials, non-randomized trials, case-control studies, cross-sectional studies, serial cross-sectional studies, and cohort studies. Studies that involve only one study group include before-after and time series designs."
Funded by: Medical Research Council (MRC), University of Bristol
About: ROBINS-I was developed in collaboration with the Cochrane Collaboration.
Observational study type: Cohort studies
Produced by: Evidence Project, John Hopkins University
About: The Evidence Project risk of bias tool: assessing study rigor for both randomized and non-randomized intervention studies
Note: This is a RoB tool that can be used with both randomized and non-randomized study designs. "Studies that involve more than one study arm or group include randomized trials, non-randomized trials, case-control studies, cross-sectional studies, serial cross-sectional studies, and cohort studies. Studies that involve only one study group include before-after and time series designs."
Funded by: Medical Research Council (MRC), University of Bristol
About: The following tools were developed in collaboration with the Cochrane Collaboration.
Produced by: US National Institutes of Health National Toxicology Program, Office of Health Assessment and Translation (OHAT)
About: This risk-of-bias tool evaluates internal validity – the assessment of whether the design and conduct of the study compromised the credibility of the link between exposure and outcome (Higgins and Green 2011, IOM 2011, Viswanathan et al. 2012). There are other aspects of a study that will impact its utility for addressing the research question such as external validity – indirectness or applicability, which are addressed elsewhere in the OHAT Approach. In other words, risk of bias addresses the question “Are the results of the study credible?” Whereas indirectness addresses the question “Did the study design address the topic of the evaluation?
Development supported by: World Health Organization/International Labour Organization
About: This tools "applies a parallel approach to the evaluation of study quality, or "risk of bias," for human and non-human animal studies, facilitating consideration of potential bias across evidence streams with common terminology and domains."
Produced by: University of Newcastle, Australia and University of Ottawa, Canada
Produced by: Evidence Project, John Hopkins University
About: The Evidence Project risk of bias tool: assessing study rigor for both randomized and non-randomized intervention studies
Note: This is a RoB tool that can be used with both randomized and non-randomized study designs. "Studies that involve more than one study arm or group include randomized trials, non-randomized trials, case-control studies, cross-sectional studies, serial cross-sectional studies, and cohort studies. Studies that involve only one study group include before-after and time series designs."
Produced by: CLARITY Group at McMaster University in conjunction with Evidence Partners (DistillerSR)
About: About CLARITY
Note: This instrument does not asses domains as is common with RoB assessment tools, but rather is a scale (more common among critical appraisal instruments) of 8 items.
Produced by: Evidence Project, John Hopkins University
About: The Evidence Project risk of bias tool: assessing study rigor for both randomized and non-randomized intervention studies
Note: This is a RoB tool that can be used with both randomized and non-randomized study designs. "Studies that involve more than one study arm or group include randomized trials, non-randomized trials, case-control studies, cross-sectional studies, serial cross-sectional studies, and cohort studies. Studies that involve only one study group include before-after and time series designs."
Produced by: University of Newcastle, Australia and University of Ottawa, Canada
Produced by: CLARITY Group at McMaster University in conjunction with Evidence Partners (DistillerSR)
About: About CLARITY
Note: This instrument does not asses domains as is common with RoB assessment tools, but rather is a scale (more common among critical appraisal instruments) of 5 items.
Stands for: COnsensus-based Standards for the selection of health Measurement INstruments
Additional information: About the COSMIN initiative
Produced by: University of Bristol
QUADAS is the home of two assessment tools.
Lee J, Mulder F, Leeflang M, Wolff R, Whiting P, Bossuyt PM. QUAPAS: An Adaptation of the QUADAS-2 Tool to Assess Prognostic Accuracy Studies. Ann Intern Med. 2022 Jul;175(7):1010-1018. doi: 10.7326/M22-0276. Epub 2022 Jun 14. PMID: 35696685.
Hayden JA, van der Windt DA, Cartwright JL, Côté P, Bombardier C. Assessing bias in studies of prognostic factors. Ann Intern Med. 2013 Feb 19;158(4):280-6. doi: 10.7326/0003-4819-158-4-201302190-00009. PMID: 23420236.
Produced by: PROBAST Group
About: Risk of Bias assessment for studies developing, validating, or updating (for example, extending) prediction models, both diagnostic and prognostic
Produced by: SYstematic Review Centre for Laboratory animal Experimentation (SYRCLE)
About: SYRCLE’s risk of bias tool for animal studies (BMC Medical Research Methodology open access article)
Produced by: US National Institutes of Health National Toxicology Program, Office of Health Assessment and Translation (OHAT)
About: This risk-of-bias tool evaluates internal validity – the assessment of whether the design and conduct of the study compromised the credibility of the link between exposure and outcome (Higgins and Green 2011, IOM 2011, Viswanathan et al. 2012). There are other aspects of a study that will impact its utility for addressing the research question such as external validity – indirectness or applicability, which are addressed elsewhere in the OHAT Approach. In other words, risk of bias addresses the question “Are the results of the study credible?” Whereas indirectness addresses the question “Did the study design address the topic of the evaluation?
Note: I am unable to locate a web site with the full instrument. This article describes the development of the instrument and includes some of the questions.
Produced by: CLARITY Group at McMaster University in conjunction with Evidence Partners (DistillerSR)
About: About CLARITY
Note: This instrument does not asses domains as is common with other RoB assessment tools, but rather is a scale (more common among critical appraisal instruments) of 3 items.
Produced by: CLARITY Group at McMaster University in conjunction with Evidence Partners (DistillerSR)
About: About CLARITY
Note: This instrument does not asses domains as is common with RoB assessment tools, but rather is a scale (more common among critical appraisal instruments) of 5 items.