Skip to Main Content

Health Research Reporting Guidelines, Study Execution Manuals, Critical Appraisal, Risk of Bias, & Non-reporting Biases

Why Critical Appraisal or Study Execution Assessment?

What is critical appraisal?

Definitions for critical appraisal evolve around a similar set of criteria:

What is Critical Appraisal (Bandolier)

  • "Critical appraisal is the process of carefully and systematically examining research to judge its trustworthiness, and its value and relevance in a particular context."

Glossary definition (Critical Appraisal Skills Program (CASP)

  • "Critical Appraisal is the process of assessing and interpreting evidence, by systematically considering its validity, results and relevance to your own context."

Critical Appraisal Tools (Centre for Evidence-Based Medicine (CEBM))

  • Critical appraisal is the systematic evaluation of clinical research papers in order to establish:
    • Does this study address a clearly focused question?
    • Did the study use valid methods to address this question?
    • Are the valid results of this study important?
    • Are these valid, important results applicable to my patient or population?

While critical appraisal can highlight bias in a study, the current version of the Cochrane Handbook points out:

"Methodological quality refers to critical appraisal of a study or systematic review and the extent to which study authors conducted and reported their research to the highest possible standard. Bias refers to systematic deviation of results or inferences from the truth. These deviations can occur as a result of flaws in design, conduct, analysis, and/or reporting. It is not always possible to know whether an estimate is biased even if there is a flaw in the study; further, it is difficult to quantify and at times to predict the direction of bias. For these reasons, reviewers refer to ‘risk of bias’ (Chapter 8)." Chapter V: Overviews of Reviews (below the table).

A separate page has been created for Risk of Bias.

Why critical appraisal?

Critical appraisal and, more specifically, critical appraisal tools provide us with a mechanism to evaluate the research methodology of a study with a critical, objective, and systematic lens. This appraisal is essential when evaluating a study for a systematic review, for determining new guidelines for patient care, or for choosing appropriate interventions. 

Uses for Critical Appraisal Tools

CA tools can be used in multiple ways and in different settings.

  1. Are you working on protocol development for a project? 
    • Use the appropriate critical appraisal tool (and reporting guideline) to ensure you engage in good study conduct and are prepared to practice clear and transparent reporting.
  2. Are you preparing for journal club?
    • Use the appropriate critical appraisal tool (and reporting guideline) to ensure the author(s) engaged in good study conduct as well as clear and transparent reporting.
  3. Are you peer reviewing manuscripts for a journal?
    • Use the appropriate critical appraisal tool (and reporting guideline) as a framework for evaluating manuscripts and providing feedback in a clear and objective manner.
  4. Are you conducting a systematic review?
    • Critical appraisal of included studies is a necessity when conducting a systematic review, even when the studies are non-randomized or observational. 

Systematic Review Critical Appraisal Tools

Checklist for Systematic Reviews and Research Syntheses

Produced by: Joanna Briggs Institute
Part of: The JBI Critical Appraisal Tools collection

  • The checklist is a Word document.

AMSTAR 2: A MeaSurement Tool to Assess systematic Reviews 

There were 2 goals in the development of AMSTAR:

  1. To create valid, reliable and useable instruments that would help users differentiate between systematic reviews, focusing on their methodological quality and expert consensus.
  2. To facilitate the development of high-quality reviews.

Critical Appraisal Tools

Produced by: Centre for Evidence Based Medicine, University of Oxford, UK
Part of: The CEBM Critical Appraisal Tools collection

CASP Checklists

Produced by:  Critical Appraisal Skills Programme, UK

CASP Systematic Review Checklist

Scoping & Other Review Types Critical Appraisal

CAT HPPR

About: Heise, T.L., Seidler, A., Girbig, M. et al. CAT HPPR: a critical appraisal tool to assess the quality of systematic, rapid, and scoping reviews investigating interventions in health promotion and prevention. BMC Med Res Methodol 22, 334 (2022). https://doi.org/10.1186/s12874-022-01821-4

Supplementary information:

  • 2 zipped files
    • File 1 Includes the checklist and user manual
    • File 2 includes methodological supporting documentation

Randomized Clinical Trial Critical Appraisal Tools

Randomized Controlled Trials

Produced by: Joanna Briggs Institute
Part of: The JBI Critical Appraisal Tools collection

  • The checklist is a Word document.

Study Quality Assessment Tools

Produced by: US National Heart, Lung, and Blood Institute (NHLBI)

This site has 6 assessment tools covering controlled interventions, SRs/MAs, observational cohort studies, cross-sectional studies, case control studies, pre-post studies (no control group), and case series.

CASP Checklists

Produced by:  Critical Appraisal Skills Programme, UK

CASP has 8 critical appraisal tools for SRs, RCTs, cohort studies, case control studies, economic evaluations, diagnostic studies, qualitative studies, and clinical prediction. Each item in the individual checklists provides a series of questions.

Randomised Controlled Trials (RCT) Critical Appraisal Worksheet

Produced by: Centre for Evidence Based Medicine, University of Oxford, UK
Part of: The CEBM Critical Appraisal Tools collection

  • The appraisal worksheets are in PDF format.
  • The appraisal worksheets are in English as well as Chinese, German, Lithuanian, Persian, and Spanish; languages other than English can be found on the home page.

Quasi-experimental (non-randomised) Trials/Studies Critical Appraisal Tools

Checklist for Quasi-experimental Studies (Non-Randomized Experimental Studies)

Produced by: Joanna Briggs Institute
Part of: The JBI Critical Appraisal Tools collection

  • The checklist is a Word document.

Study Quality Assessment Tools

Produced by: US National Heart, Lung, and Blood Institute (NHLBI)

This site has an assessment tool for pre-post studies (no control group).

Methodological index for non-randomized studies (MINORS): development and validation of a new instrument (requires U Pitt authentication) 

"Background: Because of specific methodological difficulties in conducting randomized trials, surgical research remains dependent predominantly on observational or non‐randomized studies. Few validated instruments are available to determine the methodological quality of such studies either from the reader's perspective or for the purpose of meta‐analysis. The aim of the present study was to develop and validate such an instrument."

Observational Studies Critical Appraisal Tools

Observational studies

Checklist for Analytical Cross Sectional Studies

Checklist for Case Control Studies

Checklist for Cohort Studies

Checklist for Prevalence Studies

Produced by: Joanna Briggs Institute
Part of: The JBI Critical Appraisal Tools collection

  • The checklists are all Word documents.

Study Quality Assessment Tools

Produced by: US National Heart, Lung, and Blood Institute (NHLBI)

  • This site has assessment tools for: observational cohort studies, cross-sectional studies, and case control studies.

CASP Checklists

Produced by: Critical Appraisal Skills Programme, UK

  • CASP has a critical appraisal tool for cohort studies and case control studies. Each item in the individual checklists provides a series of questions.

Newcastle-Ottawa Scale: Case Control Studies & Cohort Studies

Produced by: University of Newcastle, Australia and University of Ottawa, Canada

  • Manual: MS Word, .PDF
    • Instructions for both case control and cohort studies
  • Scale: MS Word, .PDF
    • Instruments for both case control and cohort studies

Diagnostic Studies Critical Appraisal Tools

Diagnostic Test Accuracy Studies

Produced by: Joanna Briggs Institute
Part of: The JBI Critical Appraisal Tools collection

  • The checklist is a Word document.

CASP Checklists

Produced by: Critical Appraisal Skills Programme, UK

CASP has a critical appraisal tool for diagnostic studies. Each item in the individual checklists provides a series of questions.

Diagnostics Critical Appraisal Sheet

Produced by: Centre for Evidence Based Medicine, University of Oxford, UK
Part of: The CEBM Critical Appraisal Tools collection

  • The appraisal worksheets are in PDF format.
  • The appraisal worksheets are in English as well as Chinese, German, Lithuanian, Persian, and Spanish; languages other than English can be found on the home page.

Prognosis/Prediction Critical Appraisal Tools

CHARMS: Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies

Moons KG, de Groot JA, Bouwmeester W, Vergouwe Y, Mallett S, Altman DG, Reitsma JB, Collins GS. Critical appraisal and data extraction for systematic reviews of prediction modelling studies: the CHARMS checklist. PLoS Med. 2014 Oct 14;11(10):e1001744. doi: 10.1371/journal.pmed.1001744. PMID: 25314315; PMCID: PMC4196729.

CHARMS-PF: Checklist for critical appraisal and data extraction for systematic reviews of prognostic factors studies

Found in: Riley RD, Moons KGM, Snell KIE, Ensor J, Hooft L, Altman DG, Hayden J, Collins GS, Debray TPA. A guide to systematic review and meta-analysis of prognostic factor studies. BMJ. 2019 Jan 30;364:k4597. doi: 10.1136/bmj.k4597. PMID: 30700442.

CASP Checklists

Produced by:  Critical Appraisal Skills Programme, UK

CASP has 8 critical appraisal tools including one for clinical prediction. Each item in the individual checklists provides a series of questions.

Prognosis Critical Appraisal Sheet

Produced by: Centre for Evidence Based Medicine, University of Oxford, UK
Part of: The CEBM Critical Appraisal Tools collection

  • The appraisal worksheets are in PDF format.
  • The appraisal worksheets are in English as well as Chinese, German, Lithuanian, Persian, and Spanish; languages other than English can be found on the home page.

Economic Evaluations Critical Appraisal Tools

Checklist for Economic Evaluations

Produced by: Joanna Briggs Institute
Part of: The JBI Critical Appraisal Tools collection

  • The checklist is a Word document.

CASP Checklists

Produced by:  Critical Appraisal Skills Programme, UK

CASP has a critical appraisal tool for economic evaluations. Each item in the individual checklists provides a series of questions.

Qualitative Studies Critical Appraisal Tools

Checklist for Qualitative Research

Produced by: Joanna Briggs Institute
Part of: The JBI Critical Appraisal Tools collection

  • The checklist is a Word document.

CASP Checklists

Produced by:  Critical Appraisal Skills Programme, UK

  • The checklist is in PDF format
  • CASP has a critical appraisal tool for qualitative studies. Each item in the individual checklists provides a series of questions.

Critical Appraisal of Qualitative Studies Sheet 

Produced by: Centre for Evidence Based Medicine, University of Oxford, UK
Part of: The CEBM Critical Appraisal Tools collection

  • The appraisal worksheets are in PDF format.
  • The appraisal worksheets are in English as well as Chinese, German, Lithuanian, Persian, and Spanish; languages other than English can be found on the home page.

Evaluation Tool for Mixed Methods Studies

Produced by: A. Long, U of Leeds

The ‘mixed method’ evaluation tool was developed from the evaluation tools for ‘quantitative’ and ‘qualitative’ studies,
themselves created within the context of a project exploring the feasibility of undertaking systematic reviews of research literature on effectiveness and outcomes in social care.

Case Reports & Case Series Critical Appraisal Tools

Study Quality Assessment Tools

Produced by: US National Heart, Lung, and Blood Institute (NHLBI)

This site has 6 assessment tools covering controlled interventions, SRs/MAs, observational cohort studies, cross-sectional studies, case control studies, pre-post studies (no control group), and case series.

Checklist for Case Reports & Checklist for Case Series

Produced by: Joanna Briggs Institute
Part of: The JBI Critical Appraisal Tools collection

  • The checklists are both Word documents.

Statistical Methodology Assessment

CHAMP (Checklist for statistical Assessment of Medical Papers) (2021)

About: "While CHAMP is primarily aimed at editors and peer reviewers during the statistical assessment of a medical paper, we believe it will serve as a useful reference to improve authors' and readers' practice in their use of statistics in medical research."

A tool to assess the quality of a meta-analysis (2013) (Not open access)

Higgins JP, Lane PW, Anagnostelis B, Anzures-Cabrera J, Baker NF, Cappelleri JC, Haughie S, Hollis S, Lewis SC, Moneuse P, Whitehead A. A tool to assess the quality of a meta-analysis. Res Synth Methods. 2013 Dec;4(4):351-66. doi: 10.1002/jrsm.1092. Epub 2013 Oct 18. PMID: 26053948. Not open access

All Critical Appraisal Tools in alphabetical order

CASP Checklists

Produced by:  Critical Appraisal Skills Programme, UK

CASP has 8 critical appraisal tools for SRs, RCTs, cohort studies, case control studies, economic evaluations, diagnostic studies, qualitative studies, and clinical prediction. Each item in the individual checklists provides a series of questions.

CAT HPPR

About: Heise, T.L., Seidler, A., Girbig, M. et al. CAT HPPR: a critical appraisal tool to assess the quality of systematic, rapid, and scoping reviews investigating interventions in health promotion and prevention. BMC Med Res Methodol 22, 334 (2022). https://doi.org/10.1186/s12874-022-01821-4

Supplementary information:

  • 2 zipped files
    • File 1 Includes the checklist and user manual
    • File 2 includes methodological supporting documentation

CHAMP (Checklist for statistical Assessment of Medical Papers) (2021)

"While CHAMP is primarily aimed at editors and peer reviewers during the statistical assessment of a medical paper, we believe it will serve as a useful reference to improve authors' and readers' practice in their use of statistics in medical research."

Critical Appraisal Tools

Produced by: Joanna Briggs Institute

The link above points to all of the Critical Appraisal Tools from JBI. All are in the format of a Word document.

See Chapter 3.2.7 of the JBI Manual for Evidence Synthesis for an explanation on how to conduct and describe the critical appraisal of studies in a systematic review. 

Critical Appraisal Tools

Produced by: Centre for Evidence Based Medicine, Oxford, UK

The appraisal worksheets are in English as well as Chinese, German, Lithuanian, Persian, and Spanish; languages can be found on the home page.

Downs & Black: The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions

Items from the Downs & Black checklist can be found in this article. 

Newcastle-Ottawa Scale

Produced by:  University of Newcastle, Australia and University of Ottawa, Canada

NOS was developed to assess the quality of nonrandomised studies with its design, content and ease of use directed to the task of incorporating the quality assessments in the interpretation of meta-analytic results. A 'star system' has been developed in which a study is judged on three broad perspectives: the selection of the study groups; the comparability of the groups; and the ascertainment of either the exposure or outcome of interest for case-control or cohort studies respectively.

Study Quality Assessment Tools

Produced by: US National Heart, Lung, and Blood Institute (NHLBI)

This site has 6 assessment tools covering controlled interventions, SRs/MAs, observational cohort studies, cross-sectional studies, case control studies, pre-post studies (no control group), and case series.

Additional Links