Metacognition
Do doctors know when they don't understand statistics?
Open access · cc by · source: Europe PMC
Medical students and doctors often got basic statistics wrong while feeling sure they were right, and their most common error on a diagnostic-test problem was made with as much confidence as correct answers.
Study at a glance
- Design
- Cross-sectional — Preregistered online quiz collecting answer plus confidence on 12 true/false claims (vaccine efficacy, p values) and a Bayesian positive-predictive-value problem, with participants randomly assigned to conditional-probability or natural-frequency wording for that problem
- N
- N=898 · 898 participants answered demographics and at least one exercise (522 students, 151 residents, 22 from abroad, 203 physicians); 681 completed the PPV exercise and 65 did the 6-week follow-up
- Population
- French-speaking medical students, residents and practising physicians recruited online as volunteers
- Outcome
- Accuracy and confidence on each claim, confidence-accuracy calibration and discrimination, and correctness of the PPV calculation by framing
Structured fields used in claim comparison tables when every cited study has a complete layer.
Key findings
Accuracy on individual claims varied widely, yet confidence was high overall, and people's average confidence passed the midpoint once they had just 3 of 12 correct, a pattern of overconfidence. Confidence still separated right from wrong answers, but this gap shrank from 32.5 points in low scorers to 14.2 points in higher scorers. On the test problem only 15% gave the correct answer of about 26%; 38.2% simply reported the test's sensitivity, and they did so with confidence similar to correct responders. Natural-frequency wording raised correct answers from 8.0% to 21% but did not reduce the sensitivity confusion.
Methodology
The researchers ran a preregistered online quiz with 898 medical students, residents and physicians. For each of 12 statements about vaccine efficacy and p values, participants marked true or false and how sure they were on one sliding scale. They then solved a problem asking how likely a person with a positive COVID-19 antigen test really has the disease, with the numbers randomly presented either as percentages (conditional probabilities) or as counts of people (natural frequencies), and gave a confidence range around their answer.
Limitations
Participants were self-selected volunteers from social media and mailing lists, and medical status was not verified, so the sample may not represent French clinicians. Only a few questions on three topics were asked, and the claims differed in difficulty, so the overconfidence pattern could partly reflect item selection rather than a general trait. The double-sided confidence slider is unusual in this field and its labels may have nudged confidence levels. The teaching interventions at the end could not be evaluated because only 65 people completed the follow-up, and they were already better than average.
How this study connects
Role on claims
Each row is a claim on a concept or method page where this paper supports, challenges, or qualifies the statement. Roles are hand-checked — not a model guess.
Not yet placed on a claim. This paper has study layers, but no concept page yet cites it as support, challenge, or qualifier.
Related papers in this topic
Same topic cluster — not a recommendation engine.