Skip to content
Open DegreeOpen Degree

Judge a claim made from data

I can examine a claim that cites data and say whether the data supports it, naming the comparison being made, how the data was collected, and at least one alternative explanation that has not been ruled out.

Domain
Data Literacy
Level
intermediate
Status
draft · v0.2.0
Also called
Correlation and causation, Evaluate a data claim, Data skepticism, Critical data literacy
License
CC BY-SA 4.0

Learn first

Learn it

Scope

A claim someone else has already made, with data behind it: a headline, a chart in a report, a figure on a dashboard, a number in a sales deck. The work is to read what the data can carry and say where the claim outruns it.

In scope: identifying the comparison (compared with what? measured when? among whom?), asking how the data was collected and who is missing from it, and naming alternative explanations. Designing a study, running a hypothesis test, or fitting a model are all out of scope here; this outcome is about reading a claim critically, not producing one.

What a good answer names

  1. The claim. Stated in one sentence, in the strongest form its author intends.
  2. The comparison. Almost every misleading statistic hides in a missing comparison: a rise since when, higher than whom, a rate or a raw count.
  3. The data behind it. Who was counted, who could not be, and when. A survey of people who answered is a survey of people who answer.
  4. What else would produce this pattern. A confounder, a selection effect, a change in definition or instrument, or simple chance.
  5. The gap. The difference between what the data supports and what is being claimed for it.

Common misconceptions

  • “There is a correlation, so one causes the other.” A correlation is consistent with causation in either direction, a common cause, a selection effect, or coincidence. The task is to say which of those has been ruled out, and how.
  • “A big sample makes it true.” Size fixes noise, not bias. A million responses from people who opted in can be further from the truth than a thousand chosen at random.
  • “The number is precise, so it is accurate.” Reported precision says nothing about how the quantity was defined or measured.
  • “No causal claim, no problem.” A statement that avoids saying “causes” while implying it should be read as the causal claim it is meant to be.

Notes for teachers

Work from live material rather than tidy textbook cases: a chart from this week’s news, a claim in an advertisement, a figure a learner brings in. Ask for the assessment in writing and in five sentences, so the naming is explicit rather than a general feeling of doubt.

The Calling Bullshit case studies are close to this outcome in form: a real published claim, and a worked read of what the data shows. Spurious Correlations is useful in the opposite direction — strong correlations that nobody would defend causally — because it separates the statistic from the story quickly.

Resources

ResourceKindCostProvider
Calling Bullshit: case studiesreadingfreeUniversity of Washington
Calling Bullshit: course videosvideofreeUniversity of Washington
Spurious CorrelationspracticefreeTyler Vigen
Khan Academy, Study designvideofreeKhan Academy
OpenIntro Statistics (open textbook)bookfreeOpenIntro

Courses that include it

No course includes it yet. A course is optional; the outcome stands on its own.

Prove it

Evidence of mastery

  • Given a news story, report, or dashboard that cites data, writes a short assessment naming the claim, the comparison it rests on, and how the data was collected.
  • Names at least one alternative explanation — a confounder, a selection effect, or a change in what was measured — and says what evidence would rule it out.
  • States plainly when a correlation does not establish causation, and what would have to be true, or what study would have to be run, to support the causal claim.
  • Separates the claim the data does support from the stronger claim being made for it.

Assessments that measure it

No assessment yet.

Credentials that include it

No credential yet.

What learners show

Nothing to show yet. The publisher last published on 2026-09-20, and no aggregate for this outcome has reached the 50 learners a number needs. Open Degree does not track learners. About learner evidence.

What comes next

Alignments