Skip to content
Open DegreeOpen Degree

Design a way to collect data that answers a question

I can turn a question into a plan for collecting data — naming who or what is measured, how they are chosen, what is recorded, and what comparison the plan makes possible — and say which conclusions the plan can support and which it cannot.

Domain
Data Literacy
Level
advanced
Status
draft · v0.1.0
Also called
Study design, Survey design, Sampling, Collect your own data, Research design
License
CC BY-SA 4.0

Learn first

Learn it

Scope

Producing data rather than reading it: a question, and a plan that could answer it. A class survey, a small experiment, a week of observations, a scrape of something already published.

In scope: choosing between a survey, an experiment and an observation; defining the population and how units are chosen from it; deciding what to record and in what units; building the comparison into the design rather than hoping for it afterwards; and stating the plan’s limits before any data exists. Out of scope: the analysis once the data arrives, which is Summarize a dataset and communicate findings, and formal statistical inference.

The four decisions a plan must make

  1. Who or what is measured. The population the question is about, which is rarely the group that is easy to reach.
  2. How they are chosen. Randomly, exhaustively, or by convenience — and if by convenience, said out loud, because it decides what the result can be generalized to.
  3. What is recorded. The exact question wording, the units, the instrument, the moment. A question that can be read two ways produces two datasets in one column.
  4. What is compared. Before and after, treated and untreated, one group and another. A design with no comparison can describe, but it cannot explain.

Why the design decides the conclusion

A causal claim needs a comparison whose only systematic difference is the thing being tested. That is what randomizing does: it makes the groups alike in every respect nobody thought to measure. An observation of people who chose the treatment for themselves cannot do this, however large it is, because the choosing is part of the difference.

This is the same ground as Judge a claim made from data, from the other side. Having taken apart other people’s claims, a learner writing their own plan can see exactly which criticisms it would attract.

Common misconceptions

  • “More responses make a better sample.” Coverage beats size. A thousand people chosen at random beat a hundred thousand who volunteered.
  • “Neutral wording is obvious.” It is not, and pilots are how it is found. Ask five people and at least one will read a question differently than intended.
  • “We can decide the comparison later.” The comparison is designed in or absent; analysis cannot supply one that the collection did not make.
  • “Observation is just a weaker experiment.” It answers a different question. When an experiment would be unethical or impossible, an observation with its limits stated is the honest instrument.

Notes for teachers

Give a question the learners actually care about and keep the scale small: one corridor, one week, twenty responses. Require the plan in writing before any data is collected, and require the limits paragraph in the same breath, so stating what a design cannot show is a normal part of designing rather than an admission at the end.

A useful pairing is to have one learner design and another critique with Judge a claim made from data, then swap.

Resources

ResourceKindCostProvider
Khan Academy, Sampling methodsvideofreeKhan Academy
Pew Research Center: Writing survey questionsreadingfreePew Research Center
Pew Research Center: MethodsreferencefreePew Research Center
OpenIntro Statistics (open textbook)bookfreeOpenIntro

Courses that include it

No course includes it yet. A course is optional; the outcome stands on its own.

Prove it

Evidence of mastery

  • Writes a one-page collection plan for a stated question, naming the population, how the sample is chosen, what is recorded for each unit, and when.
  • Says whether the plan is a survey, an experiment, or an observation, and why that choice fits the question.
  • Names the comparison the plan makes possible, and states plainly whether it can support a causal conclusion.
  • Identifies who the plan will miss and what that does to the conclusion.
  • Pilots the instrument on a handful of people and reports what changed as a result.

Assessments that measure it

No assessment yet.

Credentials that include it

No credential yet.

What learners show

Nothing to show yet. The publisher last published on 2026-09-20, and no aggregate for this outcome has reached the 50 learners a number needs. Open Degree does not track learners. About learner evidence.

What comes next

No outcome builds on this yet. Ask for the next concept.

Alignments