Reading a Report and the Sampling Lens

Lesson 1 of 2: Taking a Report Apart

In this lesson:

• Separate a report into claim, evidence, and source
• Use the sampling lens to judge whether the data supports the claim

Learning Objectives for This Unit

By the end of this two-lesson capstone, you should be able to:

1. Identify a report's source, study type, and central claim
2. Evaluate whether the sampling supports a population claim
3. Evaluate whether the design supports a causal claim
4. Evaluate whether uncertainty is disclosed and the claim overstated
5. Spot misleading displays and write a justified overall verdict

The Coffee Headline — Should You Believe It?

"New Study: Students Who Drink Coffee Score 15% Higher on Exams."

• It sounds solid: a study, a number, a clear takeaway
• Should you believe it — and how would you decide?

That feeling of credibility is what we're about to interrogate.

Restate It in Three Parts

Pull every report into these three before you judge it.

Claim vs Evidence vs Source

• Claim: the conclusion someone drew — coffee raises scores
• Evidence: what was actually measured — the 15%, the sample, the study
• Source: who produced and funded it — university, brand, blog?

The claim is not the evidence. They are almost never the same size.

The Claim Says More Than the Evidence Shows

The claim is confident, sweeping, causal. The evidence is some number from some group.

• The distance between them is where evaluation lives
• A number being present does not close that gap

Polish lives in the claim; truth lives in the evidence.

The Four-Lens Checklist for Reports

Each lens is a question this unit already taught you to ask.

Separate a Short Report Into Three Parts

A wellness blog reports people who do yoga sleep 40 minutes longer, from a survey of its newsletter subscribers.

• Write the claim, the evidence, and the source
• One sentence: where does the claim overreach the evidence?

Write it alone before advancing.

Lens 1: Was the Data Even Collected Well?

We separated claim from evidence — now we interrogate the evidence.

• First and most disqualifying question: was the sample any good?
• A biased or tiny sample sinks any conclusion drawn from it

This is the sampling lens — straight from A.1 and B.4.

The Four Questions of the Sampling Lens

Ask all four out loud before you trust any population claim.

Who Was Sampled, and From Where?

"Students" — but which students?

• All students? One dorm? Coffee-shop sign-up volunteers?
• These are different populations — the answer decides what 15% means

A self-selected group of coffee fans can't speak for all students.

How Were They Chosen — Random or Convenience?

• Random: everyone has a fair chance — the sample mirrors the whole
• Convenience / voluntary: whoever was easy or willing — bias baked in

Coffee-shop volunteers opted in — that's voluntary response, biased from the start.

How Many — 30 or 3,000?

Sample size sets how much random noise rides on the result.

• 30 self-selected students → the 15% could be chance wobble
• A bare "15% higher" tells you nothing about the size

Tiny sample → shaky number. The headline hides which.

A Big Biased Sample Is Worse (A.1, B.4)

• A large sample does not fix a biased one
• A.1: a big biased sample beats a small random one — false
• B.4: margin of error shrinks with size but never fixes bias

Bias is a direction error; size only sharpens the wrong target.

Undisclosed Sampling Is Itself a Red Flag

If the report hides its sample size and method, that silence counts against it.

• A credible report tells you how its data was gathered
• Missing who/how/how-many → you can't even run the lens

The presence of a number is not the presence of evidence.

30 Volunteers — Can 15% Generalize?

The study was 30 self-selected coffee drinkers on an online survey.

• Does "15% higher" support a claim about students in general?
• Decide yes or no — and name the sampling question that settles it

Stay on the sampling lens. Does it generalize?

Apply the Checklist; Flag the Missing Info

A fitness brand says 78% of users saw results in 30 days.

• Run the checklist: who / how / how many / disclosed?
• Flag every sampling fact the report fails to give you

The missing information is the finding. Write it down.

Key Takeaways From Lesson One

✓ Split every report: claim, evidence, source
✓ Sampling lens: who / how / how many / disclosed?
✓ A big biased sample beats a small random one — false

Numbers are the start of scrutiny, not the end
A percentage with no or uncertainty is a headline

Coming Up Next: The Design Lens

A report can pass the sampling lens and still overreach.

In Lesson 2, you'll match a claim's verb to its study design — and name causal overreach, the most common failure in data reporting.