Every EdTech company claims its product improves learning. Few can show it to a standard that would persuade a sceptical researcher. This section explains how education evidence is produced and judged: the US ESSA evidence tiers that districts use in procurement, the difference between randomised controlled trials and quasi-experimental and correlational designs, how to read and interpret effect sizes, why sample size and clustering decide what a study can detect, the What Works Clearinghouse and other evidence clearinghouses, third-party certifications such as Digital Promise's, and how to read vendor efficacy claims critically. The worked example computes effect sizes, confidence intervals, attrition and statistical power in Python, and simulates how "users vs non-users" comparisons manufacture effects that are not there.
What the study covers
- 1Overview: why evidence is hard in education
- 2Mechanics
- 3Key numbers
- 4Institutions and certifications
- 5Landmark cases
- 6Reading vendor efficacy claims
- 7Pitfalls
- 8How an expert thinks about it
- 9Glossary
Technical exhibits
Worked examples, checklists and case studies included with this study.
Effect Size and Power Worked Example
A stylised vendor study: Hedges' g, confidence interval, improvement index, attrition and baseline checks, sample sizes for individual and cluster RCTs, and a selection-bias simulation (all computed in Python)
Essa Tiers and Study Design Primer
The four ESSA tiers in detail, what each requires, which designs qualify, and a study-planning path for a startup
Vendor Efficacy Claims Checklist
Twenty-five questions to ask of any efficacy claim, with common red-flag phrasings decoded