Summary
Summary Chapter 5 - Introduction to impact evaluation and randomized controlled trials
- Course
- Institution
The whole lesson typed out, in detail!!
[Show more]
Seller
Follow
maari
Reviews received
Content preview
5. Introduction to Impact Evaluation and Randomized ControlledTrials (D. Emmers)
1. Intro
Impact evaluation: the changes that can be attributed to a particular intervention, such as a project, program
or policy. We measure the impact of such an intervention on certain outcomes. It helps answer key questions
to inform evidence-based policy making. Which is the most effective and the most cost-effective?
Policy Question: ”How can we improve educational attainment of students in Kenya?”
- Smaller class size?
- Better monitoring of teachers?
- Provide better inputs (textbooks, school-meals, etc?)
- Subsidize ice-cream?
Ice-cream is significantly and positively related with test scores!
Correlation is not causation: We observe a significant cross-country correlation between educational
attainment (test scores) and ice-cream consumption. Does this mean that providing free ice-cream to students
in Kenya will improve test scores? No, probably not. More likely that countries with good schools and high ice-
cream consumption are simply more wealthy. Simply observing reality is not sufficient! We need to
understand what causes an improvement in test scores rather than what is correlated with high test scores.
This is what impact evaluation aims to do!
The impact of a program: it is the comparison between the outcome at the end of the program and the same
outcome if the program had NOT happened.
,Graphically:
- The outcome could be students outcomes. It is stable over time (no high peaks). At a certain point in
time, a certain intervention happens. The average outcome now increases over time. But based on
this, we can still not conclude that this increase is caused by the intervention. It could be that
something else happened around the time of the intervention that caused the outcome to increase.
- We need a counterfactual (= a situation in which the intervention would not have been happened) to
be able to detect a causal impact. The outcome remains stable here.
- Compare the outcome of the counterfactual with the situation where the intervention has happened.
The difference between these 2 is a measure of the impact!
The most obvious PROBLEM of impact evaluation is that we cannot observe the ”counterfactual”! Either a
school got textbooks or did not, either someone went to college or not, either someone had health insurance
or not, either a policy happened or not...
What is the solution? Can we use something that can be used as a counterfactual?
We need to ”mimic” the counterfactual (= Randomized Controlled Trials), 2 things have been tried:
1. FIND a group of individuals that are probably similar to the counterfactual = Quasi-Experimental
methods. Control for Observables, Difference-in-difference (DD), Regression Discontinuity (RD). But
these methods have a huge throwback where there are omitted variable biases etc. So it’s not possible
that a certain impact of an intervention is really causal.
2. CREATE a group of individuals that approximate (are verry similar to) the counterfactual = A
RANDOMIZED EXPERIMENT! “The most credible and influential research designs use random
assignment.” (Angrist and Pischke, 2009)
Test if a medicine is effective in reducing illness. Select a sample of people (patients) and randomly
allocate half of the sample in the treatment group and half in the control group. Afterwards, an
intervention is given to the treatment group. Then collect follow-up data in both groups. Now
compare (treatment: only 1/20% got the disease, control: 3/60% got the disease so the intervention
causally had an impact) both results and get the effect!
, Material we cover in 3 following lectures:
Part 1: How Randomization Solves the Selection Problem
Part 2: Impact Evaluation using RCTs in Practice
Part 3: Departures from Perfect Randomization and How to Deal with It.
Part 4: External Validity and Some Other Cautionary Tales
Part 5: RCTs in Action - Human Capital Policies in Developing Countries
2. Selection problem
1. The problem of causal inference
A ’Simple’ Policy Question: ”What is the causal effect of textbooks on learning?”
Can we compare average test scores of students in schools that have textbooks with average test scores of
students in schools that do not have textbooks? Why not? Selection bias!
Selection Bias Students in schools that have textbooks are different from students in schools that do not have
textbooks. They might have parents that invest more time in homework. Or they might have better teachers,
or smaller class size, etc. The decision to go to a school with textbooks is driven by self-selection. Any
difference between the two groups can be attributed to both the impact of textbooks and/or pre-existing
differences. It’s like comparing apples and oranges!
Can we compare scores of a students before and after textbooks are introduced in the same school? Why not?
Omitted variable bias!
Comparing test scores for the same students before and after a school introduces textbooks will not give a
reliable estimate of the impact of textbooks on learning because other factors (unobserved to the researcher)
that affect learning might change as well. The year the textbooks are introduced a new teacher is hired who is
better in teaching than the previous one. Or the year before text books are introduced a group of low-ability
students drops out and moves to other school.
A student either has a textbook or doesn’t have a textbook. We cannot observe the same student at the same
period in time in both scenario’s. We need a comparison group: a group of students who, in the absence of
treatment (textbooks) would have had outcomes similar to those who did receive the treatment.
The benefits of buying summaries with Stuvia:
Guaranteed quality through customer reviews
Stuvia customers have reviewed more than 700,000 summaries. This how you know that you are buying the best documents.
Quick and easy check-out
You can quickly pay through credit card or Stuvia-credit for the summaries. There is no membership needed.
Focus on what matters
Your fellow students write the study notes themselves, which is why the documents are always reliable and up-to-date. This ensures you quickly get to the core!
Frequently asked questions
What do I get when I buy this document?
You get a PDF, available immediately after your purchase. The purchased document is accessible anytime, anywhere and indefinitely through your profile.
Satisfaction guarantee: how does it work?
Our satisfaction guarantee ensures that you always find a study document that suits you well. You fill out a form, and our customer service team takes care of the rest.
Who am I buying these notes from?
Stuvia is a marketplace, so you are not buying this document from us, but from seller maari. Stuvia facilitates payment to the seller.
Will I be stuck with a subscription?
No, you only buy these notes for $6.47. You're not tied to anything after your purchase.
Can Stuvia be trusted?
4.6 stars on Google & Trustpilot (+1000 reviews)
66579 documents were sold in the last 30 days
Founded in 2010, the go-to place to buy study notes for 14 years now