Samenvatting
Summary Chapter 5 - Introduction to impact evaluation and randomized controlled trials
De hele les uitgetypt, in detail!!
[Meer zien]Voorbeeld 3 van de 27 pagina's
Voorbeeld 3 van de 27 pagina's
In winkelwagen
Voorbeeld van de inhoud
5. Introduction to Impact Evaluation and Randomized ControlledTrials (D. Emmers)
1. Intro
Impact evaluation: the changes that can be attributed to a particular intervention, such as a project, program
or policy. We measure the impact of such an intervention on certain outcomes. It helps answer key questions
to inform evidence-based policy making. Which is the most effective and the most cost-effective?
Policy Question: ”How can we improve educational attainment of students in Kenya?”
- Smaller class size?
- Better monitoring of teachers?
- Provide better inputs (textbooks, school-meals, etc?)
- Subsidize ice-cream?
Ice-cream is significantly and positively related with test scores!
Correlation is not causation: We observe a significant cross-country correlation between educational
attainment (test scores) and ice-cream consumption. Does this mean that providing free ice-cream to students
in Kenya will improve test scores? No, probably not. More likely that countries with good schools and high ice-
cream consumption are simply more wealthy. Simply observing reality is not sufficient! We need to
understand what causes an improvement in test scores rather than what is correlated with high test scores.
This is what impact evaluation aims to do!
The impact of a program: it is the comparison between the outcome at the end of the program and the same
outcome if the program had NOT happened.
,Graphically:
- The outcome could be students outcomes. It is stable over time (no high peaks). At a certain point in
time, a certain intervention happens. The average outcome now increases over time. But based on
this, we can still not conclude that this increase is caused by the intervention. It could be that
something else happened around the time of the intervention that caused the outcome to increase.
- We need a counterfactual (= a situation in which the intervention would not have been happened) to
be able to detect a causal impact. The outcome remains stable here.
- Compare the outcome of the counterfactual with the situation where the intervention has happened.
The difference between these 2 is a measure of the impact!
The most obvious PROBLEM of impact evaluation is that we cannot observe the ”counterfactual”! Either a
school got textbooks or did not, either someone went to college or not, either someone had health insurance
or not, either a policy happened or not...
What is the solution? Can we use something that can be used as a counterfactual?
We need to ”mimic” the counterfactual (= Randomized Controlled Trials), 2 things have been tried:
1. FIND a group of individuals that are probably similar to the counterfactual = Quasi-Experimental
methods. Control for Observables, Difference-in-difference (DD), Regression Discontinuity (RD). But
these methods have a huge throwback where there are omitted variable biases etc. So it’s not possible
that a certain impact of an intervention is really causal.
2. CREATE a group of individuals that approximate (are verry similar to) the counterfactual = A
RANDOMIZED EXPERIMENT! “The most credible and influential research designs use random
assignment.” (Angrist and Pischke, 2009)
Test if a medicine is effective in reducing illness. Select a sample of people (patients) and randomly
allocate half of the sample in the treatment group and half in the control group. Afterwards, an
intervention is given to the treatment group. Then collect follow-up data in both groups. Now
compare (treatment: only 1/20% got the disease, control: 3/60% got the disease so the intervention
causally had an impact) both results and get the effect!
, Material we cover in 3 following lectures:
Part 1: How Randomization Solves the Selection Problem
Part 2: Impact Evaluation using RCTs in Practice
Part 3: Departures from Perfect Randomization and How to Deal with It.
Part 4: External Validity and Some Other Cautionary Tales
Part 5: RCTs in Action - Human Capital Policies in Developing Countries
2. Selection problem
1. The problem of causal inference
A ’Simple’ Policy Question: ”What is the causal effect of textbooks on learning?”
Can we compare average test scores of students in schools that have textbooks with average test scores of
students in schools that do not have textbooks? Why not? Selection bias!
Selection Bias Students in schools that have textbooks are different from students in schools that do not have
textbooks. They might have parents that invest more time in homework. Or they might have better teachers,
or smaller class size, etc. The decision to go to a school with textbooks is driven by self-selection. Any
difference between the two groups can be attributed to both the impact of textbooks and/or pre-existing
differences. It’s like comparing apples and oranges!
Can we compare scores of a students before and after textbooks are introduced in the same school? Why not?
Omitted variable bias!
Comparing test scores for the same students before and after a school introduces textbooks will not give a
reliable estimate of the impact of textbooks on learning because other factors (unobserved to the researcher)
that affect learning might change as well. The year the textbooks are introduced a new teacher is hired who is
better in teaching than the previous one. Or the year before text books are introduced a group of low-ability
students drops out and moves to other school.
A student either has a textbook or doesn’t have a textbook. We cannot observe the same student at the same
period in time in both scenario’s. We need a comparison group: a group of students who, in the absence of
treatment (textbooks) would have had outcomes similar to those who did receive the treatment.
Voordelen van het kopen van samenvattingen bij Stuvia op een rij:
√ Verzekerd van kwaliteit door reviews
Stuvia-klanten hebben meer dan 700.000 samenvattingen beoordeeld. Zo weet je zeker dat je de beste documenten koopt!
Snel en makkelijk kopen
Je betaalt supersnel en eenmalig met iDeal, Bancontact of creditcard voor de samenvatting. Zonder lidmaatschap.
Focus op de essentie
Samenvattingen worden geschreven voor en door anderen. Daarom zijn de samenvattingen altijd betrouwbaar en actueel. Zo kom je snel tot de kern!
Veelgestelde vragen
Wat krijg ik als ik dit document koop?
Je krijgt een PDF, die direct beschikbaar is na je aankoop. Het gekochte document is altijd, overal en oneindig toegankelijk via je profiel.
Tevredenheidsgarantie: hoe werkt dat?
Onze tevredenheidsgarantie zorgt ervoor dat je altijd een studiedocument vindt dat goed bij je past. Je vult een formulier in en onze klantenservice regelt de rest.
Van wie koop ik deze samenvatting?
Stuvia is een marktplaats, je koop dit document dus niet van ons, maar van verkoper maari. Stuvia faciliteert de betaling aan de verkoper.
Zit ik meteen vast aan een abonnement?
Nee, je koopt alleen deze samenvatting voor €5,99. Je zit daarna nergens aan vast.
Is Stuvia te vertrouwen?
4,6 sterren op Google & Trustpilot (+1000 reviews)
Afgelopen 30 dagen zijn er 66579 samenvattingen verkocht
Opgericht in 2010, al 14 jaar dé plek om samenvattingen te kopen