Resume
Summary Chapter 5 - Introduction to impact evaluation and randomized controlled trials
- Établissement
- Katholieke Universiteit Leuven (KU Leuven)
De hele les uitgetypt, in detail!!
[Montrer plus]
Aperçu du contenu
5. Introduction to Impact Evaluation and Randomized ControlledTrials (D. Emmers)
1. Intro
Impact evaluation: the changes that can be attributed to a particular intervention, such as a project, program
or policy. We measure the impact of such an intervention on certain outcomes. It helps answer key questions
to inform evidence-based policy making. Which is the most effective and the most cost-effective?
Policy Question: ”How can we improve educational attainment of students in Kenya?”
- Smaller class size?
- Better monitoring of teachers?
- Provide better inputs (textbooks, school-meals, etc?)
- Subsidize ice-cream?
Ice-cream is significantly and positively related with test scores!
Correlation is not causation: We observe a significant cross-country correlation between educational
attainment (test scores) and ice-cream consumption. Does this mean that providing free ice-cream to students
in Kenya will improve test scores? No, probably not. More likely that countries with good schools and high ice-
cream consumption are simply more wealthy. Simply observing reality is not sufficient! We need to
understand what causes an improvement in test scores rather than what is correlated with high test scores.
This is what impact evaluation aims to do!
The impact of a program: it is the comparison between the outcome at the end of the program and the same
outcome if the program had NOT happened.
,Graphically:
- The outcome could be students outcomes. It is stable over time (no high peaks). At a certain point in
time, a certain intervention happens. The average outcome now increases over time. But based on
this, we can still not conclude that this increase is caused by the intervention. It could be that
something else happened around the time of the intervention that caused the outcome to increase.
- We need a counterfactual (= a situation in which the intervention would not have been happened) to
be able to detect a causal impact. The outcome remains stable here.
- Compare the outcome of the counterfactual with the situation where the intervention has happened.
The difference between these 2 is a measure of the impact!
The most obvious PROBLEM of impact evaluation is that we cannot observe the ”counterfactual”! Either a
school got textbooks or did not, either someone went to college or not, either someone had health insurance
or not, either a policy happened or not...
What is the solution? Can we use something that can be used as a counterfactual?
We need to ”mimic” the counterfactual (= Randomized Controlled Trials), 2 things have been tried:
1. FIND a group of individuals that are probably similar to the counterfactual = Quasi-Experimental
methods. Control for Observables, Difference-in-difference (DD), Regression Discontinuity (RD). But
these methods have a huge throwback where there are omitted variable biases etc. So it’s not possible
that a certain impact of an intervention is really causal.
2. CREATE a group of individuals that approximate (are verry similar to) the counterfactual = A
RANDOMIZED EXPERIMENT! “The most credible and influential research designs use random
assignment.” (Angrist and Pischke, 2009)
Test if a medicine is effective in reducing illness. Select a sample of people (patients) and randomly
allocate half of the sample in the treatment group and half in the control group. Afterwards, an
intervention is given to the treatment group. Then collect follow-up data in both groups. Now
compare (treatment: only 1/20% got the disease, control: 3/60% got the disease so the intervention
causally had an impact) both results and get the effect!
, Material we cover in 3 following lectures:
Part 1: How Randomization Solves the Selection Problem
Part 2: Impact Evaluation using RCTs in Practice
Part 3: Departures from Perfect Randomization and How to Deal with It.
Part 4: External Validity and Some Other Cautionary Tales
Part 5: RCTs in Action - Human Capital Policies in Developing Countries
2. Selection problem
1. The problem of causal inference
A ’Simple’ Policy Question: ”What is the causal effect of textbooks on learning?”
Can we compare average test scores of students in schools that have textbooks with average test scores of
students in schools that do not have textbooks? Why not? Selection bias!
Selection Bias Students in schools that have textbooks are different from students in schools that do not have
textbooks. They might have parents that invest more time in homework. Or they might have better teachers,
or smaller class size, etc. The decision to go to a school with textbooks is driven by self-selection. Any
difference between the two groups can be attributed to both the impact of textbooks and/or pre-existing
differences. It’s like comparing apples and oranges!
Can we compare scores of a students before and after textbooks are introduced in the same school? Why not?
Omitted variable bias!
Comparing test scores for the same students before and after a school introduces textbooks will not give a
reliable estimate of the impact of textbooks on learning because other factors (unobserved to the researcher)
that affect learning might change as well. The year the textbooks are introduced a new teacher is hired who is
better in teaching than the previous one. Or the year before text books are introduced a group of low-ability
students drops out and moves to other school.
A student either has a textbook or doesn’t have a textbook. We cannot observe the same student at the same
period in time in both scenario’s. We need a comparison group: a group of students who, in the absence of
treatment (textbooks) would have had outcomes similar to those who did receive the treatment.
Les avantages d'acheter des résumés chez Stuvia:
Qualité garantie par les avis des clients
Les clients de Stuvia ont évalués plus de 700 000 résumés. C'est comme ça que vous savez que vous achetez les meilleurs documents.
L’achat facile et rapide
Vous pouvez payer rapidement avec iDeal, carte de crédit ou Stuvia-crédit pour les résumés. Il n'y a pas d'adhésion nécessaire.
Focus sur l’essentiel
Vos camarades écrivent eux-mêmes les notes d’étude, c’est pourquoi les documents sont toujours fiables et à jour. Cela garantit que vous arrivez rapidement au coeur du matériel.
Foire aux questions
Qu'est-ce que j'obtiens en achetant ce document ?
Vous obtenez un PDF, disponible immédiatement après votre achat. Le document acheté est accessible à tout moment, n'importe où et indéfiniment via votre profil.
Garantie de remboursement : comment ça marche ?
Notre garantie de satisfaction garantit que vous trouverez toujours un document d'étude qui vous convient. Vous remplissez un formulaire et notre équipe du service client s'occupe du reste.
Auprès de qui est-ce que j'achète ce résumé ?
Stuvia est une place de marché. Alors, vous n'achetez donc pas ce document chez nous, mais auprès du vendeur maari. Stuvia facilite les paiements au vendeur.
Est-ce que j'aurai un abonnement?
Non, vous n'achetez ce résumé que pour €5,99. Vous n'êtes lié à rien après votre achat.
Peut-on faire confiance à Stuvia ?
4.6 étoiles sur Google & Trustpilot (+1000 avis)
66579 résumés ont été vendus ces 30 derniers jours
Fondée en 2010, la référence pour acheter des résumés depuis déjà 14 ans
