Literature
• Rossi, P. H., Lipsey, M. W., & Henry, G. T. (2004). Evaluation. A systematic approach (8th ed.).
Thousand Oaks, CA: Sage. Chapter 1 - 7, 11, and 12.
• Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental
designs for generalized casual inference. Boston, MA: Houghton Mifflin. Chapter 1 - 6, and 8.
Lecture 1
Rossi, Lipsey and Freeman
Chapters 1, 2 and 3
Evaluation research starts from social problems and the social programs that are addressing or trying to
solve those problems. If we were in a live lecture room I would ask you to think first what problems in
education you can think of that need to be solved and if you can think of some social programs that are
actually trying to tackle an educational program.
A social problem is an issue that affects the (living/learning/health/etc.) conditions of many people. It is a
problem because it affects so many. That’s why it’s called social problem. We can think of many; if you
just brainstorm you will probably think of many of them. Here’s a very random sort of sample of social
problems you can think of. For example:
1. Absenteeism rates of students in higher education.
2. Bullying. This news article is talking about online bullying in particular.
3. Poor reading skills of primary school children in this article in particular, but I think it is a more
generalized problem. It doesn’t allow them to get into even further content. It’s not only about
reading, but it enables reaching other sorts of content.
, 4. This comes from a Dutch newspaper and talks about the costs that are consequence of the low
literacy levels; the cost to society.
5. Another one from a Dutch source: the stress that students experience. In this case students on
school level.
6. And of course, more timely, stress that students are suffering now – because of the COVID-19
situation and because of the change in conditions in which students have to learn – is a very
generalized problem of this pandemic.
A social program is ‘’an organized, planned, and usually ongoing effort designed to ameliorate a social
problem or to improve social,, (living/learning/health/etc.) ,,conditions.’’ (Rossi, Lipsey & Freeman, 2004,
p. 29). So again, if you can think of problems, you can also think of programs that we might know. Here
are a few examples:
1. Again, this is a Dutch problem, but it is an implementation of a program that originated in the
US, which is the Succes for All program. It is actually developed here in the Netherlands by our
institute for educational research, which is a school reform actually that tries to improve
education with focus on reading for students with problematic backgrounds; especially for low
SES students.
, 2. Then, you also have curriculum changes that, for example, try to improve the situation for some
particular students. This is our relatively recent policy of inclusive education in the Netherlands,
in which students with special learning needs are included now in regular classrooms. That is also
a social program that aims to integrate and tackle the problem of lack of integration of students
with special needs.
3. Another very famous program, and it is one of the most studied social programs in the world, is
the Head Start program, which is also a reform implemented in the US that aims to provide
young children a good quality early childhood education.
4. There are international programs, for example this one laptop per child program that has been
implemented in third world economies; low income economies I should say. That basically
provides laptops for children to include it in their educational programs.
5. Another social program that’s very well known here in The Netherlands (and at least in Europe),
is the KiVa project/program. It is an anti-bullying program that originated in Finland.
So, this is a very generous set of examples of social programs in education that attempt to solve specific
problems that have been detected in education and affect many students/children.
So, the point of this course, why we discuss social problems and programs, is that those programs need
to be evaluated at a certain point. That is the perspective we take in this point. That’s important because
we’ve been teaching this course for years and we realized that for some students it’s difficult to take the
evaluator perspective/role into this program. Many times, they go into the role of program implementer.
The whole point of this course is that we want you to get tools in order to eventually, in the future, be
able to perform this role of evaluator of a social program. So, if you think ‘is a program doing well?’, for
example ‘is the anti-bullying program doing well?’, that would be an evaluation research in very general
terms. And now again, maybe you take another second to reflect ‘what aspects of the program could be
evaluated?’; could undergo an evaluation to check whether it is doing okay of not.
, Program evaluation
Program evaluation entails the use of social research methods, in
order to systematically investigate the effectiveness of social
intervention programs, in ways that are adapted to their political
and organizational environments, and are designed to inform social
action to improve social conditions that the program started from
basically.
Let’s try to defect this definition a little bit, because there are
several aspects that are important here.
• First is the use of social research methods. In program evaluation you
need to collect, analyze evidence, and you need to interpret that
evidence. That has to be done in a way that is credible, that is valid and
reliable, and that is defensible (that you can actually backup your claims
about whether a program is working or not. For that you need the best
possible methods in order to make those claims. But also of course, that
has to be feasible and adapted to circumstances.
• Because, unlike academic research, a program evaluation has a
different aim, which is actually to finally contribute to this social good,
to be able to help in the solving of the general social problems that
have been detected. Because of that, there are several domains of
effectiveness and we’re going to discuss that in more detail in a
minute.
• Another aspect is that these evaluations are embedded in a political
and organizational environment. They are not academic in the sense that
they are not done in an academic setting and for academic purposes of
knowledge dissemination and enlarging the body of knowledge on
something. That is not the ultimate goal; it is to improve a social condition
and that social condition/the program is also embedded in a political and
organizational situation/context. So, it is key to know who is requesting
these evaluations. Who are the sponsors? Are the sponsors the ones who
put down money for this evaluation? So, it is important to know the nature
of the question. And that nature of the question is always embedded in these political and
organizational environments.
• And as I said, this information should go back to those who are able to
make decisions, to those who are finally responsible and have the power
to e.g. continue with the program or stop it. So, there are decisions
associated with program evaluations. Sometimes go/no go decisions,
sometimes scaling up decisions (e.g. if a program is working well in
Groningen, we have to scale it up to the whole Netherlands), or
increasing budget, or cutting budget. So, decisions are associated very
concrete decisions that can be associated with the program evaluation.
Therefore, the information you are providing as an evaluator should be useful and should be
used, sometimes in the most direct way, but also indirectly, but it is information that ultimately
