Inhoudsopgave
Week 1................................................................................................................................... 2
1. Hoorcollege.................................................................................................................... 2
2. Literatuur........................................................................................................................ 4
2.1 Patty - Policy evaluation in Europe..........................................................................4
2.2 Mergaert - Ex ante and ex post evaluation..............................................................8
2.3 Sloot, Jans & Steg - In it for the money, the environment, or the community?
Motives for being involved in community energy initiatives..........................................11
Week 2................................................................................................................................. 14
1. Hoorcollege.................................................................................................................. 14
2. Literatuur...................................................................................................................... 17
2.1 Pawson - What works in evaluation research?......................................................17
2.2 Rycroft-Malone - A realistic evaluation: the case of protocol-based care...............21
Week 3................................................................................................................................. 24
1. Hoorcollege.................................................................................................................. 24
2. Literatuur...................................................................................................................... 26
2.1 Leeuw - Reconstructing program theories: Methods available and problems to be
solved.......................................................................................................................... 26
2.2 Rogers - Program theory evaluation: practice, promise and problems..................29
2.3 Van Heffen, O - Beleidstheorieën in de beleidspraktijk..........................................32
Week 4................................................................................................................................. 35
1. Hoorcollege.................................................................................................................. 35
2. Literatuur...................................................................................................................... 38
2.1 Raad voor volksgezondheid en samenleving (2017) ‘’Zonder context geen bewijs’’
Over de illusie van evidence-based practice in de zorg...............................................38
Week 5................................................................................................................................. 40
1. Hoorcollege.................................................................................................................. 40
Week 6................................................................................................................................. 42
1. Hoorcollege.................................................................................................................. 42
2. Literatuur...................................................................................................................... 46
2.1 Rowe & Frewer - Public participation methods: A framework for evaluation..........46
2.2 Samenvatting WRR Rapport: ‘’Weten is nog geen doen. Een realistisch perspectief
op zelfredzaamheid.’’..................................................................................................51
2.3 Passages uit HIVOS/ISS/WRR Rapport: “Doen. Nieuwe vormen van democratie.”
.................................................................................................................................... 53
1
,Week 1
1. Hoorcollege
Wat is beleidsevaluatie?
Dit plaatje weergeeft de ideale beleidscyclus, deze
elementen zie je dus vaak terug als het beleid wordt
gemaakt. Echter zie je het bijna nooit in deze
chronologische volgorde.
Je kan beleidsevaluatie toevoegen aan de beleidscyclus op een summatieve of een
formatieve manier. Ook is er een combinatie van deze twee manieren.
● Formatief (ex ante): wat zou kunnen werken?
○ Tijdstip: Voor de implementatie, hier kijk je naar voorspellingen.
○ Hardheid uitspraken: Er is weinig direct empirische onderbouwing →
hieruit vloeien inschattingen voort.
○ Functie: Functie ter ondersteuning voor de beleidsmaker, aangezien dit te
maken heeft met de keuze voor de interventie.
● Summatief (ex post): wat hebben we gezien dat werkt? zijn de regels efficiënt
geweest?
○ Tijdstip: Na de implementatie, hier kijk je naar de effecten van het beleid.
○ Hardheid uitspraken: Je kijkt naar het verzamelde empirische bewijs, hier kan
je harde conclusies uit trekken.
○ Functie: Functie van verantwoording en verbetering van het beleid.
● Combinatie van de twee (ex tempore / ex durante): wat werkt er op dit moment?
○ Tijdstip: Tijdens de implementatie, hier kijk je nog naar de eerste effecten en
neveneffecten en voorspellingen.
○ Hardheid uitspraken: Je kan de eerste conclusies trekken
○ Functie: Functie ter bijsturing
Vanuit wetenschappelijk oogpunt wil je ze
natuurlijk alledrie uitvoeren, echter is dit
afhankelijk van draagvlak om het beleid te
evalueren en de resources waar je toegang tot
hebt.
2
,Trends en afhankelijkheden in evaluatieonderzoek (Europa
Er is een trend in Europa voor evidence-based policy, dus op basis van beleidsevaluaties
nieuw beleid maken. Het is ook een criteria / integraal geworden voor ‘’good management’’
van je ministerie, bedrijf etc.
Er zijn een aantal redenen voor deze groei:
● New public management en performance
○ Het moet efficiënter
○ Meer ‘’business like’’
● Tegelijkertijd zie je dat de EU meer verantwoording moet afleggen, dit kunnen zij
doen op basis van beleidsevaluatie.
○ Dit kwam in twee verschillende golven
1. Eerst was er een ontwikkeling van gemengde economieën. In de jaren
60/70/80 werd beleid meer een correctie van de open markt. Daarna
kwam er een toename in de vraag voor empirische en statistische
informatie, aangezien ze wilden zien wat werkt en niet werkte.
2. De tweede golf kwam voort uit de Europese Unie door het begrip
‘’accountability’’. Er was veel kritiek op de samenwerking binnen de
EU en dit was een van de reden waarom zij verantwoording af
moesten leggen. Zo moesten ze volgens de ideeën van ‘’new public
management’’ werken, ze moesten meer ‘’business like’’ werken. Ook
werd het EU budget herverdeeld, dit moest worden geëvalueerd om te
kijken of dit werkte.
Er kwam kritiek op het beleidsonderzoek:
● Je kan nooit fundamenteel bekijken of iets werkt of niet.
● Constructivisme; dit gaat erover dat mogelijk de inwoners van de EU meer moeten
worden betrokken bij het beleidsproces. Systematisch zijn ze binnen de EU te veel
gefocust op wat wel werkt en wat niet werkt. Om dit te onderzoeken zou je dus de
inwoners moeten betrekken.
● Ook moeten de belangen van actoren worden betrokken in het beleidsproces. Dus
het moet meer praktijk gericht. Het proces van actoren is namelijk bepalend voor het
beleid wat onderzocht is.
○ Voorbeelden beleidsactoren zijn: politici, beleidsmakers, media, de doelgroep
en lobbyisten.
Weerbarstige beleidspraktijk
Vaak zitten er tussen ex ante en ex post een coherentie, in deze situatie kan je echt aan
beleidsleren doen (policy learning). Maar in de EU zijn er veelal procedurele, politieke en
institutionele factoren die ervoor zorgen dat er niet wordt ge policy learned op basis van de
non-coherente relatie tussen ex ante en ex post evaluatie.
Dit zijn de factoren genoemd vanuit artikel 2 (mergaert - ex ante and ex post evaluation)
3
,Ook in Nederland zijn er weerbastige beleidspraktijken. Alle drie de factoren die worden
genoemd die zorgen er ook voor dat het in Nederland niet goed loopt. Als
beleidsonderzoeker moet je echt op zoek gaan naar ‘’die truth’’ om policy making en policy
programming te verbeteren.
Beleidsonderzoek is heel anders dan wetenschappelijk onderzoek doordat je met veel meer
belangen te maken hebt van; politici, beleidsmakers, media, doelgroep en de lobbyisten
bijvoorbeeld. Er is dus een complex maatschappelijk veld.
Als onderzoeker wordt je onderdeel van de weerbarstige beleidspraktijken.
2. Literatuur
2.1 Patty - Policy evaluation in Europe
Pattyn, V., Van Voorst, S., Mastenbroek, E., & Dunlop, C. A. (2018). Policy evaluation in Europe. In The Palgrave Handbook of
Public Administration and Management in Europe (pp. 577-593). Palgrave Macmillan, London.
In recent decades has there been a lot of attention for using evidence to inform policy. The
demand to establish the policy in evidence can be seen as the result of at least three
interlocking trends. (1) There is an increased interest in the economy, efficiency and
effectiveness of public policies. (2) Policy making has become increasingly complex, policy
4
,issues are more and more intertwined, both horizontally and vertically. (3) There is a desire
for enhancing social responsiveness and societal support by involving citizens in the policy
process.
Related to the evidence-based policy has there been a lot of attention for policy evaluation,
in this book they conceive policy evaluation as ‘’the scientific analysis of a certain policy or
part of a policy, aimed at determining the merit or worth of the policy on the basis of certain
criteria'’. They uncover three types of policy evaluation: ex ante, ex durante and ex post.
2. Waves of policy evaluation in Europe
The first wave of evaluation in Europe has its origins in the US, where policy evaluations
were used to analyse how states implemented federal policies. The European countries who
were part of this wave were mostly in Northwestern Europe. After WW2, these countries
developed mixed economies, in which government interventions were viewed as a
necessary instrument to correct the disadvantages of open markets. From the 1960’s and
onwards there was an increasing demand for the empirical and statistical information on the
functioning of policies in order to plan good interventions, which triggered a need for policy
evaluations. They used the randomized experiment for these evaluations and the
researchers were expected to remain neutral.
The second wave which started during the 1990s, moved evaluation across countries in
Central, Southern and Eeastern Europe. Two ‘’drivers’’ pushed the evaluation during the
second wave” (1) The NPM paradigm: the ideal to make governments more efficient and
businesslike, ‘’accountability’’ became important. (2) The pressure of the EU, the EU divided
money and wanted to check if the funds were well spent. Methods used in the second wave
of evaluating were associated with benchmarking and quantitative performance
measurement.
The pressure of the EU caused the eastern ministries to have their evaluation activities in
the national ministries, whilst the South- and Northwestern countries have more
decentralized evaluation systems. Also because of the EU, are second wave evaluations
more concentrated in the policy fields funded by the EU, than the first wave evaluations.
3. Present-day evaluation culture and capacity in Europe: three case studies
In this chapter are three evaluation cultures and evaluation capacities of three political
systems mapped: The UK, Flanders and the EU.
3.1 Measuring the evaluation culture and capacity of countries
Defining and measuring evaluation culture and capacity is a challenge, since the meaning of
the concepts are ambiguous and often contested.
The literature on these two concepts also reveals a lot of different indicators. It also
sometimes distinguishes between evaluation demand: the value attached to evaluations in a
given system and evaluation supply: the resources and strategies essential for conducting
evaluations. In this contribution refers evaluation culture to a shared understanding of the
importance, functions and roles of evaluation. And evaluation capacity refers to the
resources and strategies used to realize evaluations.
There is also relative little academic literature on the operalization of the concepts evaluation
culture and capacity. An exception is the International atlas of evolution, which has nine
indicators to systematically score, measure and compare countries. In this chapter they use
the following indicators per concept:
Evaluation capacity:
● An evaluation society exists
5
, ● There are institutional arrangements in the government (executive branch) for
conducting policy evaluations and disseminating results.
● There are institutional arrangements in parliament for conducting policy evaluations
and disseminating results.
● Policy evaluations occur within the supreme audit institution.
Evaluation culture:
● Policy evaluation takes place in many policy domains.
● Policy evaluations do not just focus on inputs/outputs, but also on outcomes.
3.2 Case study: the United Kingdom
The evaluation in the UK was driven by two factors: (1) the increasing professionalization of
public management which manifested itself in the creation of the Central policy review staff
(2) the desire to control public spending in challenging fiscal environments which resulted in
an International Monetary Fund. By the 1980s the processes of cost evaluation both ex ante
and ex post were well established, but it was not until the 1990s that these were formalized
and routinized. With the arrival of the first labour administration was there a ‘third’ way of
doing politics where ideology is replaced with pragmatic thinking. The challenge for the
government is to link ex ante appraisals (where costs are estimated) with ex post evaluation
(where they are realized).
The UK’s institutional arrangements for conducting and disseminating evaluations are
strong. Evaluation tasks are decentralized to departments and each department is led by a
Chief Researcher. The evaluation features are important since considering the executive
and legislature are closely entwined. This interconnectedness leaves the UK vulnerable to
political rather than managerial agendas. Although the evaluation is reactive - it is usually
part of committees triggered by crises or high profile policy failures -, the select committees
are able to use expert witnesses from inside the civil service and outside the government. So
it does not fit in with their definition of evaluation as a systematic form of policy review, but
their access to experts is a power.
Turning to the supreme audit institution, they see a stronger role for the UK parliament that
created a special body that provides select committees with reports on the policies.
They also emphasize the importance of understanding outcomes by capturing data on cost
effectiveness.
3.3 Case study: Flanders
Both NPM and the European Union played a big role in Flander. The public sector reforms
(Beter Bestuurlijk Beleid (BBB) were modeled along NPM blueprints, in this framework was
the importance of evaluation recognized and new were new tasks and instruments
concerning evaluation introduced. Departments were responsible for policy supporting tasks,
whereas agencies generate the input by means of relevant policy and managerial
information for policy information. The EU played a role in which compulsory evaluation
requirements associated with EU training and manuals spurred the building of evaluation
capacity and culture.
With BBB the institutional arrangements have improved, as they tried to anchor policy
evaluation in the policy cycle. A common trend is to embed evaluation requirements in
legislation. At the regional level have a number of ex ante tests been introduced. Contrary to
the legislature, they have also been trying to anchor policy evaluation within the executive,
but this has not worked very well and Flanders follows the international trend. The court of
Audit in Belgium has gotten its competence for performance audits, which can be seen as ex
6
