Policy evaluation - reader content
27 September 2021 11:09
1. Introduction
Question: how does policy evaluation differ from policy analysis?
Conclusion: because of its normative focus
Evidence:
• Policy analysis → insights in the content of policies and the stakeholders involved
• Policy evaluation → formulation of a judgement about the content of the policies, their (side) effects and/or the
processes through which a policy is developed and/or put into action
Question: how is policy evaluation defined?
Conclusion: a careful, in most cases retrospective assessment of merit, worth, and value of the administration, output,
and/or outcome of a policy on the basis of certain normative criteria such as effectiveness, efficiency, legitimacy, or
sustainability (Vedung 2005)
Question: which questions can policy evaluation answer?
Conclusion:
• What are the aims of policy evaluation?
• What types of policy evaluation can be chosen?
• What criteria can be chosen to evaluate policies?
• How can these evaluation criteria be measured?
• What are possible impacts of the policy evaluation?
2. What are the aims of policy evaluation?
Question: what is the main rationale underlying policy evaluation?
Conclusion:
• the idea that an evaluation can help to improve policies, their implementation and/or their effectiveness
• Can be a means for management and control
• Learning from evaluation -> increase knowledge about effectiveness and impacts of policy & may result in questioning
the original policy aims -> revision of policy aims
Evidence: Policy evaluation can be a formal requirement as a means for creating transparency and accountability. Many
European Union policies and directives, for example, include an explicit legal obligation for an evaluation of the policy
Question: how can policy evaluation be strategically requested and used?
Conclusion:
• Show success or failure of particular policies
• To glorify or condemn the persons responsible for a certain policy
• To praise or criticize the ideology and assumptions on which the policy was based
Evidence: aim evaluation = to create awareness and publicity, which can then be used to further legitimise and support a
policy or replace it with an alternative policy
3. What types of policy evaluation can be chosen?
Question: how can types of policy evaluation be distinguished?
Conclusion: om relation to their:
• Timing (when)
• Focus (what)
• Purpose (why)
Question: explain different types of policy evaluation in relation to their timing
Conclusion: a distinction can be made between evaluations that are carried out prior to the adoption of a policy,
evaluations that are carried out on the go, and those that aim to look back
Evidence:
Ex ante Ex post Ex nunc
Timing Prior to its adoption and After a policy has been adopted Addresses current, ongoing
implementation and implemented policies
Example An environmental and/or socio-
economic impact assessment
which can (or sometimes has to)
be carried out before a policy or
project is undertaken, such as
major infrastructural works like a
motorway or a large dam
Provide insight in the possible outcomes An evaluation of the policy: did it Often to assess if modification is
and effects of a policy, work or not and why? Can pose as desired
programme or project that can be a learning objective
used to choose between different
alternatives or to improve
proposed policy designs
Problematic? No, often mandatory with regard Yes, because policies are hardly No, solution to the problem of
to environmental impact ever (fully) complete ex post evaluations
assessment
Question: explain different types of policy evaluation in relation to their focus
Conclusion: those that focus mainly on effects, those that also focus on the processes through which these effects are
produced, those that focus on the (quality of the) content of policies, and those that focus on the costs and benefits of the
Policy Evaluation and Design Page 1
,produced, those that focus on the (quality of the) content of policies, and those that focus on the costs and benefits of the
policy
Evidence:
Effects Process Content costs Purpose
Focus • Goal achievement • The role of the quality of the Cost-benefit • Summative
• Intended and different policy content analysis: evaluations
unintended stakeholders • Formative
effects • Progress evaluations
• Distributional • Invested resources • Responsive
effects evaluations
Focus on the ways
in which certain
effects are
produced +
reflection of
underlying goals
When used? formative Can be used after a to assess the
evaluations that aim reconstruction of different types of
to facilitate learning the policy theory costs and benefits
during the policy of the policy
implementation To answer the (whether or not in
process questions: monetary values)
• to what
extent is the
policy
evidence-
based?
• How feasible
is the
execution of
the policy?
Aim to modify policies to analyse whether An assessment can To find out whether to enhance learning
to increase their the process is still be carried out on the benefits of the through the very
effectiveness or going in the right whether the policy policy process of the
efficiency direction and theory is complete implementation evaluation, often in
whether it needs to and consistent outweigh its costs close cooperation
be changed or even with relevant
stopped stakeholders
Question: what are summative, formative and responsive evaluations?
Conclusion: newer types of evaluations aimed at enhancing learning through the very process of the evaluation, often in
close cooperation with relevant stakeholders
Evidence:
Summative evaluation Formative evaluation Responsive evaluation
When when new decisions about an while the policy is being implemented engages stakeholders to assess a
used? already implemented policy policy in light of the values and
are about to be taken practices of these stakeholders, can
occur before, during or after a policy
is implemented
Aim to inform such decisions by to facilitate learning and improvement • to strengthen mutual understanding
assessing whether and how a by providing the insights needed to as a basis for the improvement of
policy should be continued, rethink and revise the way in which a practices
revised or stopped policy is implemented • to generate qualitative evidence
about the effectiveness of a policy
based on criteria that include goals
and intentions, but also concerns of
relevant stakeholders
Can more radical adjustments at n incremental adjustments to the • stakeholders are actively involved
result in one point in time, based on policy during the process of its throughout the evaluation process.
decisions to revise or even implementation • They are invited to discuss about
stop the policy evaluation objectives, criteria,
methods, and the interpretation of
the findings.
• implies that no standardised method
can be described, since the ‘design’
of the evaluation gradually emerges
and evolves through discussion with
the stakeholders (Abma, 2005)
Example a summative assessment in the a formative assessment in the form of
form of an end-of-term exam a written assignment can provide a
can determine whether or not student with qualitative feedback in
a student has passed the order to improve learning during the
course course
Further reading
Huitema, D., Jordan, A., Massey, E., Rayner, T., Van Asselt, H., Haug, C. & Stripple, J. (2011). The evaluation of climate policy:
theory and emerging practice in Europe. Policy Sciences, 44(2), 179-198.
Policy Evaluation and Design Page 2
, theory and emerging practice in Europe. Policy Sciences, 44(2), 179-198.
http://link.springer.com/article/10.1007/s11077-011-9125-7
Summary • Most policy evaluations take the complexity of climate governance hardly into account, nor do they question official policy goals
• Result: most policy evaluations do not contribute to a more reflexive from of policy
• There is a gap between the theories and practices of policy evaluation
○ Existing theories acknowledge the importance of reflexive and participatory forms of evaluations, but these are hardly used i n practice
• Result: reduces the potential of policy evaluation to offer novel insights and contribute to addressing environmental issues
• Concern of policy evaluators: the development of climate policy has attracted far more attention than the results it delivers in practice
• Climate change wicked problem
• The long timescales over which climate change manifests itself imply that the outcomes of actions taken now to mitigate its worst effects will not be observed for a
considerable time
• Climate policies -> adaption & mitigation
• Problems:
• The evaluation of environmental policy has developed more slowly than in other policy realms
• Key concepts not fully agreed upon
• Methodological challenges remain
• Number of issues debated in the middle ground between rationalist and constructivist theoretical approaches. But consensus on that evaluation should:
1. Be capable of acknowledging and handling the inherent complexity of policy making
2. Be reflexive in challenging both the means and the goals of policy
3. Be participatory in nature
• No consensus on the definition of policy evaluation
• Different stages:
• Assist national parliaments in assessing the lawfulness of government actions
• After WWII -> more administrative, managerial and economic questions relating to the function of governments
• After 1990s-> political questions related to public support of policies
• Commonly used policy evaluation criteria:
Rationalistic approach Middle ground Constructivist approach
• Views policy as a means to • Center stage: complexity, reflexivity and participation • Autonomous character of policy (tends
achieve certain predefined goals • Complexity acknowledged to follow its own course of
• Policy development as a task for a • choice of approach should depend on the object of evaluation and the objective development)
central actor • formal synthesis of the two main approaches by Fischer: • Goal policy evaluation = to offer insights
• Policies evaluated to inform new into the discourses and frames that are
policy making practices used by various actors to make sense of
• Instrumental: allows a principal to the world around them, incl. the nature
assess whether policy goals are of problems and performance of
being met or not policies
• Vital tasks: collecting objective • Evaluation interactive process,
facts and describing the stakeholders included
functioning of programs in light of • No sole evaluator, but process
goals facilitator
• Criticism: limited usability, too • Heart of evaluation: claims, concerns
uncritical of pre-established goals, and issues identified by stakeholders
neglects possible negative side (not goals embodied in policy)
effects, lacks stakeholder • Outcomes supposed to be debated
participation • Criticism: not the only way to promote
learning
• Relevance of complexity, reflexivity and participation to climate change on page 7&8
• Evaluation practices vary greatly within the EU. Evaluation activity at EU level quite high
• Evaluation criteria used:
Policy Evaluation and Design Page 3
27 September 2021 11:09
1. Introduction
Question: how does policy evaluation differ from policy analysis?
Conclusion: because of its normative focus
Evidence:
• Policy analysis → insights in the content of policies and the stakeholders involved
• Policy evaluation → formulation of a judgement about the content of the policies, their (side) effects and/or the
processes through which a policy is developed and/or put into action
Question: how is policy evaluation defined?
Conclusion: a careful, in most cases retrospective assessment of merit, worth, and value of the administration, output,
and/or outcome of a policy on the basis of certain normative criteria such as effectiveness, efficiency, legitimacy, or
sustainability (Vedung 2005)
Question: which questions can policy evaluation answer?
Conclusion:
• What are the aims of policy evaluation?
• What types of policy evaluation can be chosen?
• What criteria can be chosen to evaluate policies?
• How can these evaluation criteria be measured?
• What are possible impacts of the policy evaluation?
2. What are the aims of policy evaluation?
Question: what is the main rationale underlying policy evaluation?
Conclusion:
• the idea that an evaluation can help to improve policies, their implementation and/or their effectiveness
• Can be a means for management and control
• Learning from evaluation -> increase knowledge about effectiveness and impacts of policy & may result in questioning
the original policy aims -> revision of policy aims
Evidence: Policy evaluation can be a formal requirement as a means for creating transparency and accountability. Many
European Union policies and directives, for example, include an explicit legal obligation for an evaluation of the policy
Question: how can policy evaluation be strategically requested and used?
Conclusion:
• Show success or failure of particular policies
• To glorify or condemn the persons responsible for a certain policy
• To praise or criticize the ideology and assumptions on which the policy was based
Evidence: aim evaluation = to create awareness and publicity, which can then be used to further legitimise and support a
policy or replace it with an alternative policy
3. What types of policy evaluation can be chosen?
Question: how can types of policy evaluation be distinguished?
Conclusion: om relation to their:
• Timing (when)
• Focus (what)
• Purpose (why)
Question: explain different types of policy evaluation in relation to their timing
Conclusion: a distinction can be made between evaluations that are carried out prior to the adoption of a policy,
evaluations that are carried out on the go, and those that aim to look back
Evidence:
Ex ante Ex post Ex nunc
Timing Prior to its adoption and After a policy has been adopted Addresses current, ongoing
implementation and implemented policies
Example An environmental and/or socio-
economic impact assessment
which can (or sometimes has to)
be carried out before a policy or
project is undertaken, such as
major infrastructural works like a
motorway or a large dam
Provide insight in the possible outcomes An evaluation of the policy: did it Often to assess if modification is
and effects of a policy, work or not and why? Can pose as desired
programme or project that can be a learning objective
used to choose between different
alternatives or to improve
proposed policy designs
Problematic? No, often mandatory with regard Yes, because policies are hardly No, solution to the problem of
to environmental impact ever (fully) complete ex post evaluations
assessment
Question: explain different types of policy evaluation in relation to their focus
Conclusion: those that focus mainly on effects, those that also focus on the processes through which these effects are
produced, those that focus on the (quality of the) content of policies, and those that focus on the costs and benefits of the
Policy Evaluation and Design Page 1
,produced, those that focus on the (quality of the) content of policies, and those that focus on the costs and benefits of the
policy
Evidence:
Effects Process Content costs Purpose
Focus • Goal achievement • The role of the quality of the Cost-benefit • Summative
• Intended and different policy content analysis: evaluations
unintended stakeholders • Formative
effects • Progress evaluations
• Distributional • Invested resources • Responsive
effects evaluations
Focus on the ways
in which certain
effects are
produced +
reflection of
underlying goals
When used? formative Can be used after a to assess the
evaluations that aim reconstruction of different types of
to facilitate learning the policy theory costs and benefits
during the policy of the policy
implementation To answer the (whether or not in
process questions: monetary values)
• to what
extent is the
policy
evidence-
based?
• How feasible
is the
execution of
the policy?
Aim to modify policies to analyse whether An assessment can To find out whether to enhance learning
to increase their the process is still be carried out on the benefits of the through the very
effectiveness or going in the right whether the policy policy process of the
efficiency direction and theory is complete implementation evaluation, often in
whether it needs to and consistent outweigh its costs close cooperation
be changed or even with relevant
stopped stakeholders
Question: what are summative, formative and responsive evaluations?
Conclusion: newer types of evaluations aimed at enhancing learning through the very process of the evaluation, often in
close cooperation with relevant stakeholders
Evidence:
Summative evaluation Formative evaluation Responsive evaluation
When when new decisions about an while the policy is being implemented engages stakeholders to assess a
used? already implemented policy policy in light of the values and
are about to be taken practices of these stakeholders, can
occur before, during or after a policy
is implemented
Aim to inform such decisions by to facilitate learning and improvement • to strengthen mutual understanding
assessing whether and how a by providing the insights needed to as a basis for the improvement of
policy should be continued, rethink and revise the way in which a practices
revised or stopped policy is implemented • to generate qualitative evidence
about the effectiveness of a policy
based on criteria that include goals
and intentions, but also concerns of
relevant stakeholders
Can more radical adjustments at n incremental adjustments to the • stakeholders are actively involved
result in one point in time, based on policy during the process of its throughout the evaluation process.
decisions to revise or even implementation • They are invited to discuss about
stop the policy evaluation objectives, criteria,
methods, and the interpretation of
the findings.
• implies that no standardised method
can be described, since the ‘design’
of the evaluation gradually emerges
and evolves through discussion with
the stakeholders (Abma, 2005)
Example a summative assessment in the a formative assessment in the form of
form of an end-of-term exam a written assignment can provide a
can determine whether or not student with qualitative feedback in
a student has passed the order to improve learning during the
course course
Further reading
Huitema, D., Jordan, A., Massey, E., Rayner, T., Van Asselt, H., Haug, C. & Stripple, J. (2011). The evaluation of climate policy:
theory and emerging practice in Europe. Policy Sciences, 44(2), 179-198.
Policy Evaluation and Design Page 2
, theory and emerging practice in Europe. Policy Sciences, 44(2), 179-198.
http://link.springer.com/article/10.1007/s11077-011-9125-7
Summary • Most policy evaluations take the complexity of climate governance hardly into account, nor do they question official policy goals
• Result: most policy evaluations do not contribute to a more reflexive from of policy
• There is a gap between the theories and practices of policy evaluation
○ Existing theories acknowledge the importance of reflexive and participatory forms of evaluations, but these are hardly used i n practice
• Result: reduces the potential of policy evaluation to offer novel insights and contribute to addressing environmental issues
• Concern of policy evaluators: the development of climate policy has attracted far more attention than the results it delivers in practice
• Climate change wicked problem
• The long timescales over which climate change manifests itself imply that the outcomes of actions taken now to mitigate its worst effects will not be observed for a
considerable time
• Climate policies -> adaption & mitigation
• Problems:
• The evaluation of environmental policy has developed more slowly than in other policy realms
• Key concepts not fully agreed upon
• Methodological challenges remain
• Number of issues debated in the middle ground between rationalist and constructivist theoretical approaches. But consensus on that evaluation should:
1. Be capable of acknowledging and handling the inherent complexity of policy making
2. Be reflexive in challenging both the means and the goals of policy
3. Be participatory in nature
• No consensus on the definition of policy evaluation
• Different stages:
• Assist national parliaments in assessing the lawfulness of government actions
• After WWII -> more administrative, managerial and economic questions relating to the function of governments
• After 1990s-> political questions related to public support of policies
• Commonly used policy evaluation criteria:
Rationalistic approach Middle ground Constructivist approach
• Views policy as a means to • Center stage: complexity, reflexivity and participation • Autonomous character of policy (tends
achieve certain predefined goals • Complexity acknowledged to follow its own course of
• Policy development as a task for a • choice of approach should depend on the object of evaluation and the objective development)
central actor • formal synthesis of the two main approaches by Fischer: • Goal policy evaluation = to offer insights
• Policies evaluated to inform new into the discourses and frames that are
policy making practices used by various actors to make sense of
• Instrumental: allows a principal to the world around them, incl. the nature
assess whether policy goals are of problems and performance of
being met or not policies
• Vital tasks: collecting objective • Evaluation interactive process,
facts and describing the stakeholders included
functioning of programs in light of • No sole evaluator, but process
goals facilitator
• Criticism: limited usability, too • Heart of evaluation: claims, concerns
uncritical of pre-established goals, and issues identified by stakeholders
neglects possible negative side (not goals embodied in policy)
effects, lacks stakeholder • Outcomes supposed to be debated
participation • Criticism: not the only way to promote
learning
• Relevance of complexity, reflexivity and participation to climate change on page 7&8
• Evaluation practices vary greatly within the EU. Evaluation activity at EU level quite high
• Evaluation criteria used:
Policy Evaluation and Design Page 3
