Nielsen Main question: To present an evaluation framework based on recent intervention research and process-oriented organization theory
Findings: Mechanisms (process) can explain outcomes. Employees as targets of change, management is crucial. Phases from
Evaluation initiation to implementation should be considered linear change phases. Pre- and post-measurement of expected outcomes are
Framework, important. KeyConcepts: Effective evaluation: movement from one fixed state to another fixed state (outcome of the intervention),
Process change is always occurring, Process evaluation: Looks at how certain interventions are interpreted and acted upon by people in the
evaluation,
Mental
organization, with conditions that can hamper or strengthen them (process of the intervention, what’s happening during the change),
models, evaluation at every stage, process and outcome intertwined, Omnibus context: how the intervention fits with the culture and
Outcome conditions of the group. Discrete contexts: what specific events may have influenced effects of the intervention? Organizational
evaluation,
M/E: actors: (middle manager crucial for change) employees, management, senior management and middle managers. Mental models: are
Managers, about sense-making, why do you do the things you do. This influences actors’ behavior and thus intervention outcomes. These are
Employees also individual but may become shared over time. Evaluation framework: Consists of: Organizational actors (first category), Mental
models of those actors (second category), Contextual factors (third category), Design and processes (fourth category), consisting of
five phases: Initiation Screening, Action planning, implementation. Mental models: different views of the world by different people
(what do people think) Important mental models: Do M/E find the intervention if benefit, what do M/E think of management
activities. Type of change effect on evaluation approach.
Visse Main question: interpretive framework for understanding the moral aspects of evaluation practice. Responsive evaluation: to create a mutual understanding of the social practice at hand (not
necessarily shared!). It is a reflective practice which requires openness from the evaluator him/herself, it is relational work Mutual understanding: People involved know each other’s perspective
Responsive and it is ok to have different perspectives. Shared understanding: people have or create the same perspective Morality: taking care for people involved in evaluation and the cause. Shaped in
Evaluation interaction with stakeholders in daily practice. Evaluator cares, takes responsibility for that (unknowingly) Being an evaluator depends on: Identity: how is the evaluator seen by themselves and
Kirkpatrick: others? Relation: on which relationships does the evaluator (not) focus and why? Values: what values matter in responsive evaluation and evaluation practice? Report of evaluation: Thick
Rea – Lea – description. Steps: Identify Stakeholders - find a variety of perspectives – Find joint perspectives – Find homogenous groups – Find Heterogenous groups. Important: check if there are other
Beh - Res perspectives, Sorting out different themes, in-depth validation of findings. Idiosynatric perspectives: Individual persepctives
Salas What matters: Before training: Training Needs Analysis (TNA) (what needs to be trained), Learning climate (factors that make learning possible: expectations, attendance necessary)
Training During Training: Individual characteristics (self-efficacy, goal orientation, motivation), Instructional strategies and principles. Technology based training. After training: Transfer (supportive
climate, debriefing, maximize knowledge elicitation in the job), evaluation (did the training do its job). Transfer is about behavior
Blume Main question: to explore the impact of predictive factors on the transfer of training in different tasks and contexts. Findings Transfer consists of 2 major dimensions: generalization: the extent to which the
Transfer knowledge and skill acquired in a learning setting are applied to different settings, people, and/or situations from those trained. (applying knowledge). Maintenance: the extent to which changes that result from a
learning experience persist over time. (knowledge retention) KeyConcepts: Transfer: Occurs if the aims, method and approaches used for the learning task are like the transfer task Professional Learning/Team
learning/Organizational Learning Near transfer: Transfer tasks that are highly like the learning tasks (vb. Working on small engine jet in training and a large engine jet in the field). Far transfer: Soft skills
Where the tasks and situations in the learning situation are quite different from the transfer setting. (vb. Applying principles of electricity from t raining to troubleshooting complex mechanical) Lateral transfer:
When a skill spreads over a broad set of situations at the same level of complexity or difficulty. (vb. Applying learned rules and procedures in multiple situations). Vertical transfer: Occurs when an acquired skill
affects the acquisition of a complexer/ simpler skill. (e.g. a pilot also learning about communication)
Botke; Adv: Main question: Occurs when an acquired skill affects the acquisition of a more complex or superordinate skill. (e.g. a pilot gaining knowledge and skills related to flying a plane and then having to
practical, learn the appropriate team interrole behaviours required for effective flight cockpit performance. Keyfindings: Transferstages: motivation transfer, use of new behaviors, individual
conceptual
performance improvement, Improvement of organizational performance Work factors influencing soft skill transfer: job-related factors, social support variables, organizational facilitation of
Dis:
P motivation, learning. Supervisors are important for transfer, Transfer enhancing interventions: goal setting, relapse prevention, and using optimistic previews, can positively impact transfer outcomes. H, It
f learning, emphasizes the importance of studying the mediating variables through which these interventions influence transfer. Self-efficacy and organizational culture are mediators in this process.
outcomes organizations should emphasize the relevance of using new skills in the work context, provide support and coaching, and make trainees aware: potential challenges, the possibility of relapse,
encourages use of new behaviors. Micro to macro transfer. Optimistic previews important
Sofo Action Learning: through action or during the action. helpful or essential infrastructure that supports reflective inquiry. Fundamental components of action learning are real-world action, learning from shared
experience, involves several individuals in the process and complex problems (ill-structured) Reflective inquiry: the practical aspects of learning rather than teaching. Reflection on what you do, see and hear.
Asking questions is the inquiry aspect. Self-reflection is central. Reflexive recognizing mental patterns and patterns of communication within the team that promote or undermine learning/Discovery Questions
help to articulate the problem and understand the complexity of the problem. Levels of learning: Individual/team/group learning. Coach important: asking questions, equity, responsibility, efficiency, Iterative
process/double loop learning, output zorgt voor input, je achterliggende gedachten veranderen
Guba Paradigms: basic beliefs Positivism: Hypothesis driven allows predictions. Best way to obtain knowledge is through unbiased, objective research with limited researcher influence.
Constructivism: There are multiple realities. These realities are locally (socially) constructed (ontology). Interprets instead of tests (descriptive research). Researcher inseparable from
Paradigms, phenomenon under study, no such thing as objectivity. Distinguishing cause and effect is impossible because both are shaping each other. Participatory: Reality is subjectively defined
positivism, by one's experience. Humans are part of a whole, no gods (ontology: what is reality), being present in the world is the ground of our being and knowing (epistemology: what is
constructivis knowledge), we have obligations towards others with whom we are linked (Axiology). Both qualitative and quantitative research can be both paradigms. Methodology: how do we
m
gain knowledge Participatory: Onderzoeker doet zelf mee, reflecteren,
Verdonschot Findings: Evaluation studies that aim to measure effects on the level of behavior and impact, carry learning potential. This learning potential in certain moments: generative
moments. Influential moments: creating the impact map, in-depth interviews, final presentation. Learning from generative moments is possible when the people involved work on
Generative a concrete product that then serves as a boundary object and thus connects stakeholders with different backgrounds. Success case method: mixed method approach that combines a
moments short survey combined with in-depth interviews. Impact map: creates a visual depiction of the learning process. It highlights roles, interactions and results. One of the elements of this
map are the work behaviors that the learning intervention needs to promote. Generative moments: moments that vitalize, and create energy, and give room to new ideas to emerge.
regarded as process yields of an investigation that stimulate the development of stakeholders. Most moments take place during presentation.