Collaborative Evaluation Designs as an Authentic Course Assessment

Collaborative Evaluation Designs as an Authentic Course Assessment Michelle Searle Queen’s University Cheryl Poth University of Alberta Abstract: Strategies for optimizing evaluator education involve ongoing discussions within the field of evaluation. There remains unrealized potential for authentic assessments to support and measure learning that demonstrates the integration of evaluation theory with practice. In this practice note, we advance collaborative evaluation designs as a useful authentic assessment for generating the type of applied learning evidence required within an evaluation­focused graduate course. First, we provide a brief review of educational literature, highlighting the value of authentic assessments and a description of the contexts where we effectively applied the collaborative evaluation design. Next, we offer details of the four instructional processes involved in the authentic assessment: planning, development, feedback, and measurement. Finally, we advance lessons learned from our instructional experiences and the learner benefits of the collaborative evaluation design as an authentic assessment. Keywords: assessment, post­secondary, teaching evaluation Résumé : L’amélioration continue de la formation en évaluation dépend, en partie d’échanges parmi ceux et celles qui en sont chargés. Il existe toujours un potentiel inexploité d’évaluation authentique visant à appuyer et à mesurer l’apprentissage qui démontre l’intégration de la théorie et la pratique. Dans la présente note sur la pratique, nous présentons des modèles d’évaluation collaborative comme outils d’évaluation authentique et utile pour arriver au type de preuve d’apprentissage requis dans un cours de cycle supérieur en évaluation. Premièrement, nous faisons un bref examen de la littérature pédagogique, en soulignant la valeur des évaluations authentiques et une description des contextes dans lesquels nous avons appliqué effi cacement le modèle d’évaluation collaborative. Ensuite, nous décrivons de façon plus détaillée les quatre processus d’enseignement qui font partie d’une évaluation authentique : la planifi cation, l’élaboration, la rétroaction et la mesure. Finalement, nous faisons part des leçons apprises de notre expérience d’enseignement et des avantages, pour les apprenantes et apprenants, du modèle d’évaluation collaborative comme évaluation authentique. Mots clés : évaluation, postsecondaire, enseignement de l’évaluation Corresponding author: Michelle Searle, Faculty of Education, Duncan McArthur Hall, 511 Union Street, Queen’s University, Kingston, ON, K7M 5R7; © 2021 Canadian Journal of Program Evaluation / La Revue canadienne d’évaluation de programme 35.3 (Special Issue / Numéro spécial), 422–436 doi: 10.3138/cjpe.69691 Collaborative Evaluation Designs as Course Assessment 423 Strategies for optimizing evaluator education involve ongoing discussions within the fi eld of evaluation (e.g., Chouinard et al., 2017; Gullickson et al., 2019; Poth et al., 2020; Trevisan, 2004). Post­secondary contexts are well known for their use of assessment strategies such as tests, quizzes, and essays to measure learning at the end of a course. While these strategies may be well suited for measuring learners’ acquired knowledge of specific evaluation concepts, they may be insuffi cient for generating reliable and valid evidence of applications of complex evaluation skills in practice. The need for this type of learning experience is well established in the field; Gredler and Johnson (2001 ), for example, reflected on the value of directed evaluation experience as an adjunct to course­based work, citing the knowledge broadened by encountering real­world issues and the validation of emerging professional identities. Indeed, learners and instructors of evaluationfocused courses may well be better served by assessments that measure learning outcomes focused on the practical application of knowledge and skills (Knight, 2012). Yet there remains unrealized potential for authentic assessments within evaluation­focused coursework, evaluations that support and measure learning that demonstrates the integration of evaluation theory with practice. As evaluation educators who have extensive training and experiences as instructors within K­12 and post­secondary education systems and who are well­established assessment scholars, we are well positioned for advancing the use of authentic assessments. We draw from our knowledge and research products highlighting the benefits of authentic assessment for both learners and instructors to bring the usefulness of authentic assessment into the dialogue of evaluator education. As one of the challenges of doing so is using fi eld­specifi c terminology, and because prominent evaluators remind us that language matters (e.g. Hopson, 2000; Patton, 2000), we pause for a moment to clarify a key term. While the word authentic is oft en used within educational literature to describe tasks, rubrics, and assessments that focus on real­world practice (Gulikers et al., 2008), we use the term authentic assessments to distinguish the generation of learning evidence within the integration of evaluation theory with practice. Th is usage aligns with that of Wiggins (1998 ) as a way to identify assessments when the learner is asked to complete a task that is highly similar to what they would encounter in their future professional practice. Authentic assessments within evaluator education place an emphasis on the learning processes that occur within real­world contexts and direct attention to the need for, and ways of, integrating feedback and reflection to extend learning. We advance authentic assessment as a term to be leveraged by evaluator educators to describe the processes involved in supporting and measuring desirable applied learning evidence that is specifi c to evaluation. In this practice note, we advance collaborative evaluation designs as a useful authentic assessment for generating the type of applied learning evidence required within an evaluation­focused graduate course. First, we provide a brief review of educational literature, highlighting the value of authentic assessments and a description of the contexts where we effectively applied the collaborative evaluation doi: 10.3138/cjpe.69691 CJPE 35.3, 422–436 © 2021 424 Searle and Poth design. Next, we offer details of the four instructional processes that involved the authentic assessment: planning, development, feedback, and measurement. Finally, we advance lessons learned from our instructional experiences and the learner benefits of the collaborative evaluation design as an authentic assessment. THE USEFULNESS OF AUTHENTIC ASSESSMENTS Within the landscape of educational assessments, authentic assessments are those in which learners are required to demonstrate combinations of knowledge, skills, and disposition in a real­life context (Maclellan & Soden, 2004; Mueller, 2005; Wiggins, 1993 ). The value of authentic assessments is premised on research about situated learning (Stein, 1998) and social constructivist theory (Kalina & Powell, 2009). Authentic assessments contribute to thinking about assessment as being integrated throughout the learning process; as described by Maclellan and Soden (2004 ), “within these perspectives, assessment is not only about judging how much people know but judging how, when and whether they use what they know” (p. 316). Th e benefits of this for evaluator education include directing learners’ attention to what is valued—not just accumulated knowledge but also its application in real­world contexts. We frame seven key characteristics for authentic assessment: 1. involving learners in setting goals and assessment criteria; 2. performing a task, which includes locating resources to develop ideas and creating an artifact or product over time and in a real­world setting/ professional context; 3. engaging in continuous dialogue with others (e.g., peers, instructors, community partners) for the purpose of ongoing formative assessment; 4. using higher­level thinking and/or problem­solving skills; 5. integrating prior knowledge with inputs from classroom instruction and resources to create a product 6. demonstrating knowledge, skills, and attitudes related to a particular concept; and 7. advancing metacognitive, collaborative, and interpersonal skills as well as intellectual products (Airasian, 2001; Linn & Miller, 2005; Wiggins, 1993). We are not alone in our application of authentic assessments; indeed, their usefulness is well established in vocational training and competency­based programs where learners integrate theory and practice and move toward a mastery of skills needed in their future professional roles (Gulikers et al., 2008). Authentic assessments have been shown to positively affect learners’ knowledge and motivation (e.g., Herrington & Herrington, 1998; Raymond et al., 2013). Motivation may increase when assessments align with learners’ perceptions for enhancing their knowledge and/or engaging in practical processes (Fook & Sidhu, 2010; Gulikers © 2021 CJPE 35.3, 422–436 doi: 10.3138/cjpe.69691 Collaborative Evaluation Designs as Course Assessment 425 et al., 2008). Surely learners and instructors would welcome the motivational and professional learning benefits of authentic assessments in evaluation­specifi c coursework. OUR TEACHING CONTEXTS FOR EVALUATION-FOCUSED AUTHENTIC ASSESSMENTS The collaborative evaluation design as an authentic assessment has become a key feature of our instructional approaches to teaching evaluation­focused courses at two Canadian institutions over the past decade. Its usefulness goes beyond its support and measurement of specific Canadia