CALIBRATING TEST ITEM BANKS FOR AN INTRODUCTORY STATISTICS COURSE

CHAPTER 1 INTRODUCTION In modern education, testing has become an important tool for measuring a student’s knowledge of a subject. In smaller classes a teacher might design a test specifically for the group of students in the course at the time. In a larger class, particularly for one that has multiple sections and/or instructors, one common test may be given to all of the students in the course in a given semester. One reason for doing this is to ensure that the measurement of student ability consistent across sections, instructors and time. The concern in using such common test for large groups is whether the questions on the tests are actually valid and reliable when it comes to measuring student ability. Over the last few decades the importance of statistics has grown in many fields, and with this growth the importance of statistics education has also increased. Many degrees now require students to take an introductory statistics course as part of their undergraduate program. To compensate for the increasing number of students enrolled in introductory statistics courses resulting from the increase in requirements, universities are offering these introductory courses in large coordinated lecture format. Due to this increased emphasis on these courses for general education of undergraduates, statisticians are becoming more involved in the reform of statistical education, particularly with the introductory statistics courses (Garfield, Hogg, Schau, & Whittinghill, 2000). An example of this type of large scale coordinated lecture is the introductory statistics course offered at the University of Georgia: STAT 2000. It satisfies the quantitative reasoning, 1 general education requirement for undergraduate students at UGA. The UGA undergraduate bulletin for the academic year states that the courses includes the collection of data, descriptive statistics, probability, and inference. Topics include sampling methods, experiments, numerical and graphical descriptive methods, correlation and regression, contingency tables, probability concepts and distributions, confidence intervals, and hypothesis testing for means and proportions (University of Georgia, 2014). Each fall and spring semester approximately 1300 students take the STAT 2000 course. The students are separated into multiple large lecture classes with approximately 200 students in each lecture. These lectures are taught by three to five different lecturers each semester. Accompanying the lectures are weekly labs that students must attend. These labs are taught by teaching assistants from the Department of Statistics. For the fall and spring semesters there are typically nine to twelve total teaching assistants each semester each of whom are responsible for running from one to five of the 47 lab sections associated with the courses. The entire course is overseen by a coordinator who chooses the material, specifies order in which the material is taught, and constructs the tests, homework assignments, and labs that constitute the students’ grades. In the labs, students answer questions through the online testing software WebAssign ( WebAssign testing software. At the end of the semester there is an optional fifth test: a cumulative assessment available for students who wish to have the opportunity to make up a 2 missed test or improve their grade. All students receive the same test questions during a semester, but the tests are changed slightly for each student by having the numbers within the question and order of the questions and responses randomized. This ensures the students receive a similar test, in terms of content and difficulty, while preventing possible cheating, which can be difficult to monitor with computer-based testing administered to 1300 students over the course of two days. The students are given 50 minutes to complete each test. Each test consists of multiple choice and fill-in-the-blank calculation questions from an item bank that is cycled through over a period of three years. Having an item bank from which to pull items raises the question, “Are all of these items properly assessing students’ statistical achievement?” Tests can be studied using various perspectives and the items on the test can be evaluated according to different theories of measurement. In chapter 2, the two major theories in educational measurement, Classical Test Theory (CTT) and Item Response Theory (IRT), are discussed. The principles for each theory are discussed along with how tests are evaluated under each theory to determine if they are functioning properly, and the one that is best for calibrating tests in STAT 2000 is identified. Then in chapter 3, we discuss the various methods for performing the analysis along with testing for goodness-of-fit and describe the data that were collected. In chapter 4, we discuss our analysis and results. Finally in chapter 5, we conclude with a discussion of future research possibilities as well as things that could be done to improve this analysis in the future. 3