This study document covers all the content needed for the PSY3007S Psychometrics section. It has content from lecture slides, explanations from lecturers, summaries of readings, as well as helpful comparison tables and summaries of content to aid studying. The document is well organised to help mak...
PSY3007F Psychometrics
PSY2015F Summary
- psychometrics science or study of psychological measurement
o Measurement of mental capacity/thought process/personality aspects
o Especially by mathematical/statistical analysis of quantitative data
o Construction and application of psychological tests
o measurement assignment of numbers to objects and events according to rules
Assignment of numbers to individuals in a systematic way as a means of representing
o Test measurement device/technique used to quantify behaviour > aid in understanding and prediction
- Defining psychometrics
o Psychometrics measure many different things
o Psychometrics aim to operationally define and quantify the things it measures
o Psychometrics is concerned with constructing tests to measure things and evaluating how ʻgoodʼ these
tests are
o Psychological measurement is the measurement of human behaviour
Overt > behaviour observable
Covert > behaviour mental/social/physical action
Rust & Golombok, 2009
Modern Psychometrics: Constructing your own questionnaire
- Questionnaires > used to measure a wide variety of attributes and characteristics
Purpose of the questionnaire
- Write down clearly and precisely the purpose of the questionnaire
Making a blueprint
- blueprint (aka test specification) > framework for developing the questionnaire
- usually has content areas on horizontal axis and manifestations along vertical axis
o between 4 and 7 categories along each axis > otherwise too narrow or too cumbersome
- content areas > write down content areas to be covered by questionnaire
o if not clear cut > consult experts in field
- manifestations > write down ways in which content areas of questionnaire may become manifest
o psychological questionnaires > behavioural, cognitive, affective manifestations
- draw blueprint, labelling each content area (column) and each manifestation (row)
o each cell in blueprint represents an interaction
- decide on weightings
o assign percentage to each content area so that the total percentages of content areas adds up to 100%
o assign percentage to each manifestation area so that the total percentages of manifestation areas
adds up to 100%
o insert percentages into blueprint
- decide how many items to include in pilot questionnaire
o take into account > desired final number of items, size of blueprint, time available for testing,
characteristics of respondents
o balance number of items and compliance from respondents
o usually minimum 20 items
o allow 50% more items in blueprint than intended for final version
- enter number of items to be written for each cell into blueprint
Writing items
- Item Type > decide which type is most appropriate for questionnaire
o Alternative choice item an item for which respondent is given two choices from which to select a
response
Use > knowledge based questionnaires, sometimes personality
:) assessing knowledge of facts/comprehension of material, fast, easy to use
:( correct response often not clear cut for ability/aptitude/achievement, 50% chance guessing
correct, restricting
o Multiple choice item an item for which respondent is given more than two choices from which to
select a response
, Two parts > stem (statement or question containing problem), options (list of possible
responses with one correct and other distractors)
4/5 responses best to reduce probability of guessing correctly
Use > knowledge based, not used person based
:) well suited to variety in ability/aptitude/achievement questionnaires, easy administering
and scoring, effects of guessing reduced
:( time and skill needed to write multiple choice items, not all options effective reduces
reliability
o Rating scale item item for which possible responses lie along continuum
hard to meaningfully differentiate above 7 options
options in rating scales are interdependent while MCQ options are independent of each other
Use > widely used person based questionnaires
:) more precise expression
:( varied interpretations of response options, bias towards extremes or neutral answers
Type of option > should be chosen to suit material to be presented in questionnaire
Number of options > based on nature of questionnaire
- All questionnaires
o Make sure items match the blueprint
o Write each item clearly and simply, avoid subjective words, avoid double-barrel items
- Knowledge based questionnaires
o Make sure alternate choice items are clear cut > can be classified without a doubt as true/false
o For multiple choice > make sure each item has only one correct/best response
- Person based questionnaires
o Acquiescence tendency to agree with items regardless of their content
Reduce this by reversing items
check that reversed item has same meaning as original item, avoid double negative
o social desirability tendency to respond to an item in a socially acceptable manner
exclude items which are clearly socially desirable or undesirable
o indecisiveness tendency to use ‘don’t know’ or ‘uncertain’ options
eliminate neutral/middle categories
o extreme response tendency to choose extreme option regardless of direction
reduce by using clear, unambiguous, specific items
- ordering of items
o pick interesting and unthreatening item to start with
o randomize the rest, make adjustments if too many similar looking items together
o knowledge based items > order of increasing difficulty from easy to hard
Designing the questionnaire
- good design is crucial for producing a reliable and valid questionnaire
o clear layout and easy to understand > less intimidating, taken more seriously
- background information > include headings and space for responses, obtain background information and date
completed
- instructions > clear and unambiguous
o providing information increases compliance (confidentiality, response speed, honest response)
- layout
o Clear visual hierarchy is necessary
o number each item, keep lines short
o ensure straight vertical margin on left hand side of page, arrange response options for straight vertical
margin on right hand side of page
o headings at top
o clear visual relationship between item and response options
o separate each item with a space
o if using more than one type of item > group similar items together, each group will need instructions
o print out questionnaire with large type that is easily readable
o design can be used to portray/disguise purpose of questionnaire
- scoring questionnaire > allocate score to each response option, add up score for each item to produce total
questionnaire score
Piloting the questionnaire
- administer questionnaire and obtain relevant demographic information from a group of people similar to those
for whom the final questionnaire is intended
, o analysis of results helps select best items for final version
- respondents should vary in terms of demographic characteristics
Item analysis
- decide which items from the pilot version of your questionnaire to include in final version
o taking account of facility, discrimination, and, if appropriate, distractors
o order items and design questionnaire as before
- must look at distractors for knowledge based multiple choice items
- facility
o most questionnaires are designed to differentiate respondents according to whatever knowledge or
characteristic is being measured
o a good item is one for which different respondents give different responses
o facility index indicates the extent to which respondents answer an item in the same way
if everyone answers same way > item is redundant > remove
calculated by dividing number of respondents who obtained correct response for item by total
number respondents
- discrimination
o the ability of each item to discriminate respondents according to whatever the questionnaire is
measuring
o items should only be selected for final version if they measure same knowledge or characteristic as
other items in the questionnaire
o measured by correlating each item with the total score for the questionnaire > higher correlation
coefficient = more discriminating item
o pearson product-moment formula used to calculate correlation coefficient
- distractors
o examination of the use of distractor options by respondents who don’t choose the correct/best option
should be carried out too ensure that each distractor is endorsed by a similar proportion of
respondents
- must also consider how may items required for final version, and how well items fit into blueprint
Reliability
- reliability estimate of the accuracy of a questionnaire
- test-retest > administer same questionnaire to same respondents under same circumstances on two occasions
and correlate the scores
- parallel forms > construct two equivalent forms of the questionnaire and administer both to same respondents
in order to correlate the scores
- split half > questionnaire is divided into two halves, and correlation between halves is used to produce an
estimate of reliability for the whole questionnaire
o calculate the split half reliability for the final version of your questionnaire using data from the relevant
items from all of the respondents in the pilot study plus additional respondents if necessary
o for each respondent, calculate total score for even items in the final questionnaire and total score for
odd items
o correlate odd items with the even items using pearson product-moment formula
o use correlation coefficient in spearman-brown formula to obtain estimate of reliability for the whole
questionnaire
Validity
- validity extent to which a questionnaire measures what it intended to measure
o determined in relation to purpose of questionnaire
- face validity > appearance of questionnaire to respondents
- content validity > relationship between content and purpose of questionnaire
- criterion related validity > relationship between scores on the questionnaire and a relevant/reliable/available
criterion measure
- predictive validity > relates scores on a questionnaire to a future criterion measure
standardization
- Standardise your questionnaire using a relevant group or groups of as many respondents as possible
o Present norms in terms of the mean and standard deviation for each group or stratification
- Good norms allow for meaningful interpretation of scores of individuals
- Norms are not always necessary
o Not crucial if comparing groups of respondents
- Must include as many respondents as possible in standardization group to ensure it is representative
, W1 Writing and Evaluating Test Items
W1L1 Item Writing
- Choice of format is decided by the objectives and purpose of the test
o What are the questions that you want to ask?
Item writing guidelines
- Define clearly what you want to measure
o Cannot construct test items if you don’t know what you want to measure
- Generate an item pool
o Ideal scenario > generate more than you need, test with sample
o Best items are selected from pool after item analysis
Item analysis > tells us which items good and which items are bad
- Avoid long items
o If tedious to read, respondents may not understand what you are trying to ask
- Keep reading difficulty appropriate
o Avoid jargon, account for education level
- Use clear and concise wording (grammatically correct)
o Avoid double-barrelled items
e.g. is this tool interesting and useful
o Avoid double negatives
- Mix positively and negatively worded items in the same test
o Can help prevent response sets
o Response sets participants understand what the question is asking, and thus decide their stance and
answer all items in the same way
- Make sure items are as culturally neutral as possible
- MCQ items
o Vary position of correct answer, make all distractors (incorrect answers) plausible
- True/false Qs
o Both statements same length, equal numbers of both
Fowler, 1993 (recommended)
Designing questions to be good measures
- Good questions are reliable and valid
o Answers to questions are only valuable to the extent that it can be shown to have a predictable
relationship to facts or subjective states that are of interest
o Good questions > maximise relationship between answers recorded and what researcher is trying to
measure
- Not always perfect correspondence between answers people give and reality
o When answers are a good measure, it is usually due to careful design
Increasing reliability of answers
- For a good question we need
o Researchers side of question/answer process to be full scripted so that question as written full
prepares respondent to answer
o question means the same thing to every respondent
o kinds of answers that constitute an appropriate response to the question are communicated
consistently to all respondents
- Avoiding inadequate wording
o Incomplete wording, poor wording
o Optional wording can be used when necessary, but must not be probing
- Ensuring consistent meaning for all respondents
o account for different cultural backgrounds and education
o Use words that are short and widely understood
The benefits of buying summaries with Stuvia:
Guaranteed quality through customer reviews
Stuvia customers have reviewed more than 700,000 summaries. This how you know that you are buying the best documents.
Quick and easy check-out
You can quickly pay through credit card or Stuvia-credit for the summaries. There is no membership needed.
Focus on what matters
Your fellow students write the study notes themselves, which is why the documents are always reliable and up-to-date. This ensures you quickly get to the core!
Frequently asked questions
What do I get when I buy this document?
You get a PDF, available immediately after your purchase. The purchased document is accessible anytime, anywhere and indefinitely through your profile.
Satisfaction guarantee: how does it work?
Our satisfaction guarantee ensures that you always find a study document that suits you well. You fill out a form, and our customer service team takes care of the rest.
Who am I buying these notes from?
Stuvia is a marketplace, so you are not buying this document from us, but from seller hollykane. Stuvia facilitates payment to the seller.
Will I be stuck with a subscription?
No, you only buy these notes for $7.77. You're not tied to anything after your purchase.
