Task 2 Fool the assistant
Questions task 2:
- What score do you expect?
Quite high, he seems quite full of himself.
- Does it matter that a different person takes the test?
It can matter yes, because the impression the person gives can differ from the previous person
and can influence the patient.
- What problems arise from this set-up and how can you prevent these?
It can influence the answers the client will give and negatively influence the test results.
- Recall that you yourself filled in the Self Confidence Test and you kept the record of this
at item level. Please, input items responses of all group members at item level into an
SPSS file and calculate Cronbach’s alpha (Analyze>Scale>Reliability Analysis). What is
the Cronbach’s alpha? Is it sufficient?
Parallel forms reliability
Parallel forms reliability measures the correlation between two equivalent versions of a test. You
use it when you have two different assessment tools or sets of questions designed to measure
the same thing.
In educational assessment, it is often necessary to create different versions of tests to ensure
that students don’t have access to the questions in advance. Parallel forms reliability means
that, if the same students take two different versions of a reading comprehension test, they
should get similar results in both tests.
Why it’s important
If you want to use multiple different versions of a test (for example, to avoid respondents
repeating the same answers from memory), you first need to make sure that all the sets of
questions or measurements give reliable results.
Bron: Kaplan & Saccuzzo
History and theory of reliability
● Basics of test score theory: classical test theory
○ Assumes that each person has a true score that would be obtained if there were
no errors in measurement
■ But: measurement instruments are imperfect
○ Difference between true score and observed score → determined by
measurement error
○ We are really interested in measurement error
○ X-T=E
● Errors → random!
○ Less likely to force investigator to make wrong conclusions
, ● Rubber yardstick analogy
○ Conceptualization of error, analogous to the idea that a phenomenon can be over
or underestimated.
■ E.g. measuring a table
● With wooden stick → always the same measurements
● With rubber stick → stick can shrunk or stretch, so
different measurements
○ We would not get the same score on each measurement
Basic sampling theory: normal distribution
○ 3 distributions in 4.2
■ Left: greatest error
● No dependence on a single observation → far from true
score
■ Right: least error
● Most observations extremely close to true score →
conclusions from less observations likely to produce less
errors
Classical test theory
○ Assumes that the true score for an individual will not change with repeated
applications of the same test
■ Different scores → because of random errors
○ Standard error of measurement
■ Standard deviation of errors as basic measure of error
○ Criticisms
■ Requires that exactly the same test items be administered to each person
● Relatively small number of items concentrate on an individual’s
particular level of ability
Domain sampling model
● Central concept of classical test theory
● Considers the problems created by using limited number of items to represent a larger
and more complicated construct
○ E.g. spelling ability
■ No time to spell every word in dictionary → we choose some
words the person has to spell to determine the spelling ability
best = sample of words
Questions task 2:
- What score do you expect?
Quite high, he seems quite full of himself.
- Does it matter that a different person takes the test?
It can matter yes, because the impression the person gives can differ from the previous person
and can influence the patient.
- What problems arise from this set-up and how can you prevent these?
It can influence the answers the client will give and negatively influence the test results.
- Recall that you yourself filled in the Self Confidence Test and you kept the record of this
at item level. Please, input items responses of all group members at item level into an
SPSS file and calculate Cronbach’s alpha (Analyze>Scale>Reliability Analysis). What is
the Cronbach’s alpha? Is it sufficient?
Parallel forms reliability
Parallel forms reliability measures the correlation between two equivalent versions of a test. You
use it when you have two different assessment tools or sets of questions designed to measure
the same thing.
In educational assessment, it is often necessary to create different versions of tests to ensure
that students don’t have access to the questions in advance. Parallel forms reliability means
that, if the same students take two different versions of a reading comprehension test, they
should get similar results in both tests.
Why it’s important
If you want to use multiple different versions of a test (for example, to avoid respondents
repeating the same answers from memory), you first need to make sure that all the sets of
questions or measurements give reliable results.
Bron: Kaplan & Saccuzzo
History and theory of reliability
● Basics of test score theory: classical test theory
○ Assumes that each person has a true score that would be obtained if there were
no errors in measurement
■ But: measurement instruments are imperfect
○ Difference between true score and observed score → determined by
measurement error
○ We are really interested in measurement error
○ X-T=E
● Errors → random!
○ Less likely to force investigator to make wrong conclusions
, ● Rubber yardstick analogy
○ Conceptualization of error, analogous to the idea that a phenomenon can be over
or underestimated.
■ E.g. measuring a table
● With wooden stick → always the same measurements
● With rubber stick → stick can shrunk or stretch, so
different measurements
○ We would not get the same score on each measurement
Basic sampling theory: normal distribution
○ 3 distributions in 4.2
■ Left: greatest error
● No dependence on a single observation → far from true
score
■ Right: least error
● Most observations extremely close to true score →
conclusions from less observations likely to produce less
errors
Classical test theory
○ Assumes that the true score for an individual will not change with repeated
applications of the same test
■ Different scores → because of random errors
○ Standard error of measurement
■ Standard deviation of errors as basic measure of error
○ Criticisms
■ Requires that exactly the same test items be administered to each person
● Relatively small number of items concentrate on an individual’s
particular level of ability
Domain sampling model
● Central concept of classical test theory
● Considers the problems created by using limited number of items to represent a larger
and more complicated construct
○ E.g. spelling ability
■ No time to spell every word in dictionary → we choose some
words the person has to spell to determine the spelling ability
best = sample of words
