Construction and Analysis of
Questionnaires
In this course you will learn how to:
1. operationalize: Converting non-directly observable traits, beliefs, or behaviors into concrete questions.
2. assess dimensionality: Grouping questions into meaningful scales
3. evaluate validity and reliability: Investigating the (technical) quality of the scale scores
11 lectures + 1 Q&A lecture 6 tutorials 2 SPSS labs (practice for SPSS test)
, L1: Introduction
Why are questionnaires important?
I. Questionnaires are used by:
• Governments: To measure national economic well being, to make policy decisions
• Organizations: To investigate the difference in salaries b/n women and men, work safety
• Companies: To select candidates, to test “usability” of a product, to measure satisfaction, for marketing,
evaluation
• Social Science Researchers: to predict voting, political attitudes, measure life quality, personality, behavior
II. Definition of a questionnaire
Merriam Webster: “A questionnaire is a written set of questions that are given to people in order to collect
facts or opinions about something” (typical performance)
There are two types of performance measured:
• Typical Performance: What people will do? Questions without a specific context, the individual is NOT AWARE
of evaluation. There is no right or wrong answer! Determined mostly by personality! - QUESTIONNAIRE
• Maximum Performance: What people can do? The individual is motivated to demonstrate their best (e.g.,
achievement test, intelligence test, task-related knowledge, skills). Often only right-wrong answers! - TEST
|Survey - research method for collecting information characterized by: 1) structured or systematic set of data
2) investigating what may be responsible for a particular phenomenon • It is possible to collect information using a
questionnaire, by interviewing (qualitative surveys) or observing So: survey ≠ questionnaire ≠ test| (not exam)
MORE about Questionnaires: • Item = statement/question + response alternatives
• 1 measurement instrument/ questionnaire tries to isolate 1 characteristic and quantify to what extent
someone “has that”. Thus, it gives an incomplete description of reality on purpose.
Questionnaires often consist of different scales => multiple “scales” - multiple items within scale, so multiple
scales with multiple items in it
E.g., questionnaire to measure employee satisfaction:
Scale 1: Measures work atmosphere (items 1-20); Scale 2: Measures satisfaction with work tasks (item 21-40);
Why? In the end, we are interested in answering research questions and testing hypotheses.
Example: H: Are more intelligent people more often depressed? How can we measure depression, how
intelligence? → Constructs that we cannot directly observe have to be operationalized and made
measurable. Observable constructs don’t have much measurement error, but those that cannot be directly
measured have.
III. Measurement in social sciences
Questionnaire => An approximation of something that cannot be 100% precisely measured.
• Therefore, our questionnaires must be of high quality: → reliable and valid
Reliability: Reliability is about the precision and repeatability of the measurement!
→Reliability refers to how consistently/ how precisely you are measuring something.
→If the same result can be consistently achieved by using the same methods, the measurement is reliable.
Validity: Validity of a measure tells you how accurate the measure is! Is what we measure really what we want to
measure? Do the differences in scores resulting from the questionnaire reflect differences in the construct ?
Questionnaires - Content of questionnaires: Behaviour, Beliefs, Opinions, Values, Attitudes, Demographics
Some features of questionnaires:
• Mode of questionnaire administration:
- Interactional mode: With interviewer or self-administered
- Technological mode: phone, computer, internet, paper-and-pencil - CAPI, PAPI, CATI, WAPI, CASI, SAPI
- Mix of administration modes
• Question type: Open-ended versus closed-ended questions
Pros: Cons:
+ Very fast and efficient way to collect data - Low response rates (especially for online surveys)
+ Cheap (especially if purchasing is done digitally) - Answer to questions may deviate from actual
+ Suitable for research in large samples facts/behavior (e.g., due to social desirability
1
, Lecture 2: Making Constructs Measurable: How?
3 theoretical questions from the 1st and 2nd lecture will be on the exam
• Constructs: unobservable concepts • Latent variables (psychological traits) often referred to as latent traits
I. Constructs in social science research - social science research is based on the relations b/n
constructs
• Constructs are mental representations that are typically based on experience; abstract summaries of a number
of characteristics, behaviors, and attitudes that share something in common.
• Problem: Constructs cannot be measured -> Solution: 1) Operationalize it 2) Questionnaire-based indirect measurement
II. Questionnaire research
• This course is about: latent constructs -> as a result: about scale construction→using questionnaires
Option 1: Use an existing validated questionnaire Option 2: Create your own set of questionnaire items
III. How can we make a construct measurable?
Example – social capital:
1. Develop our nominal definition: “Social capital consists of networks of social relations …” (Stone, 2001).
2. Define the subdimensions: • Structure of social relationships - networks • Quality of social relationships – norms
3. Developing indicators: illustration for
social capital by a book …
4. Further subdimensions of networks and
norms: Networks:• Type • Size and
capacity • Norms: • Trust • Reciprocity •
This can be further subdivided
Construct <- Dimensions <- Subdimensions <-
Further Subdimensions
We are working on this... until we have
something measurable → operational
definition!
The best course of action is to follow a systematic process. Important when constructing a scale: Reliability and Validity.
IV. Scale constructions: know all of them for the exam\
1) Construct method - Deductive in nature: Starts from a theoretical framework. Items are derived
from a theoretical definition of the construct!
- Specification of the nomological network: The construct is studied in detail, a precise definition is
provided incl all subdimensions and related and unrelated constructs are defined.
- A homogeneous set of items covering all aspects of the construct is produced.
- For scale construction much attention is paid to convergent (related) and divergent (other ones are
related) validity.
2) Facet method - Deductive in nature: Starts from a conceptual analysis (4 steps). Items are derived
from a mapping sentence. Scale construction based on dimensionality analysis.
1. Identifying the behaviors that are essential to the construct
2. Essential aspects are specified called facets (Qs about facets in the exam)
a. Facets related to the content of the item: situational facets, behavioral facets
b. Facet related to the answer alternatives of the item
3. For each facet a number of facet element is defined called structs
4. The facets are completely cross-classified each combination of unique structs is called a structuple
Example: Anxiety associated with visiting the dentist →Combinations of elements: 2 structs x 2 structs x 3 structs
= 12 structuples, for each element, an item needs to be constructed!
2
, 3) Rational method - Deductive in nature: Expert knowledge is used to define the construct and to
assess the items; rationality of the considerations of experts. Items are based on informal criteria
(working definition)! Often used, for example in clinical settings (diagnostic comparison).
• Scale construction based on high face validity:
• Good face validity means that anyone who reviews your measure says that it seems to be measuring
what it’s supposed to.• For example, statements like ‘I am depressed’, ’I enjoy the daily tasks of my work
a lot’.
• We want to measure participants’ age. What is the best way to ask this in order to achieve high face
validity?
4) Prototypical method - Inductive method: Atheoretical in nature, the method does not analyze the
content of a construct. Both the rational and the prototypical method use personal evaluations.
• Also known as “act frequency approach” (personality/disposition→constructs are represented by sets
of behavior (called acts); some acts are considered to be more prototypical of the construct)
• The prototypical method is more systematic and extensive, using standardized evaluations and a
large sample of judges. Main goal is to represent the most (proto)typical acts of the construct.
• Item production: Act nomination by respondents (often non-experts but can also be experts): “Think of
persons with extreme positions on the construct to be operationalized, and to write down behaviors that
exemplify this construct”.
• Selection of items, scale construction: Experts rate the extent to which an item “fits” the construct.
Example: Dominance (Buss & Craik, 1983)
• Act nomination: Question to undergrads
“Think of the three most dominant (fe)males’
acts..“
• Based on act nomination, a list of 100 acts
was derived.
• Scale construction based on prototypicality
ratings: Judges rate on an x-point scale (e.g., 7-
point scale) how good the act represents the
disposition
.
5) Internal method - Also known as
internal consistency method. Inductive method: Atheoretical in nature, the method does not analyze the
content of a construct.
• A very large set of items is created using any method and combinations of methods (e.g., ask
experts→rational method, then put together a number of existing scales related to your construct of
interest). Items are relevant for the content domain!
• Item production: homogeneous. The meaning of a given scale is derived from the content of its
constituting items! Items that fail to show homogeneity are typically removed. • Scale construction based
on homogeneity analysis • Item selection: using reliability and validity assessment methods!
Example: Big Five of Personality using NEO-PI-R (Costa + McCrae): Neuroticism, Extraversion, Openness,
Conscientiousness & Agreeableness. The questionnaire was developed by combining several existing
approaches!
6) External method - Inductive method: Atheoretical in nature, the method doesn’tt analyze the content
of a construct.
• External criterion: Usually a variable that is theoretically or practically relevant and empirically related to
the questionnaire (e.g., behavioral measure, clinical status, ...).
• Item production: heterogeneous. The item set typically touches on many different aspects of the
construct.
• Maximizing “prediction” (e.g., job interview at google)
• Selection of items/scale construction: optimize relationship with an external criterion, i.e., item-criterion
relation. This is maximizing the criterion-oriented validity!
3
