Survey Workshop Lectures, Dr. Marcel van Egmond
Chapter 2: Types of Error in Surveys
• design and data collection a ect potential error
Error associated with who answers
• error because sample does not completely mirror population -> sampling error
• random error
• bias, when population is not represented properly
Error associated with answers
• how well answers correspond to the „truth“
• errors can be caused by misunderstanding questions, not having the information, distorting
answers to look better, etc.
• validity
• people tend to over-and underreport depending on the topic of the question
Chapter 6: Designing Questions to be Good Measures
• increasing reliability of answers: ensuring consistent measurement by providing the same set
of questions to every respondent
• good question:
a. question-answer process is entirely scripted
b. question means the same to every respondent (no space for misinterpretations)
c. kinds of answers that constitute an appropriate response to the question are
communicated consistently to all respondents
• no inadequate and incomplete wording
• open questions:
• can provide unexpected answers
• may describe view of respondent better
• respondents can answer in own words
• closed questions
• respondent can answer more reliable
• answers are more likely to be analytically interesting
• easier to record for researchers
Chapter 7: Evaluating Survey Questions and Instruments
De ning Objectives
• which variables are dependent
• which variables are independent
• which variables are needed as control or intervening variables to explain patterns and to
check hypothesis
Preliminary Question Design Steps
Focus Groups
• conduct focused discussions with people who are in your study population
• e.g. when goal is to measure number of visits to doctor, group discussion focuses on
what counts as visit to doctor
Drafting Questions
• background questions are standard to many surveys
1
, • copies of original survey instruments from major survey organization often useful to
draft own questions
• all questions should be tested to make sure they work
Pre-survey Evaluation
Critical Systematic Review
• when set of questions is drafted
Cognitive Laboratory Interviews
• can people consistently understand and answer the questions?
• early versions often contain confusing, cannot be read as written or are virtually
unanswerable by anyone
• respondents should evaluate survey
Design, Format and Layout of Survey Instrument
• order the questions: rst easy, straight-forwarded questions (help to get respondent into
survey)
• help respondents to handle common navigational challenges (how to correct previous answer)
Lecture 1, Wed. 9.01.2019, Valid Answers? Proper Questions!
1. Errors in Surveys: What could possibly go wrong?
a. depending on chosen sample, sample must be representative for population
b. sample can be biased
c. we don’t want random error (although overall we can expect good representation of
opinion) nor bias (systematic bias can give you insight too, either under- or overestimating)
d. not accurate answers because more desirable
Random Biased (systematic)
Sample: sampling error Sample bias
• it happens • First row students?
Questions: invalidity Question: bias
• „honest mistakes“ • Did you read the literature for today?
• problem if you want to look at individuals (still • massive overreport of reading the literature,
valid for population as whole) because people give more desirable answer
(because lecturer asked, if fellow students would
ask you would get a more honest answer)
2. Asking Questions
a. double-barreled
b. leading/biased
c. double-negative
Simple questions:
• „Is your systolic pressure normal at this moment?“
• Do respondents have the requisite knowledge? Questions should be understandable for
everyone. Answers can be easy for you, but di cult for others
• just give as many answer options as you really need/are relevant for you
• don’t make things complicated
2
Chapter 2: Types of Error in Surveys
• design and data collection a ect potential error
Error associated with who answers
• error because sample does not completely mirror population -> sampling error
• random error
• bias, when population is not represented properly
Error associated with answers
• how well answers correspond to the „truth“
• errors can be caused by misunderstanding questions, not having the information, distorting
answers to look better, etc.
• validity
• people tend to over-and underreport depending on the topic of the question
Chapter 6: Designing Questions to be Good Measures
• increasing reliability of answers: ensuring consistent measurement by providing the same set
of questions to every respondent
• good question:
a. question-answer process is entirely scripted
b. question means the same to every respondent (no space for misinterpretations)
c. kinds of answers that constitute an appropriate response to the question are
communicated consistently to all respondents
• no inadequate and incomplete wording
• open questions:
• can provide unexpected answers
• may describe view of respondent better
• respondents can answer in own words
• closed questions
• respondent can answer more reliable
• answers are more likely to be analytically interesting
• easier to record for researchers
Chapter 7: Evaluating Survey Questions and Instruments
De ning Objectives
• which variables are dependent
• which variables are independent
• which variables are needed as control or intervening variables to explain patterns and to
check hypothesis
Preliminary Question Design Steps
Focus Groups
• conduct focused discussions with people who are in your study population
• e.g. when goal is to measure number of visits to doctor, group discussion focuses on
what counts as visit to doctor
Drafting Questions
• background questions are standard to many surveys
1
, • copies of original survey instruments from major survey organization often useful to
draft own questions
• all questions should be tested to make sure they work
Pre-survey Evaluation
Critical Systematic Review
• when set of questions is drafted
Cognitive Laboratory Interviews
• can people consistently understand and answer the questions?
• early versions often contain confusing, cannot be read as written or are virtually
unanswerable by anyone
• respondents should evaluate survey
Design, Format and Layout of Survey Instrument
• order the questions: rst easy, straight-forwarded questions (help to get respondent into
survey)
• help respondents to handle common navigational challenges (how to correct previous answer)
Lecture 1, Wed. 9.01.2019, Valid Answers? Proper Questions!
1. Errors in Surveys: What could possibly go wrong?
a. depending on chosen sample, sample must be representative for population
b. sample can be biased
c. we don’t want random error (although overall we can expect good representation of
opinion) nor bias (systematic bias can give you insight too, either under- or overestimating)
d. not accurate answers because more desirable
Random Biased (systematic)
Sample: sampling error Sample bias
• it happens • First row students?
Questions: invalidity Question: bias
• „honest mistakes“ • Did you read the literature for today?
• problem if you want to look at individuals (still • massive overreport of reading the literature,
valid for population as whole) because people give more desirable answer
(because lecturer asked, if fellow students would
ask you would get a more honest answer)
2. Asking Questions
a. double-barreled
b. leading/biased
c. double-negative
Simple questions:
• „Is your systolic pressure normal at this moment?“
• Do respondents have the requisite knowledge? Questions should be understandable for
everyone. Answers can be easy for you, but di cult for others
• just give as many answer options as you really need/are relevant for you
• don’t make things complicated
2
