Summary
Summary methods: experiments and surveys
- Course
- Institution
This is a summary of the course methods: experiments and surveys. I incorporated my notes in this survey to make the slides more clear.
[Show more]
Seller
Follow
britt1910
Reviews received
Content preview
Methods: Experiments and surveysLecture 1: introduction
Models are used by the central bank, etc., to get to policy advice. Within these models, we
need input like consumer and investor preferences and expectations. We can gather this kind
of data by doing experiments and surveys. This course is about data collection.
Pools of participants
- Sample of university students: a lot of internal validity
- Amazon Mechanical Turk: platform to recruit participants. Loss of internal validity.
Bots may respond, etc.
- Representative survey panel (LISS panel of Tilburg university): a lot of external
validity but internal validity is a bit lost (we do not have control over what
participants do in the experiment; they might communicate with others).
- Market research companies: more complexity possible, but often expensive
- Professionals: bring special expertise, not so much external validity
- Customers and clients: a lot of external validity
- Special groups: children/ elderly
All these groups have different costs and benefits and provide different implications for
internal and external validity (a lot of internal validity in laboratory experiments, but a lot of
external validity in field experiments).
Tools
- Pen and paper questionnaire
o Various degrees of complexity
o Interactive decisions are difficult
- Computerized experiment or survey (for example, veconlab)
o Programming toolboxes z-Tree, o-Tree.
Z-Tree = Zurich Toolbox enables unexperienced programmers to
quickly create experiments.
O-Tree = more complicated and requires some knowledge of
webserver infrastructure for the initial setup.
o A lot of complexity is feasible
- Simple online survey tools (for example, Qualtrics, SurveyMonkey, etc.).
o Simple tasks or questions
o No interaction possible
Lecture 2: survey structure, response modes and layout
Survey structure
Before conducting a survey, you need to think about how many questions you want to ask,
which of them are most important (put these first as participants lose concentration during the
task), what kind of topics you want to discuss, if there would be order effects and if this is the
case, you need to randomize (block or page randomization).
,Randomization = switch agree and disagree button sometimes. This makes the survey a bit
more difficult for people, but it rules out the left-right response bias.
We also need to think about how many questions in total we will ask and how many questions
we put on a page. If you have a coherent structure, you can improve the comprehension by
putting together questions with a similar topic.
Cross infection and suggestive questions can cause problems if they are on the same page.
In general, we need to minimize the number of questions that we ask, and we need to make
use of conditioning (that is, only ask question 5 if they answered Yes in question 4).
We also need to consider the possibility of revising answers. Of course, we want to know
people’s true views, so we should give them any possibility to reveal it. However, there might
be learning by answering later questions, in case revision leads to less ‘raw’ answers.
If we would have complex programming with conditioning, revisions might become
problematic. Also, if we have chained questions, later paid tasks depend on earlier answers
and the possibility of revision may lead to strategically wrong answers. I will explain the last
point by use of an example:
This person values the iPhone at 900$. The person knows that one out of the many steps is
actually paid. Therefore, he is incentive compatible: he chooses the option that is worth more
to him truthfully.
Using titration/bisection, we can zoom into his valuation. We just add more steps to get to
know his true valuation better.
The problem with titration is that people can answer untruthfully to get better/higher offers
afterwards. This becomes especially problematic when participants can go back and forth in
the survey and find out that this is indeed the case. The expected payoff over the 3 rounds is
higher if you do as if you value the iPhone at 1100$ instead of 900$. Therefore, the person
might want to misreport.
However, chained questions with real payments are still highly efficient in experimental
research. It is risky to give participants the possibility to revise their answer. There is also a
high risk that participants learn the mechanism if this design is not applied in a large
anonymous study. People can communicate with each other about the experiment in this case
, and share the procedure and payments, and therefore, get to know strategies that might give
them better payments. It is therefore important to disguise the method (people should not
know what is going on).
Introduction to a survey
- Don’t make it too long.
- There is a trade-off between giving a lot of details and respondents limited attentional
resources.
- Detailed instructions are only good if you can ensure that people will read and
understand them (like in an experiment, or one-on-one interview).
- We can introduce control questions that need to be answered correctly before
participants can proceed (can be annoying) or we can use thin slicing with
instructions. This means that we give general instructions and them later we make
group-of-items specific instructions.
Response modes
- Slider bars
o Intuitive visual response format
o Sometimes it is better if you do not indicate a starting point as this could
influence the decision (minimize framing bias)
o Often without step sizes indicated
- Open entry boxes
o They do not suggest potential answers and illicit the research question with
that
o They do not constrain answer by respondents
o However, we lose observations when people put unclear or confusing answers
o It may be time consuming, and the coding may become arbitrary (some people
might fill in male, other men, other people Mr., etc.)
o What answers to delete and what answers not is up to you. In general, you
have to make sure that the decision can be justified (based on research or well-
established procedures).
- Multiple choice
o Tight control over answers and structure
o However, asking about income may be problematic. Using ranges is often
more likely to be answered truthfully.
o If you force an answer, people might just drop out (attrition).
o Often you want to add the choice like “don’t want to say” or “don’t know”
- Multiple choice – Likert scale
o This is a subjective scale which is often used in psychometric measurements
o We assume linearity (going from 1 to 2 is just as big as going from 3 to 4)
o However, some people might not want to choose 1 or 5 ever, so the answers
may be biased in the middle.
- Multiple option check boxes
o Tight control, but it can suggest something
o There could be order effect, which necessitates us to randomize items
o If you limit how many boxes they can check, you force them to really select
their highest interests
o Create bottom where they can put their own insight
- Distributing weights
The benefits of buying summaries with Stuvia:
Guaranteed quality through customer reviews
Stuvia customers have reviewed more than 700,000 summaries. This how you know that you are buying the best documents.
Quick and easy check-out
You can quickly pay through credit card or Stuvia-credit for the summaries. There is no membership needed.
Focus on what matters
Your fellow students write the study notes themselves, which is why the documents are always reliable and up-to-date. This ensures you quickly get to the core!
Frequently asked questions
What do I get when I buy this document?
You get a PDF, available immediately after your purchase. The purchased document is accessible anytime, anywhere and indefinitely through your profile.
Satisfaction guarantee: how does it work?
Our satisfaction guarantee ensures that you always find a study document that suits you well. You fill out a form, and our customer service team takes care of the rest.
Who am I buying these notes from?
Stuvia is a marketplace, so you are not buying this document from us, but from seller britt1910. Stuvia facilitates payment to the seller.
Will I be stuck with a subscription?
No, you only buy these notes for $4.81. You're not tied to anything after your purchase.
Can Stuvia be trusted?
4.6 stars on Google & Trustpilot (+1000 reviews)
52510 documents were sold in the last 30 days
Founded in 2010, the go-to place to buy study notes for 14 years now