Experimental Research
LECTURE 1: Introduction
First experiment done in marketing
- Told organ meat is good for you
- Think of reasons organ meat is good for you yourself (co-creation)
→ more likely to adopt if you have to come up with reasons yourself
Experimental research consist of 3 phases:
- Phase I: This phase deals with the formulation of the problem statement and hypotheses about the
relationship between (an) independent variable(s) and (a) dependent variable(s). We will learn how
to generate a researchable problem statement together with the specific hypotheses that will be tested
in an experiment.
- Phase II: This phase deals with the design of an experiment in which you (1) manipulate (an)
independent variable(s) to observe effects on dependent variable(s), and (2) control for confounding
factors. We will discuss (a) how the independent and dependent variables are operationalized in
experiments, and (b) how to control for confounding variables. We also discuss differences between
“real” experiments and quasi-experiments.
- Phase III: This final phase deals with the data analysis and interpretation of the experimental
findings. We will discuss (a) how to statistically analyze experimental designs using ANOVA
(Analysis of Variance) techniques, (b) how to interpret results obtained from an experiment, and (c)
how these results can lead to derive new hypotheses to be tested in a follow-up experiment.
Course content
Theoretical perspective with mainly be tested in exam
Practical perspective more for assignments: can use both R and SPSS
Will not get computer EXAM but need to know how to analyse it say which test, use mediation which
is the IV, but do not have to do stats yourselves) (will never ask syntax questions on exam are only
open ended questions)
- Articles
- Videos
- Lecture slides
Final grade : 30% assignment, 70% exam (min. 5.0 to pass)
Assignment A: 1/3 , B 2/3
Assignment A: Design an A/B test
Phase I: develop a research question
Phase II: Develop an appropriate research design
- Example booking.com: show different versions of the website to test which one works best we
will design new version of web page and make proposal how to test it. E.g., increase sign up rate:
try to think what customer would do when entering this page; how could we nudge them/what
hinders them. Design the new version.
Assignment B1 & B2: Create (conduct), and Analyse a study
Phase 1: develop a research question
Phase 2: Develop an appropriate research design
Phase 3: Data analysis and interpretation
(do not have to collect data ourselves hew will do it via site and we have to analyse the data)
1
,So why study experiments?
1) Best way to get causal knowledge
- Econometric/modelling techniques can always try to control for variables and get as close as
possible to causality. But in the end, only true randomization can really do this most convincingly.
- Gordon et al. find that various (correlational) methods of trying to get to causal effects of
advertising effectiveness are inaccurate.
- A true experiment is needed to get accurate data of your advertisement.
(we’ll discuss this paper later)
Reason:
• 1.a) more and more companies seem to realize point 1
• 1.b) this helps them learn, grow, and prevent mistakes
Papers:
Thomke & manzi (2014)
Also show that with larger business decisions, experiments can really help.
Should you open your stores an hour later? You can calculate the costs of that hour and compare this
to the sales in that hour. But, would those people in the early hour not just have appeared in the store
later anyway if it would have been closed that hour?
Kohavi & thomke (2017)
Most business progress happens by small improvements, not some huge disruptions (p.77)
- E.g., Opening a new tab when clicking on a link : +8.9% engagement (because if you read that
tab you could close it and back on MSN again)
- E.g., amazon moved credit card offer to shopping cart (from main page): >$20 million dollar
annually
Experiments help guide investment choices (also because you get more accurate effect size estimates
than if you would use modelling approaches)
Big data is past behaviour. You make an inference on what you think happened. But if you change
somethings, do you hit the right spot? Will it change something?
→ an experiment does exactly this. It is forward looking, not backward looking.
And if you change something, you can often think of 10 different new ways to do so, but eventually
have to pick one. Every idea is uncertain, so how do you know which is best?
→ An experiment can test which idea is best as well
Why do we do research?
Behavioural research
Goal Describing behaviour Predicting behaviour Explaining behaviour
Types Descriptive Correlational Experimental (&Quasi-
Experimental)
Descriptive
- Describes behaviour, thoughts or feelings
- E.g., public opinion poll/survey
- Survey research
- Changes can be measured, if respondents fill out the survey at different points in time
(longitudinal or panel design)
2
, Examples:
- site visitors indicate that the shipping costs were too high
- for males this was 30%, for females this is 50%
Causality
But most often, we want to know whether one thing causes another:
Will our sales increase if we launch this new product?
- Do we sell more when it is warm or cold outside? (we need to know for our planning of staff)
- Etc.
Most naïve solution is regression (correlation)
Correlation
- Investigates the relationships among variables
- Its aim is to discover correlations between variables
- Used to describe the relationship between two or more naturally occurring variables
Example
- Does The Weather Affect our Mood?
We can run a correlation between temperature and mood. I pick temperatures today in different places
in Europe, and measure the mood of people in those places.
Temperature Mood
Possible in this case, because theoretically temperature must be the independent variable (as mood
cannot change actual temperature).
Downside correlation: does NOT prove causality → can also be the opposite direction
- However, people might take more days off when temperature is high (however still indirect effect
of temperature)
- Maybe in south the food is better so might have an effect on mood
- Temperature is also often related to chance of rain, so maybe that is what creates a good or bad
mood
→ so even if there is strong correlation does not necessarily mean temperature is driver of mood
Move from simple correlation to multiple regression and add humidity
Temperature Mood
Humudity
But, there might be other factors that we did not yet measure. Humidity and temperature are related, and
maybe humidity is what drives the effect?
Slightly less naïve method: multiple regression
But it is clear that we are still missing many likely other factors that might play a role (e.g., the food is
better in the warmer southern bits of Europe, which can make people happier)
Another further improvement on multiple regression → time series analysis (granger-
causality/predictive causality)
3
