Summary
Experimental Research summary lectures and stat videos
- Course
- Institution
ER lectures and stat videos
[Show more]
1 review
By: cabloemendaal • 2 year ago
Seller
Follow
k12i
Reviews received
Content preview
Experimental ResearchLectures week 1
Intro
When you want people to do something, you can ask them to think about reasons that would
convince themselves. It seems that if people think about reasons to convince others, they will adjust
their own behavior sooner.
4 phases experimental research:
1. This phase deals with the formulation of the problem statement and hypotheses about the
relationship between (an) independent variable(s) and (a) dependent variable(s). We will
learn how to generate a researchable problem statement together with the specific
hypotheses that will be tested in an experiment.
2. This phase deals with the design of an experiment in which you (1) manipulate (an)
independent variable(s) to observe effects on dependent variable(s), and (2) control for
confounding factors. We will discuss (a) how the independent and dependent variables are
operationalized in experiments, and (b) how to control for confounding variables.
3. This phase deals with conducting the experiment and collecting data. We will discuss
different types of experimental designs (e.g., completely randomized designs, single factor
designs, factorial designs, mixed designs, etc.) and understand the differences between main
and interaction effects.
4. This final phase deals with the data analysis and interpretation of the experimental findings.
We will discuss (a) how to statistically analyze experimental designs using ANOVA (Analysis of
Variance) techniques, (b) how to interpret results obtained from an experiment, and (c) how
these results can lead to derive new hypotheses to be tested in a follow-up experiment.
Why do we study experiments? The only exact way to get to causal knowledge -> randomization is
very important. Then you can really see if changes in behavior are because of the treatment (not
because of differences in the sample). This is more and more important in companies. It helps them
learn, grow and prevent mistakes. It can also help with larger business decisions. An experiment is
very accurate. Big data is about past behavior, but if you change something, do you hit the right
spot? An experiment can test this.
Thomke & Manzi (2014), part of the literature: Also show that with larger business decisions,
experiments can really help. Should you open your stores an hour later? You can calculate the costs
of that hour, and compare this to the sales. But, would those people in the early hour not just have
appeared in the store later anyway if it would have been closed that hour?
Kohavi & Thomke (2017), part of the literature: Most business progress happens by small
improvements, not some huge disruptions (p. 77). Opening a new tab when clicking a link on MSN:
+8.9% engagement. Amazon moved credit card offer to shopping cart (from main page):
> $20 million dollar annually.
Different research types
Different research types:
, Descriptive = describes behavior, thoughts or feelings. A common example is public opinion
polls.
o Survey research
o Changes can be measured, if respondents fill out the survey at different points in
time (longitudinal or panel design)
Correlational = investigates the relationship among various psychological variables. Its aim is
to discover correlations between variables. Used to describe the relationship between two
ore more naturally occurring variables. E.g. does weather affect our mood? But it cannot
establish causality! Importantly: we determine whether one variable is related to another by
seeing whether scores on the two variables covary (whether they vary and change together).
Correlation coefficient = indicates the degree to which two variables are related; -1 to +1
Experimental = its aim is to find whether certain variables cause changes in behavior,
thought or emotions. It involves manipulating an IV and assessing potential changes in an DV.
The key is randomization of subjects to treatments. If behavioral change occurs, we can infer
that X causes Y.
Quasi-experimental = in some cases it might not be possible to manipulate the IV, so then
this method is used. Example = safety belt in car. Use control condition.
3 requirements causality:
Correlation is only one of the necessary conditions for causality
Directionality (logical in time)
Elimination of extraneous variables
Descriptive or correlational?
For the descriptive type of data, we discussed this example: 40% of site visitors who do not convert
to customers, indicate that the shipping costs were too high. for males this was 30%, for females this
is 50%. But if correlational data is the relation between two variables, is the gender -> free shipping
preference not a correlational datapoint (instead of a descriptive one)?
Randomization = Arbitrarily assigning each participant to one condition of the experiment.
If I assign person A to the control group, and assign person B to a treatment group,
person A and B are not the same
However, if I do this for every subject, the average person in the control group is the same as
the average person in the treatment group
With large samples, true randomization creates balance in for example age, gender,
preference, etc. of participants. The groups are thus the same on average
Any difference we thus later find must have occurred because of the treatment
As the groups are the same on average on all aspects (again, if your sample is large enough),
it is better than for example matching techniques that economists / marketing modelers tend
to use
Why? Because for matching we have to a priori predict the possible confounds (alternative
explanations), and match people based on those confounds. However, there could well be
confounds that we do not yet know about.
, Example of why randomization is important based on my prior experience in this class:
Students in a prior year of this class worked on the group assignment, and wanted to
compare whether people wanted to play in a lottery with a 30 million prize, compared to a
29.7 million prize. Every subject should be randomly assigned to one of these conditions. In
two(!) student groups, students split the study in two separate questionnaires (because they
felt this was easier to program). Some students in the group shared the 30-version with their
friends, other students shared the 29.7-version with their friends. The allocation to
treatments is no longer random! (my friends will be different from your friends, which might
also affect the results)
Goals of science: finding regularities/patterns and predict
outcomes.
Defines constructs
Conceptual definition: Consumer dissatisfaction may be
defined as: "... cognitive state of being inadequately rewarded in a buying situation for the
sacrifice he (i.e., the buyer) has undergone
Operational definition (e.g.,): “I am dissatisfied with the service being provided”
Basic experimental statistics
The base:
If we observe a difference on a measure between two groups, what makes us more likely to believe
that there actually is a difference?
A larger difference between the group
Less variation between individuals in each group
A larger sample size
These latter two help to determine the precision of the estimate
There is a mean, but there is always a variation (uncertainty around it). Statistical test: test whether
means are
different.
Mean: average of everyone we tested in each condition
Standard deviation is a measure of variability in the data: do people’s individual responses
differ a lot from the mean, or only a little bit?
We need both the mean and a measure of variance to test how likely it is that a difference is due to
error.
Formula t-test:
In other words, all else being equal:
A larger difference between means gives a larger t-value
A smaller variance gives a larger t-value
A larger sample size gives a larger t-value
, The t-value will lead to a p-value (the chance that we find this data, if there would have been no
effect in real life). But having an estimate how likely it is that the effect really exists is only one thing
we actually like to now. The other this is the effect size. How large is the effect? The problem of only
looking at significance is that with a large sample size, (practically) everything becomes significant.
Effect size in t-test: For the effect size, we want to know how much
of the variation in the data we can explain. What is variation?
Standard deviation*. How much can we explain? Well, the
difference between the means. This effect size for t-tests is called
Cohen’s d.
Compare the t-test formula to Cohen’s d = the main difference is
that the t-test sample size is in the formula: a larger sample makes t larger (and p smaller). But a
larger sample does not make Cohen’s d larger: the effect size is independent of sample size (how
large the difference is, should not be influenced by how many people you asked).
Note: this is the formula (Cohen’s d) when the cell size is equal: both conditions have an equal
number of participants, so weighting is not needed.
Cohen’s d in SPSS -> SPSS does not report this. You can calculate it with the formula (by yourself), use
an online resource, or use another program (R or JAMOVI).
Stat videos week 1
Video 1: Basics of why we test
Example:
Does taking pictures with the goal to share them with others reduce enjoyment of the moment? IV:
instructed to make some Pictor for self or others. DV: how much did you enjoy jour photo-taking
experience? Outcomes: people who made the pictures for themselves enjoyed it more than people
who made it for others/sharing.
The mean enjoyment for people who made the pictures for themselves is higher. But even though
this is the case, you have to test it. There is variation. We can never be sure about a difference by
only looking at 2 means.
What is the p-value?
The p-value is the chance that you find your data (or a more extreme form) if you assume the null-
hypothesis is true. If there is no effect, this data is quite unlikely. So, if you assume that there is no
effect, how likely is it that you find this data. If you see a huge difference in your data, it becomes
very unlikely that there is no effect. In other words: If there is not effect, this data is quite unlikely.
Don’t ever round p-values but give precise values.
What does marginally significant mean? It simply means that the data is a bit less surprising if you
assume the null-hypothesis is true.
Een p-waarde, die staat voor kans, is een statistische maat tussen 0 en 1. In klinische onderzoeken
wordt de p-waarde gebruikt om aan te duiden of een waargenomen resultaat het gevolg kan zijn van
toeval of niet. Strikt genomen is de p-waarde een maat voor de kans dat de nulhypothese ten
onrechte is verworpen (en het gevonden verschil tussen onderzoeksgroepen dus in werkelijkheid op
The benefits of buying summaries with Stuvia:
Guaranteed quality through customer reviews
Stuvia customers have reviewed more than 700,000 summaries. This how you know that you are buying the best documents.
Quick and easy check-out
You can quickly pay through credit card or Stuvia-credit for the summaries. There is no membership needed.
Focus on what matters
Your fellow students write the study notes themselves, which is why the documents are always reliable and up-to-date. This ensures you quickly get to the core!
Frequently asked questions
What do I get when I buy this document?
You get a PDF, available immediately after your purchase. The purchased document is accessible anytime, anywhere and indefinitely through your profile.
Satisfaction guarantee: how does it work?
Our satisfaction guarantee ensures that you always find a study document that suits you well. You fill out a form, and our customer service team takes care of the rest.
Who am I buying these notes from?
Stuvia is a marketplace, so you are not buying this document from us, but from seller k12i. Stuvia facilitates payment to the seller.
Will I be stuck with a subscription?
No, you only buy these notes for $8.04. You're not tied to anything after your purchase.
Can Stuvia be trusted?
4.6 stars on Google & Trustpilot (+1000 reviews)
67474 documents were sold in the last 30 days
Founded in 2010, the go-to place to buy study notes for 14 years now