Summary
Summary chapter 3
- Course
- Institution
Summary of 5 pages for the course applied multivariate data analysis at EUR (chapter 3 notes)
[Show more]
Seller
Follow
oliviaward
Content preview
Chapter 3: The Phoneix of StatisticsMisconceptions about statistically significant results
- Misconception 1: A significant result means that the effect is important
- Very small and unimportant effects can be statistically significant with large
amounts of data while important and large effects may not be stat sig with small
amounts of data
- Misconception 2: A non-significant result means that the null hypothesis is true
- Non-sig result allows us to reject the alternative hypothesis but doesn’t tell us
that the effect size is 0
- Misconception 3: A significant result means that the null hypothesis is false
- Null hypothesis represents a statement of probability such as ‘if the null
hypothesis is correct, then this test statistic is highly unlikely’
Why does the NHST fail?
- It tries to test which of the null and alternative hypotheses are correct, but the
significance of the test provides no evidence about either hypothesis
- It encourages all or nothing thinking where if p<.05 then an effect is significant but if
p>.05 it is not
- NHST is influenced by the intentions of the scientist
- Empirical probability (long-run probability) is the proportion of events that have
the outcome in which you’re interested in an indefinitely large collective of events
- NHST is based on long-run probabilities with the alpha set at .05 probability of a
type 1 error so that across repeated identical experiments the probability of
making a type 1 error is 0.05; type 2 error is also a long run probaility
- For both the probability is not what is probable for every individual but for
what happens in the long run
- The p-value is the probability of getting a test statistic at least as large as the one
observed relative to all possible values of t(null) from an infinite number of
identical replications of the experiment
- Two different intentions for collecting data: collectin a certain number of data
points and collecting data up until a certain time point
- These influence how the p-values
Publication Bias
- Current incentive structure in science in invidiaulistic rather than collective where
individuals are rewarded for publishing and getting funding
- Publication bias: significant findings are about 7x more likely to be published
- Reviewers and editors tend to reject non-significnat results
- Researchers may selectively report their results to focus on sig findings
- Researcher degrees of freedom: researcher has many decisions to make when
designing and analysisng and they can be misused to exclude cases that make the
results non-signfincat
P-hacking and harking
, - P-hacking: researcher degrees of freedoms that lead to the selective reporting of
significant p-values
- Harking: hypothesizing after the results are known
- Both p-hacking and harking you are not controlling the type 1 error rate
- P-curve: number of p-values you would expect to get for each value of p
- Smaller p-values are more frequently reported than larger non-significant ones
EMBERS
- Effect Sizes
- Meta-analysis
- Bayesian Estimation
- Registration
- Sense
Sense
- 6 principles for using NHST
- You can use the value of p (not whether it is above or below an arbitrary
threshold) to indicate how incompatible the data are with the null hypothesis
- Smaller ps indicate more incompatibility with the null
- Don’t interpret p-values as probability that the hypothesis is true and not that data
was by random chance alone
- Decisions should not be based on whether p-value passes a scientific threshold
- Don’t p-hack: be transparent about everything
- Don’t confuse stat sign with practical importance (p-value doesn’t measure size
of the effect)
- Alone p-value doesnt provide a good measure of evidence regarding model or
hypothesis
Preregister
- Pre-registration: practice of making all aspects of your research process publicly
available before data collection
- Open science: movement to make the process, data and outcomes of research freely
available to everyone
Effect sizes:
- Problem with NHST is doesn’t tell us the importance of an effect
- Effect size: objective and standardized measure of the magnitude of observed effect
- Not affected by sample size (unlike p-values) , which makes it less misleading
than relying on p-values
- Common measures of effect size: cohen’s d, pearson’s correlation coefficient r, and
odds ratio
- When group sizes vary better to use cohen’s d because r can be biased
- Cohen’s d:
- Large effect: .8, medium .5 and small .2
The benefits of buying summaries with Stuvia:
Guaranteed quality through customer reviews
Stuvia customers have reviewed more than 700,000 summaries. This how you know that you are buying the best documents.
Quick and easy check-out
You can quickly pay through credit card or Stuvia-credit for the summaries. There is no membership needed.
Focus on what matters
Your fellow students write the study notes themselves, which is why the documents are always reliable and up-to-date. This ensures you quickly get to the core!
Frequently asked questions
What do I get when I buy this document?
You get a PDF, available immediately after your purchase. The purchased document is accessible anytime, anywhere and indefinitely through your profile.
Satisfaction guarantee: how does it work?
Our satisfaction guarantee ensures that you always find a study document that suits you well. You fill out a form, and our customer service team takes care of the rest.
Who am I buying these notes from?
Stuvia is a marketplace, so you are not buying this document from us, but from seller oliviaward. Stuvia facilitates payment to the seller.
Will I be stuck with a subscription?
No, you only buy these notes for $7.09. You're not tied to anything after your purchase.
Can Stuvia be trusted?
4.6 stars on Google & Trustpilot (+1000 reviews)
75759 documents were sold in the last 30 days
Founded in 2010, the go-to place to buy study notes for 14 years now