Once you start to understand how exciting the world of statistics can
Once you start to understand how exciting the world of statistics can be, it is tempting to fall into the trap of chasing statistical significance. That is, you may be tempted always to look for relationships that are statistically significant and believe they are valuable solely because of their significance. Although statistical hypothesis testing does help you evaluate claims, it is important to understand the limitations of statistical significance and to interpret the results within the context of the research and its pragmatic, “real world” application.
As a scholar-practitioner, it is important for you to understand that just because a hypothesis test indicates a relationship exists between an intervention and an outcome, there is a difference between groups, or there is a correlation between two constructs, it does not always provide a default measure for its importance. Although relationships are significant, they can be very minute relationships, very small differences, or very weak correlations. In the end, we need to ask whether the relationships or differences observed are large enough that we should make some practical change in policy or practice.
For this Discussion, you will explore statistical significance and meaningfulness.
To prepare for this Discussion:
- Review the Learning Resources related to hypothesis testing, meaningfulness, and statistical significance.
- Review Magnusson’s web blog found in the Learning Resources to further your visualization and understanding of statistical power and significance testing.
- Review the American Statistical Association’s press release and consider the misconceptions and misuse of p-values.
- Consider the scenario:
- A research paper claims a meaningful contribution to the literature based on finding statistically significant relationships between predictor and response variables. In the footnotes, you see the following statement, “given this research was exploratory in nature, traditional levels of significance to reject the null hypotheses were relaxed to the .10 level.”
Post your response to the scenario in which you critically evaluate this footnote. As a reader/reviewer, what response would you provide to the authors about this footnote?
https://class.waldenu.edu/webapps/bbgs-deep-links-BBLEARN/app/course/rubric?course_id=_16783742_1&rubric_id=_2104081_1 (this is the rubric)
***also you must receive a 90 percent or better on this assignment***
Required Readings
Frankfort-Nachmias, C., Leon-Guerrero, A., & Davis, G. (2020). Social statistics for a diverse society (9th ed.). Thousand Oaks, CA: Sage Publications.
Chapter 8, “Testing Hypothesis: Assumptions of Statistical Hypothesis Testing” (pp. 241-242)
Wagner, III, W. E. (2020). Using IBM® SPSS® statistics for research methods and social science statistics (7th ed.). Thousand Oaks, CA: Sage Publications.
Chapter 6, “Testing Hypotheses Using Means and Cross-Tabulation”
Warner, R. M. (2012). Applied statistics from bivariate through multivariate techniques (2nd ed.). Thousand Oaks, CA: Sage Publications.
Applied Statistics From Bivariate Through Multivariate Techniques, 2nd Edition by Warner, R.M. Copyright 2012 by Sage College. Reprinted by permission of Sage College via the Copyright Clearance Center.
Chapter 3, “Statistical Significance Testing” (pp. 81–124)
Applied Statistics From Bivariate Through Multivariate Techniques, 2nd Edition by Warner, R.M. Copyright 2012 by Sage College. Reprinted by permission of Sage College via the Copyright Clearance Center.
Magnusson, K. (n.d.). Welcome to Kristoffer Magnusson’s blog about R, Statistics, Psychology, Open Science, Data Visualization [blog]. Retrieved from http://rpsychologist.com/index.html
As you review this web blog, select [Updated] Statistical Power and Significance Testing Visualization link, once you select the link, follow the instructions to view the interactive for statistical power. This interactive website will help you to visualize and understand statistical power and significance testing.
Note: This is Kristoffer Magnusson’s personal blog and his views may not necessarily reflect the views of Walden University faculty.
American Statistical Association (2016). American Statistical Association Releases Statement on Statistical Significance and P-Values. Retrieved from http://www.amstat.org/newsroom/pressreleases/P-ValueStatement.pdf
As you review this press release, consider the misconceptions and the misuse of p-values in quantitative research.
Document: Week 5 Scenarios (PDF)
Use these scenarios to complete this week’s Assignment.
© 2016 Laureate Education, Inc. Page 1 of 2
Week 5
Scenarios
1. The p-value was slightly above conventional threshold, but was described as “rapidly approaching significance” (i.e., p =.06).
An independent samples t test was used to determine whether student satisfaction levels in a quantitative reasoning course differed between the traditional classroom and on-line environments. The samples consisted of students in four face-to-face classes at a traditional state university (n = 65) and four online classes offered at the same university (n = 69). Students reported their level of satisfaction on a five- point scale, with higher values indicating higher levels of satisfaction. Since the study was exploratory in nature, levels of significance were relaxed to the .10 level. The test was significant t(132) = 1.8, p = .074, wherein students in the face-to-face class reported lower levels of satisfaction (M = 3.39, SD = 1.8) than did those in the online sections (M = 3.89, SD = 1.4). We therefore conclude that on average, students in online quantitative reasoning classes have higher levels of satisfaction. The results of this study are significant because they provide educators with evidence of what medium works better in producing quantitatively knowledgeable practitioners.
2. A results report that does not find any effect and also has small sample size (possibly no effect detected due to lack of power).
A one-way analysis of variance was used to test whether a relationship exists between educational attainment and race. The dependent variable of education was measured as number of years of education completed. The race factor had three attributes of European American (n = 36), African American (n = 23) and Hispanic (n = 18). Descriptive statistics indicate that on average, European Americans have higher levels of education (M = 16.4, SD = 4.6), with African Americans slightly trailing (M = 15.5, SD = 6.8) and Hispanics having on average lower levels of educational attainment (M = 13.3, SD = 6.1). The ANOVA was not significant F (2,74) = 1.789, p = .175, indicating there are no differences in educational attainment across these three races in the population. The results of this study are significant because they shed light on the current social conversation about inequality.
3. Statistical significance is found in a study, but the effect in reality is very small (i.e., there was a very minor difference in attitude between men and women). Were the results meaningful?
An independent samples t test was conducted to determine whether differences exist between men and women on cultural competency scores. The samples consisted of 663 women and 650 men taken from a convenience sample of public, private, and non-profit organizations. Each participant was administered an instrument that measured his or her current levels of cultural competency. The
© 2016 Laureate Education, Inc. Page 2 of 2
cultural competency score ranges from 0 to 10, with higher scores indicating higher levels of cultural competency. The descriptive statistics indicate women have higher levels of cultural competency (M = 9.2, SD = 3.2) than men (M = 8.9, SD = 2.1). The results were significant t (1311) = 2.0, p <.05, indicating that women are more culturally competent than are men. These results tell us that gender-specific interventions targeted toward men may assist in bolstering cultural competency.
4. A study has results that seem fine, but there is no clear association to social change. What is missing?
A correlation test was conducted to determine whether a relationship exists between level of income and job satisfaction. The sample consisted of 432 employees equally represented across public, private, and non-profit sectors. The results of the test demonstrate a strong positive correlation between the two variables, r =.87, p < .01, showing that as level of income increases, job satisfaction increases as well.
,
Student 1
I am going to break down the sentence “given this research was exploratory in nature, traditional levels of significance to reject the null hypotheses were relaxed to the .10 level.” In order to critically analyze its meaning, the operative phrases to be analyzed include exploratory, traditional levels of significance, null hypothesis, and .10 level. Now, the first concept mentioned is ‘exploratory in nature’. Exploratory is typically a purpose of qualitative research and if this is the case, the data obtained from a qualitative study should not be quantified. However, in this case, I believe the exploratory in nature means that the research is still exploring possibilities and is exploring options for a more formal hypothesis.
In many cases, the best way to test a hypothesis is to state the null hypothesis which means there is no relationship or no difference (Frankfort-Nachmias, Leo-Guerrero, & Davis, 2020). So, what the statement is saying is that they used a traditional level of significance to confirm the hypothesis and reject the null hypothesis. Relaxing traditional levels of significance to the .10 level means lowering the likelihood of rejecting the null hypothesis (Warner, 2020). The .10 is an alpha level use to test the level of significance of the data. What I would tell the authors about the footnote is that although most statistics apply a .5 level which represents a low risk of a type 1 error (incorrectly rejecting the null), it is common to see researchers use the .10 in exploratory research (Warner, 2020).
References:
Frankfort-Nachmias, C., Leon-Guerrero, A., & Davis, G. (2020). Social statistics for a diverse society (9th ed.). Thousand Oaks, CA: Sage Publications.
Warner, R. M. (2012). Applied statistics from bivariate through multivariate techniques (2nd ed.). Thousand Oaks, CA: Sage Publications.
Student 2
Scenario: A research paper claims a meaningful contribution to the literature based on finding statistically significant relationships between predictor and response variables. In the footnotes, you see the following statement, "given this research was exploratory in nature, traditional levels of significance to reject the null hypotheses were relaxed to the .10 level."
The scenario states, "given this research was exploratory in nature, traditional levels of significance to reject the null hypotheses were relaxed to the .10 level". Exploratory research is implemented during an investigation when defining an existing problem without a provided conclusion/results. This statement alone speaks to the type of research conducted and how it is being carried out thus far. When referring to the traditional level of significance, it means that a hypothesis has been formulated and the "p-value" is less than or equal to a considered significance. The significance level is crucial because it shows the probability in which the null hypothesis could be rejected when it is true. According to Frankforth-Nachmias & Leon-Guerrero (2020), a null hypothesis is a statement that reflects "no difference which contradicts the research hypothesis and is always expressed in terms of population parameters; When the null hypothesis is rejected, it strengthens the belief in the research hypothesis. It increases the researcher's confidence. When testing a null hypothesis, first, the researcher makes a "guess" about a specific value, then there is a random selection. The researcher then compares the mean of the observed sample (Warner, 2012).
As a reader/reviewer, the response I would provide to the authors about this footnote is that within the exploratory research, the traditional level of significance is the probability in which the research guess may be true despite the possibility of being rejected. If the null hypothesis has become relaxed to a .10 level, that means that the confidence of the null hypothesis being rejected has decreased. This reflects that the research is progressing in confidence, and there is a conclusion of the correct results.
Resources
Frankfort-Nachmias, C., Leon-Guerrero, A., & Davis, G. (2020). Social statistics for a diverse society (9th ed.). Thousand Oaks, CA: Sage Publications.
Chapter 8, "Testing Hypothesis: Assumptions of Statistical Hypothesis Testing" (pp. 241-242)
Wartner, R.M. (2012). Applied Statistics from bivariate through multivariate techniques (2nd ed.). Thousand Oaks, CA: Sage Publications.
Chapter 3, "Statistical Significance Testing" (pp. 81-124)
Student 3
The statistical significance and meaningfulness are essential and vital to evaluating statistics because the statistical significance focuses on a statistic's value. In contrast, the meaningfulness factors applicability in the real world (Laureate Education (Producer), 2016f). A researcher needs to align the statistics with the claim and justifying using the data (Laureate Education (Producer), 2016f).
With the statement "traditional levels of significance to reject the null hypotheses were relaxed to the .10 level," it would signify that the predictability or P-value is less strict. Dr. Matt Jones stated in the video Meaningfulness vs. Statistical Significance that just because you have a P-value below 0.01, it doesn't mean the effect is significant, the difference may be small between two groups (Laureate Education (Producer), 2016f). Therefore, a level of .10 level may allow for additional probabilities but it may not be a very large significance.
In this specific case, because they indicate the research is exploratory, by relaxing the level to .10 they allow additional probabilities, analysis, or data with continued literature. By reporting this information, the authors are also allowing the reader to determine if data is usable or applicable for their research or study. It is an ethical thing to report.
Laureate Education (Producer). (2016). Meaningfulness vs. statistical significance [Video file]. Baltimore, MD: Author.
Student 4
Scenario: A research paper claims a meaningful contribution to the literature based on finding statistically significant relationships between predictor and response variables. In the footnotes, you see the following statement, "given this research was exploratory in nature, traditional levels of significance to reject the null hypotheses were relaxed to the .10 level."
In VandenBos , 2007, defines exploratory research a study that is conducted when not much is known about a particular phenomenon. In exploratory research, one typically seeks to identify multiple possible links between variables. The null hypothesis contradicts the research hypothesis and states that there is no difference between the population mean and some specified value. It is also referred to as the hypothesis of "no difference," In hypothesis testing, we hope to reject the null hypothesis to provide indirect support for the research hypothesis. Rejection of the null hypothesis will strengthen our belief in the research hypothesis and increase our confidence in the importance and utility of the broader theory from the research hypothesis was derived. (Frankfort-Nachmias, Leon-Guerrero & Daivs 2020).
Sharma, 2017, states that most often level of significance of 5% is chosen as a standard practice. However, levels like 1% and 10% can also be chosen e.g. if our p-value is 0.7, we say that our results are insignificant at 5% level (and we should accept our null hypothesis at this level) and are significant at 10% level (and we should reject our null hypothesis at this level). Following the example set by Sir Ronald Fisher, most users of statistics assume than a value of .05 represents an acceptably small risk of Type I error in most situations. However, in exploratory research, investigators are sometimes willing to use alpha levels (such as a = .10) that correspond to a higher risk of Type I error. (Warner, 2012) I would say to the authors, that there is a 10% chance of finding relationships between predictor and response given that the null hypothesis is true.
Frankfort-Nachmias, c,. Leon-Guerrero, A., & Davis, G. (2020). Social statistics for a diverse society (9th ed.) Thousand Oaks, CA: Sage Publications
Sharma, S. (2017). p-value and level of significance explained. Data Science Central, [blog] Retrieved from https://www.datasciencecentral.cpm/profiles/blogs/p-value-and-level-of-significance-explained
VandenBos, G. R. (Ed.) (2007). APA dictionary of psychology
Warner, R. M. (2012). Applied statistics from bivariate through multivariate techniques (2nd ed.). Thousand Oaks, CA: Sage Publications.
S
tudent 1
I
am
going
to
break
down
the
sentence
“given
this
research
was
exploratory
in
nature,
traditional
levels
of
significance
to
reject
the
null
hypotheses
were
relaxed
to
the
.10
level.”
In
order
to
critically
analyze
its
meaning,
the
operative
phrases
to
be
a
nalyzed
include
exploratory,
traditional
levels
of
significance,
null
hypothesis,
and
.10
level
.
Now,
the
first
concept
mentioned
is
‘exploratory
in
nature’.
Exploratory
is
typically
a
purpose
of
qualitative
research
and
if
this
is
the
case,
the
data
obta
ined
from
a
qualitative
study
should
not
be
quantified.
However,
in
this
case,
I
believe
the
exploratory
in
nature
means
that
the
research
is
still
exploring
possibilities
and
is
exploring
options
for
a
more
formal
hypothesis
.
In
many
cases,
the
best
way
to
test
a
hypothesis
is
to
state
the
null
hypothesis
which
means
there
is
no
relationship
or
no
difference
(
Frankfort
–
Nachmias,
Leo
–
Guerrero,
&
Davis,
202
0
).
So,
what
the
statement
is
saying
is
that
they
used
a
traditional
level
of
significance
to
confirm
the
hypothesis
and
reject
the
null
hypothesis.
Relaxing
traditional
levels
of
significance
to
the
.10
level
means
lowering
the
likelihood
of
rejecting
the
null
hypothesis
(Warner,
2020).
The
.10
is
an
alpha
level
use
to
test
the
level
of
significance
of
th
e
data.
What
I
would
tell
the
authors
about
the
footnote
is
that
although
most
statistics
apply
a
.5
level
which
represents
a
low
risk
of
a
type
1
error
(incorrectly
rejecting
the
null),
it
is
common
to
see
researchers
use
the
.10
in
exploratory
research
(
Warner,
2020)
.
References
:
Frankfort
–
Nachmias,
C.,
Leon
–
Guerrero,
A.,
&
Davis,
G.
(2020)
.
Social
statistics
for
a
diverse
societ
y
(9th
ed.).
Thousand
Oaks,
CA:
Sage
Publications
.
Warner,
R.
M.
(2012)
.
Applied
statistics
from
bivariate
through
multivariate
technique
s
(2nd
ed.).
Thousand
Oaks,
CA:
Sage
Publications
.
Student 2
Scenario
:
A
research
paper
claims
a
meaningful
contribution
to
the
literature
based
on
finding
statistically
significant
relationships
betwe
en
predictor
and
response
variables.
In
the
footnotes,
you
see
the
following
statement,
"given
this
research
was
exploratory
in
nature,
traditional
levels
of
significance
to
reject
the
null
hypotheses
were
relaxed
to
the
.10
level.
"
Student 1
I am going to break down the sentence “given this research was exploratory in
nature, traditional levels of significance to reject the null hypotheses were
relaxed to the .10 level.” In order to critically analyze its meaning, the operative
phrases to be analyzed include exploratory, traditional levels of significance, null
hypothesis, and .10 level. Now, the first concept mentioned is ‘exploratory in
nature’. Exploratory is typically a purpose of qualitative research and if this is the
case, the data obtained from a qualitative study should not be quantified.
However, in this case, I believe the exploratory in nature means that the research
is still exploring possibilities and is exploring options for a more formal
hypothesis.
In many cases, the best way to test a hypothesis is to state the null hypothesis
which means there is no relationship or no difference (Frankfort-Nachmias, Leo-
Guerrero, & Davis, 2020). So, what the statement is saying is that they used a
traditional level of significance to confirm the hypothesis and reject the null
hypothesis. Relaxing traditional levels of significance to the .10 level means
lowering the likelihood of rejecting the null hypothesis (Warner, 2020). The .10
is an alpha level use to test the level of significance of the data. What I would tell
the authors about the footnote is that although most statistics apply a .5 level
which represents a low risk of a type 1 error (incorrectly rejecting the null), it is
common to see researchers use the .10 in exploratory research (Warner, 2020).
References:
Frankfort-Nachmias, C., Leon-Guerrero, A., & Davis, G. (2020). Social
statistics for a diverse society (9th ed.). Thousand Oaks, CA: Sage Publications.
Warner, R. M. (2012). Applied statistics from bivariate through multivariate
techniques (2nd ed.). Thousand Oaks, CA: Sage Publications.
Student 2
Scenario: A research paper claims a meaningful contribution to the literature based
on finding statistically significant relationships between predictor and response
variables. In the footnotes, you see the following statement, "given this research was
exploratory in nature, traditional levels of significance to reject the null hypotheses
were relaxed to the .10 level."
Collepals.com Plagiarism Free Papers
Are you looking for custom essay writing service or even dissertation writing services? Just request for our write my paper service, and we'll match you with the best essay writer in your subject! With an exceptional team of professional academic experts in a wide range of subjects, we can guarantee you an unrivaled quality of custom-written papers.
Get ZERO PLAGIARISM, HUMAN WRITTEN ESSAYS
Why Hire Collepals.com writers to do your paper?
Quality- We are experienced and have access to ample research materials.
We write plagiarism Free Content
Confidential- We never share or sell your personal information to third parties.
Support-Chat with us today! We are always waiting to answer all your questions.
