fSyrhO/Ogy
in
the
Schools
Volume
29.
April
1992
DETECTING POTENTIAL HUCKSTERISM
IN
META-ANALY SIS
USING
A FOLLOW-UP FAIL-SAFE TEST
JONATHAN
R.
BROWN
State University
of
New York
Meta-analysis is an analysis
of
analyses. It is a technique widely used by researchers
and practitioners to aggregate and summarize statistically reported empirical educa-
tional research. In
10
years, meta-analysis appeared more than
600
times in research
journals and dissertation abstracts. Although most meta-analyses were reported as
significant, few
of
the findings determined how many unpublished ”no-effect” studies,
if
sampled, would have invalidated significance.
If
significant meta-analysis results
are overrepresented through selective sampling, hucksterism in the form
of
sampling
bias exists. An explanation for using a follow-up test called the fail-safe Nis provided
with tables constructed to assist researchers and practitioners to estimate, without
calculation, the relative stability
of
meta-analysis results. The implication is that fail-
safe
N
should routinely be used and reported in meta-analysis research.
Whenever an educational controversy arises, reported research is sought to sup-
port a particular position. Educators often initiate a futile search for the “perfect” study.
They may also tend to cite only research that supports their personal biases. The old
research and the new, however, often remain isolated and variegated in a data base filled
with conflicting findings (Kavale, 1984). Meta-analysis is a process designed to help pro-
fessionals resolve conflicting research findings. The fail-safe
N
is a follow-up procedure
to assess meta-analysis results to detect possible misrepresentation,
or
hucksterism (selling
or
advertising in an aggressive and questionable way). The purpose of this article is to
describe briefly a way to use the fail-safe
N.
Meta-analysis is a widely used methodological technique to aggregate and summarize
statistically a number of empirical educational research studies analyzed to test a specified
research hypothesis. From 1981 to 1991, meta-analysis appeared more than
600
times
in abstracts of articles cited in Educational, Psychological, and Dissertation Abstracts.
Meta-analysis is important because it helps advance theory and knowledge. This ad-
vancement comes as the result of summarizations, theoretical linking
or
modeling across
a wide variety
of
samples, and suggesting that certain relationships are understood and
additional investigation is warranted. It is a technique that may be used with minimum
technical knowledge about complex statistics to draw meaningful relationships for every-
day applications (Brown
&
Brown, 1987).
The fail-safe Nstatistic is a follow-up test used with meta-analysis results to estimate
the number of new, unpublished,
or
unretrieved nonsignificant (null-result) studies that
would, on the average, change the significance of a meta-analysis study to nonsignificance.
Nonsignificance would result in concluding that the sampled studies were not congruent
with some specified research hypothesis analyzed by meta-analysis (Carson, Schriesheim,
&
Kinicki, 1990). The fail-safe
N
is based on initial work by Rosenthal (1979). Rosen-
thal’s work was later modified by Orwin (1983).
A significant meta-analysis study, Rosenthal(l979) reasoned, could be made non-
significant by adding some number of hypothetical studies that averaged null results.
Correspondence and requests for reprints should be addressed to Jonathan
R.
Brown, State University
of
New
York,
Fredonia,
NY
14063.
179
资源评论
