|
2.8 Data Analysis and
Interpretation
 The
time and effort required for data analysis and interpretation
depends on the study's purpose and the methodology used.
Analysis and interpretation may take several days to several
months. In many private sector research studies involving only a
single question, however, data analysis and interpretation may
be completed in a few minutes.
For example, a business or company may be interested in
discovering the amount of interest in a new product or service.
After a survey, for example, the question may be answered by
summarizing only one or two items on the questionnaire that
relate to demand for the product or service. In this case,
interpretation is simply "go" or "no-go."
Every analysis should be carefully planned and performed
according to guidelines designed for that analysis. Once the
computations have been completed, the researcher must "step
back" and consider what has been discovered. The results must be
analyzed with reference to their external validity and the
likelihood of their accuracy.
Researchers must determine through analysis whether their work
is valid internally and externally.
This chapter has touched briefly on the concept of external
validity; an externally valid study is one whose results can be
generalized to the population. To assess internal validity, on
the other hand, one asks: Does the study really investigate the
proposed research question?
2.8.1 Internal Validity
 Control
over research conditions is necessary to enable researchers to
rule out all plausible rival explanations of results.
Researchers are interested in verifying that "y is a function of
x," or y = f(x). Control over the research conditions is
necessary to eliminate the possibility of finding that y =
f(b), where b is an extraneous variable. Any such
variable that creates a rival explanation of results is known as
an artifact (also referred to as extraneous variable). The
presence of an artifact indicates a lack of internal validity:
the study has failed to investigate its hypothesis.
Suppose, for example, that researchers discover through a study
that children who view television for extended lengths of time
have lower grade point averages in school than children who
watch only a limited amount of television. Could an artifact
have created this finding? It may be that children who view
fewer hours of television also receive parental help with their
school work: parental help (the artifact), not hours of
television viewed, may be the reason for the difference in grade
point averages between the two groups.
Sources of internal invalidity may arise from several places.
Those most frequently encountered are described in the list
below.
Researchers should be familiar with these sources to achieve
internal validity in the experiments they conduct.
1. History:
Various events occurring during a study may affect the
subjects' attitudes, opinions, and behavior. For example, to
analyze an oil company's public relations campaign for a new
product, researchers first pretest subjects concerning
their attitudes toward the company. The subjects are next
exposed to an experimental promotional campaign (the
experimental treatment); then a posttest is
administered to determine whether changes in attitude occurred
as a result of the campaign. Suppose the results indicate that
the public relations campaign was a complete failure—that the
subjects displayed a very poor perception of the oil company in
the posttest. Before the results are reported, the researchers
need to determine whether an intervening variable could have
caused the poor perception. An investigation discloses that
during the period between tests, subjects learned from a
television news story that the oil company was planning to raise
gasoline prices by 20%. The news of the price increase—not the
public relations campaign — may have acted as an artifact that
created the poor perception. The longer
the time period between a pretest and a posttest, the greater
the possibility that history might confound the study.
2. Maturity:
Subjects' biological and psychological characteristics change
during the course of a study. Growing hungry or tired or
becoming older may influence the manner in which subjects
respond to a research study. An example of how maturation can
affect a research project was seen in the early 1980s when radio
stations around the country began to test their music playlist
in auditorium sessions (where listeners are invited to a large
hotel ballroom to rate short segments of songs. Some unskilled
research companies tested up to 500 or 600 songs in one session
and wondered why the songs after about the 400th one tested
dramatically different from the other songs. Without a great
deal of investigation, researchers discovered that the
respondents were physically and emotionally drained once they
reached 400 songs (about 2 hours), and merely wrote down any
number just to complete the project.
3. Testing:
Testing in itself may be an artifact, particularly when
subjects are given similar pretests and posttests. A pretest may
sensitize subjects to the material and improve their posttest
scores regardless of the type of experimental treatment given to
subjects. This is especially true when the same test is used
for both situations. Subjects learn how to answer questions and
to anticipate researchers' demands. To guard against the effects
of testing, different pretests and posttests are required. Or,
instead of being given a pretest, subjects can be tested for
similarity (homogeneity) by means of a variable or set of
variables that differs from the experimental variable. The
pretest is not the only way to establish a point of prior
equivalency (the groups were equal before the experiment)
between groups—this can also be accomplished through sampling
(randomization and matching).
4. Instrumentation:
Also known as instrument decay, this term refers to
the deterioration of research instruments or methods over the
course of a study. Equipment may wear out, observers may
become more casual in recording their observations, and
interviewers who memorize frequently asked questions may fail to
present them in the proper order.
5. Experimenter bias:
There is a variety of ways in which a researcher may influence
the results of a study. Bias can enter through mistakes made
in observation, data recording, mathematical computations, and
interpretation. Whether
experimenter errors are intentional or unintentional, they
usually support the researcher's hypothesis and are considered
bias.
Experimenter bias can also enter into any phase of a research
project if the researcher becomes swayed by a client's wishes
for how a project will turn out.
The following example describes a situation that can cause
significant problems for researchers if they do not remain
totally objective throughout the entire project. The example is
not included here to suggest that research always works
this way, nor is it an endorsement of the situation.
Researchers are sometimes hired by individuals or companies to
"prove a point" or to have "supporting information" for a
decision (this is usually unknown to the researcher).
For example, the program director at a television station may
have a particular dislike for a program on the station and wants
to "prove" his "theory" correct. A researcher is hired
under the premise of finding out whether the audience likes or
dislikes the program. In this case, it is very easy for the
program director to intentionally or unintentionally sway the
results just through the conversations with the researcher in
the planning stages of the study. It is possible for a
researcher to intentionally or unintentionally interpret the
results in order to support the program director's desire to
eliminate the program. The researcher may, for instance, have
like/dislike numbers that are very close, but may give the
"edge" to dislike because of the program director's influence.
Experimenter bias is a potential problem in all phases of
research, and those conducting the study must be aware of
problems caused by outside influences.
Several procedures can help to reduce experimenter bias. For
example, individuals who provide instructions to subjects and
make observations should not be informed of the purpose of the
study; experimenters and others involved in the research should
not know whether subjects belong to the experimental group or
the control group (this is called a double blind experiment);
and automated devices such as tape recorders should be used
whenever possible to provide uniform instructions to subjects.
(See Chapter 5 for more information
about control groups.)
Researchers can also ask clients not to discuss the intent of a
research project beyond what type of information is desired. The
program director should say only that information is desired
about the like/dislike of the program and should not discuss
what decisions will be made with the research. In cases where
researchers must be told about the exact purpose of the project,
or where the researcher is conducting the study independently,
experimenter bias must be repressed at every phase.
6. Evaluation apprehension:
Concept of evaluation apprehension is similar to demand
characteristics, but it emphasizes that subjects are essentially
afraid of being measured or tested. They are interested
in receiving only positive evaluations from the researcher and
from the other subjects involved in the study. Most people are
hesitant to exhibit behavior that differs from the norm and will
tend to follow the group, even though they may totally disagree
with the others. The researcher's task is to try to eliminate
this passiveness by letting subjects know that their individual
responses are important.
7. Causal time-order:
The organization of an experiment may in fact create problems
with data collection and/or interpretation. It may be that
results of an experiment are not due to the stimulus
(independent) variable, but rather to the effect of the
dependent variable. For example, respondents in an experiment
about how advertising layouts in magazines influence their
purchasing behavior may change their opinions when they read or
complete a questionnaire after viewing several ads.
8. Diffusion or imitation of treatments:
In situations where respondents participate at different times
during one day or over several days, or groups of respondents
are studied one after another, respondents may have the
opportunity to discuss the project with someone else and
contaminate the research project. This is a special problem with
focus groups where one group often leaves the focus room while a
new group enters.
9. Compensation:
Sometimes individuals who work with a control group (the one
that receives no experimental treatment) may unknowingly treat
the group differently since the group was "deprived" of
something. In this case, the control group is no longer
legitimate.
10. Compensatory rivalry:
In some situations, subjects who know they are in a control
group may work harder or perform differently to out-perform the
experimental group.
11. Demoralization:
Control group subjects may literally lose interest in a project
because they are not experimental subjects. These people may
give up or fail to perform normally because they may feel
demoralized or angry that they are not in the experimental
group.
The sources of internal invalidity are complex and may arise in
all phases of research.
For this reason, it is easy to see why the results from a single
study cannot be used to refute or support a theory or
hypothesis. To try and control these
artifacts, researchers use a variety of experimental designs and
try to keep strict control over the research process so subjects
and researchers will not intentionally or unintentionally
influence the results.
2.8.2 External Validity
 External
validity refers to how well the results of a study can be
generalized across populations, settings, and time. The external
validity of a study can be severely affected by the interaction
in an analysis of variables such as subject selection,
instrumentation, and experimental conditions. A study that lacks
external validity cannot be projected to other situations. The
study is only valid for the sample tested.
Most procedures to guard against external invalidity relate to
sample selection. Here, three
considerations must be taken into account:
1. Use random samples.
2. Use heterogeneous samples and replicate the study several
times.
3. Select a sample that is representative of the group to which
the results will be generalized.
Using random samples rather than convenience or available
samples allows researchers to gather information from a variety
of subjects rather than those who may share similar attitudes,
opinions, and lifestyles.
As we will see later on, a random sample means that everyone
(within the guidelines of the project) has an equal chance of
being selected for the research study.
Several replicated research projects using samples with a
variety of characteristics (heterogeneous) allow researchers to
test hypotheses and research questions and not worry that the
results will only relate to one type of subject.
Selecting a sample that is representative of the group to which
the results will be generalized is basic common sense.
For example, the results from a study of a group of high school
students cannot be generalized to a group of college students.
A fourth way to increase external validity is to conduct
research over a long period of time.
Mass media research is often designed as short-term projects:
subjects are exposed to an experimental treatment and are
immediately tested or measured. However, in many cases, the
immediate effects of a treatment are negligible. In advertising,
for example, research studies designed to measure brand
awareness are generally based on only one exposure to a
commercial or advertisement. It is well known that persuasion
and attitude change rarely take place after only one exposure;
they require multiple exposures over time. Logically, such
measurements should be made over a period of weeks or months to
take into account the sleeper effect: that attitude change may
be minimal or nonexistent in the short run and still prove
significant in the long run. |
Most
researchers do not actually conduct every phase of every project
they supervise.
The
format used in presenting results depends on the purpose of the
study.
One
important point is that the results of any single study are, by
themselves, only indications of what might exist. A study
provides information that says, in effect, "This is what may be
the case." To be relatively certain of the results of any study,
the research must be replicated. Too often, researchers conduct
one study and report the results as if they are providing the
basis for a theory or law. The information presented in this
chapter, and in other chapters that deal with internal and
external validity, argues that this cannot be true.
All
researchers quickly discover that research projects do not
always turn out the way they were planned. It seems that
Murphy's Law — anything that can go wrong will go wrong — holds
true in any type of research. It is therefore necessary to be
prepared for difficulties, however minor, in conducting a
research project. Planning and flexibility are essential.
Presented below is what is known as the TAT
(They're Always There) laws. Although these "laws" are somewhat
tongue-in-cheek, they are nonetheless representative of the
problems one may expect to encounter in research studies.