A Framework for Media Comparison Analysis

Bryan R. Warnick
Nicholas C. Burbules (?)
University of Illinois at Urbana Champaign
Media comparison research has been a hotly disputed area of educational
research. This research aims at finding significant differences in educational
outcomes between students who are taught using different types of media. Thus,
one group of students will usually be taught using one media, another group will
be taught with a different media, and the results of subsequent tests will be
compared. This research hopes to find, if there is an educational difference
between media, which media are more effective in the process of teaching and
learning. At first glance, this research would seem to be an important helping in
navigating classrooms through the wealth of new media that have recently
become available to educators.
Media-comparison studies, however, have been accused of being both
useless and conceptually incoherent. Media comparison studies are said to be
useless because the majority of such research finds no statistically significant
difference between learning groups. The No-Significant Difference website
(http://teleeducation.nb.ca/nosignificantdifference/) lists 114 studies dating

from 1920 that show no significant different between technologies, compared to

only 39 that do. According to Thomas Russell, one of the sites organizers, some
doctoral candidates in Adult and Continuing Education Programs are being
discouraged from pursuing such research, in that it amounts to beating a dead
horse. And rightly so, he argues: The No Significant Difference Phenomenon
listing will no longer be complied when the futility of conducting such studies is
recognized and their production ceases.
Russell also advances a theory about why studies commonly find no
significant difference between media. He writes:
Technology is not neutral, despite the fact that study after study has
concluded that using it in the classroom neither improves nor diminishes
instruction for the masses. The truth lies in the fact, often acknowledged
but ignored, that students are not alike. Individual differences in learning
styles dictate that technology will facilitate learning for some, but will
probably inhibit learning for others, while the remainder experience no
significant difference. Therefore, when lumping all the students together
into a fictional "mass," those who benefit from the technology are balanced
by a like number who suffer; when combined with the no-significantdifference majority, the conglomerate yields the widely reported "no
significant difference" results.
Russell is probably correct that this conglomerate effect has been playing a part
in much media comparison research. However, this result may occur with any
line of research into teaching and learning, not just the research into media
differences. Indeed, individual differences are obscured in almost any
statistically based study, but this does not mean that it is impossible to

demonstrate broader trends. Thus, the conglomerate effect, as a possibility

shared with other programs in educational research, does not seem to be a
compelling reason to stop media comparison studies. Russells argument,
however, seems to be that, given the existing evidence, the conglomerate effect
has been shown to be the necessary result of media comparison research. It
seems unwise to make such strong statements, though, even if they are valid
summaries about the state of research. New media are constantly emerging, and
what was true of the past may not hold for the future.
Perhaps the most prominent critic of media comparison research has been
Richard E. Clark. In his numerous attacks on media comparison research, he
writes that media are mere vehicles that deliver instruction but do not influence
student achievement any more than the truck that delivers our groceries causes
changes in our nutrition. He also claims any necessary teaching method could
be designed into a variety of media presentations. To the question of why some
students have found a significant difference between educational media, he
argues that these studies have confounded media with method. That is, the
studies that have found a difference are really only finding a difference between
two methods of instruction and not between the media themselves. Thus, such
studies have failed to hold method constant making it inappropriate to
conclude that media are the important independent variable.

In support of his argument that media will never influence learning, he

presents a medical analogy. Educational media are compared to the various
ways in which medicine can be delivered to the body (e.g., pills, injections,
suppositories, etc.). Clark argues that the manner of delivery makes no
difference to the healing properties of the medicine: All of these different media
are often capable of delivering a necessary active chemical ingredient with
necessary levels of efficacy, but with more or less equal effects on our physical
symptoms. The chemical component of the medicine he then compares to the
method of instruction. Methods of instruction, like the physicians medicine, can
be distributed in many ways that will be equally effective.
This debate highlights important issues that arise in researching
educational technology. The remainder of this paper will discuss some of
conceptual confusions that seem have arisen in this debate. There are at least
three levels on which conceptual confusion can occur as media are compared. It
can occur (1) when there is an unclear notion of what it means to make a
comparison, (2) when there is a simplistic view of how technologies can be
compared, and (3) when there are confusions relating to educational media
comparison specifically. The goal of this paper is to attempt to untangle some of
these confusions, so that media comparison studies may become more profitable.
WHAT DOES IT MEAN TO MAKE A COMPARISON?

Nicholas Burbules, in an article entitled, A Grammar of Difference: Some

Ways of Rethinking Difference and Diversity as Educational Topics, undertakes
a conceptual analysis of difference, a topic that has much to do with the
concept of comparison since finding differences is accomplished through
comparison. Burbules makes several points about the necessary assumptions
that need to be made in order to say X is different from Y, and which thus
undergird the process of comparing. A comparisons of X and Y must be done
within a framework that makes the comparison possible. It would be difficult to
compare, for example, the color orange with the number three -- there must be
some common ground that will make the comparison interesting. Thus, any
comparative framework will necessarily imply similarity along with any
difference (to fruitfully compare apples with oranges we may assume a common
category of food, and focus on taste, nourishment, and texture). Moreover,
assertions about what is, and what is not different, are in a constant state of
flux as the categories making up the comparative framework change. What we
find different between X and Y at time T1, may turn out to be irrelevant at T2, or
even turn into a similarity not because X and Y have changed (although that
may happen), but because the conceptual framework we are using has changed.
In sum, to make a comparison involves a making various and mutable

assumptions about similarity. Similarities will necessarily be assumed in the

process of comparing and finding difference.
What else is involved in a comparison? A comparison typically involves a
point or aim that gives the comparer motivation to make the comparison. There
are at least two types of aims that can give a comparison a purpose. First,
comparisons can be evaluative. In an evaluative comparison, we want to find out
which of two or more things is better suited for a given purpose, P. An example
of an evaluative comparison would be comparing two teams to see discover
which one is superior. Second, there are interpretive comparisons. In an
interpretative comparison, we do not wish to find out which is better for a given
purpose; instead, we simply want to increase understanding by holding up one
thing in light of another. An example of an interpretive comparison would be
comparing the development of Leer with Hamlet. While it is possible to do
interpretative comparisons in research on educational technology, much media
comparison research is often of the evaluative type of comparison. This will type
of comparison, then, will be our focus here.
An aim will include with it a comparative standard. This comparative
standard will stipulate the standards by which success at achieving the aim will
be known. A standard will spell out (or operationalize if you will) what we
mean by superior. That is, it will stipulate the measure according to which

superior will be judged. When making comparisons in pedagogical

techniques, the common standard is, of course, an objective test, administered
after an intervention. In these cases, the objective test gives the standard by
which to differentiate effectiveness of the intervention. The higher test score is
the standard for choosing which educational intervention is superior.
A comparison must also stipulate an aspect of comparison. That is, in
order to compare two things, we need to ask, Which aspect of the things are to
be compared, among the infinite possibilities? For example, in comparing two
teams to find out which is superior, I may compare many things: hair colors, shoe
sizes, scoring averages, heights, weights, birthdays, past records, favorite foods,
etc. To make a comparison successful, I need to narrow down this list of
possibilities to a list of relevant features. Only some of these aspects of
comparison will be relevant to the comparison under any given aim.
ASSUMED SIMILARITIES IN FRAMEWORKS OF COMPARISON
Using these three parts of a comparison -- aim, standard, and aspect
along with the idea that for any comparison to succeed certain similarities must
be assumed, lets look more closely at the media-comparison studies. What
similarities tend to be assumed in this field of research? First, media studies
usually assume a common purpose between two technologies only with

reference to a common purpose can two media be easily compared. For any
given comparison, two media must be evaluated in terms of a similar goal.
In assessing whether purposes have been achieved, media comparison
studies usually also assume a similar standard of evaluation between the two
media under consideration. It would be difficult to compare two educational
media, for example, if different tests were given to those who used a different
media. Such an act would, of course, confound the variables, thus making a
media comparison problematic. Both groups must be given the same test. Media
comparison studies must operate under a similar standard of evaluation.
Media comparison studies generally look at the common ability of two
different media to develop skills, increase knowledge, or otherwise promote
learning. The experience of two media may be different in many ways one may
increase eyestrain, another may come with a soporific humming noise, another
may be simply new and exciting. Media comparison studies must focus on a
narrow aspect of the media, and generally chooses to focus on the efficacy with
which media promotes an intended given learning outcome. Media comparison
studies must focus on a similar aspect between two technologies.
To the extent that media comparison studies assume similarities in order
to make a comparison, they will also suffer from various problems. Many people
would be quick to point out that the standard of comparison, objective test

scores, may often miss what differences there may be between two media. There
has been some debate, for example, about the differences between actual cadaver
dissection in medical school and anatomical simulations. Empirical research has
reported the usual no significant difference result between simulated
dissection and real dissection when anatomy knowledge is tested by objective
tests. Yet there remain differences between these two experiences, of course. At
the very least, one situation comes with the smell of formaldehyde, the other
doesnt. But there are other differences more closely tied to education: in one
situation there is the knowledge you are working with a dead human body, in the
other you isnt. Perhaps the best way to describe this difference would be one of
mood. There is a different mood between cutting into a human body and
using an anatomical simulation. This mood may or may not make an important
difference to education, but it would be a difficult thing to reduce to an
objective test. Assuming a framework of similarity when it comes to the
standard of objective tests will be problematic when it comes to finding
differences like alternation is mood.
To further underscore this point, consider Clarks medical analogy:
medicine can be delivered to the body in various ways, thorough pills, injections,
suppositories, etc. The manner of delivery, Clark argues, makes no difference to
the healing properties of the medicine. The medium of delivery fails to affect

the medicinal effect of the drug. We may grant this point, but that does not mean
that media of delivery does not matter to healing. Medical instruments also come
with mood. Needles may invoke a prohibitive mood in the patient; he or she
may refuse treatment. If the patient refrains from the medicine because of this
mood, then that can be said to be a factor affecting the healing process. If healing
is comparable to learning, and if media influence mood, then the medical
analogy would support of the point that, contrary to Clark, media matters to
learning. Learning cannot be divorced from things like the mood in which
learning takes place.
Problems also arise when we assume a common purpose for media
technology. Technology always changes the ends of human activity. Technical
ends do not simply remain fixed as means change. Using a new technology
opens up new ends, it does not just increase the speed or efficacy involved in
reaching the old ends; rather, using new technologies influences ones ends. In a
sense, then, existing means have helped to construct existing ends. This fact
often muddies technology comparisons. It would make no sense to evaluate the
difference between automobile-infused cultures with that of a culture that walks
everywhere under the framework of ends constructed in the walking society.
That is, it would make no sense to compare the time it takes to walk to the nearby
drug store with that it takes to drive there, and then declare triumphantly that a

significant difference of 3 minutes had been found. This comparison would miss
everything important about how automobiles change things. The fact that
automobiles exist changes the ends the walker does not have the end of just
going to the drugstore anymore. To take an educational example, consider using
a long division calculator program to teach long division, and then comparing
this to teaching long division through paper and pencil. This is problematic,
because the fact that a long-division calculator program exists changes the end of
the activity. The point of math education may then no longer be learning
algorithms, but may shift to learning how to use the application. Thus, it is not
always helpful to compare different technologies in relation to a common end.
Finally, assumptions about the similar aspects of comparison are also
problematic. The assumed aspect of media comparison studies is usually the
ability to promote an intended learning outcome. Such studies may miss the
aspects that surround the medias ability to promote an intended learning
outcome. The narrow focus may leave out social, cultural, and physiological
differences that may exist between the media. Comparing the ability of a book
and a computer to simply promote an intended learning outcome may leave out
the fact that students learn about their surrounding cultural life by using its tools.
Using computers in schools sends the message that it is good to use computer it
serves as a mark of societal validation (the same holds true for books). Thus,

using a media may promote future economic consumption of that media. If this
is true, this phenomenon would certainly be a type of learning, but it would be
left out in studies focused on the aspect of specific, intended learning outcomes.
COMPARING TECHNOLOGIES: PROBLEMS AND POSSIBILITIES
Suppose we wanted to compare the speed of traveling by horse with the
speed of traveling by car. This seems to be an easy experiment: designate a
beginning and end point, and time how long it takes to reach the end point using
the two travel techniques. Indeed, it seems we dont even have to do the
experiment traveling by car is obviously much faster than traveling by horse.
But consider if I was mounted on a horse, riding around the cattle car of a
moving train. In this context, it seems that I could both truly say I was traveling
by horse (in the context of a moving car) and that I was traveling faster than a car.
This thought experiment, though, need not be so silly consider that the car has
broken down. Then traveling by horse is faster than traveling by car (in the
context of a broken-down car). The use of a technology cannot be divorced from
the context in which the technology finds itself. The physics of a tool
underdetermines its context of use.
What does this mean? At first glance, it would seem to make any general
statements comparing two technologies impossible. Since contexts are always
variable across two or more situations, it makes no sense to make any general

statement across these contexts. To compare technologies is to also compare

contexts of use, and contexts of use cannot be stipulated in a generalized
statement. At the same time, it also seems extreme to say that you cant make
any general statements about the comparative speed of a horse versus the
comparative speed of car. Why does it seem misguided? The reason is that
general comparative statements make built in assumptions about normal uses.
When we way that a car is faster than a horse, we make a statement assuming a
normal context of use one does not usually travel by horse in the context of a
moving cattle-car. This is understood when the comparative statement is made.
So, in order to compare two technologies, we need to determine what a
normal context of use is. Admittedly, this can sometimes be a difficult problem.
While it may be easy to see what something like scissors are normally use for,
the case become murky with some technologies like computers that seem to
have an unending variety of functions. There do seem to be several
characteristics of technologies, though, that offer some hope in stipulating a
normal context of use. These characteristics show that stipulating a normal
context of use is at least theoretically possible.
First, an important characteristic to remember about technologies is that
they are not neutral. They are not neutral in many senses. Household appliances
may be built specifically for women, for example, and thus betray a lack of

political neutrality. This comes from the fact that technologies can suggest a
context of use. By having a single keyboard, computers suggest a context of
solitary use, rather than group work (although groups can, of course, awkwardly
use a single computer keyboard). Thus, although technologies may still be used
awkwardly in a wide variety of contexts, their designs often suggest one context
over another.
Moreover, even the awkward contexts of use are not indefinitely variable.
Some contexts of tool use are impossible, for example, because of the physics of a
tool. One could not, no matter how inventive one was, use a fork made entirely of
plastic to cut through a diamond. Thus, plastic forks will never by found in the
context of being used for diamond cutting. Furthermore, something may be
technically possible, but never arise because it presents contradictions in practice.
For example, it may be possible to use motor oil to build a bathtub, assuming
temperatures were low enough to carve a bathtub out of frozen motor oil.
However, in such a context to make the physics work, no one would want to use
this bathtub to actually take a bath it would be too cold.
If we were to take an example of this type of limitation on normal uses
from education, we could turn to an example by Clark. As he attempts to show
that one task can be performed by several technologies, he writes:

Many writers seem to suggest that these methods are somehow intrinsic to
a given medium. My argument is that the usual uses of a medium do not
limit the content it is capable of presenting. Computers can present
realistic, real-time documentary information [like a television], and
television can present semantically dense simulations [like a computer].
While it may be true that educational methods can be replicated by TV and
computers, it is not true that a book could ever present real-time documentary
information the delays of the publishing process forbid it. Thus, not all
methods can be replicated by in all media, although any one method may be
replicated by multiple media. While educational media might not have any one
intrinsic method, there may exist certain methods that any one media
intrinsically cannot have. Technologies cannot be used in an infinite variety of
contexts for an infinite variety of purposes.
So, with these two factors limiting context use in mind, how are we to find
normal contexts of use? First, it is important that normal contexts of use cannot
be found by consulting the intentions of a designer. Often technologies are used
in contexts (and for purposes) that are vastly different than what was intended
on being designed. Instead, to find normal uses, the best thing to do is look at
how people actually use the technology. Sometimes, there will be a fact of the
matter with regard to a technologys context of use. From my observations
around campus today, I found that bicycles are more often used in the purpose of
traveling to class than as a weapon; I also found that single seat bicycles were

much more common than multiple seats, etc. If I were comparing bicycles with
another technology, such observations would serve as my assumed context of
normal use.
Having said that, there should be added to caveats. First, if a fact of the
matter is indeed found concerning a normal context of use, it is important to note
that this may change abruptly. It may have been true 10 years ago that the
normal use of classroom computers was for drill-and-practice exercises.
Although this still may be a widely shared context of use, it can no longer be
assumed that this is the normal context of use. Second, with regard to
information technologies, it is doubtful that any one relatively stable normal
context of use can ever be stipulated. If a thing is broken down into data and
reconstituted into virtually, then the thing breaks free from any limitations
placed upon it by physics. If the idea of a palm-tree is digitized and placed in
a virtual world, it can be planted and flourish on an iceberg. The thing can be
refigured and reshaped to fit almost any context.
FURTHER ISSUES IN MEDIA COMPARISION
One of the major points raised by critics of media comparison research is
that economic differences between two media should be separated from
differences in learning. Clark writes, for example, Media may have important
influence on the cost or speed of learning but only the use of adequate

instructional methods will influence learning. The point of such assertions is to

combat the irrationality that sometimes may enter when decisions are made
about educational technology. Often, costly information technologies are
thought to be absolutely necessary, even though they have yet to demonstrate any
difference in learning outcomes. Thus, money and effort is wasted to secure
high-tech media, when other less-costly methods could easily fit the existing
need. This point is well taken.
As a conceptual point, however, the distinction between economic
differences and learning differences fails to hold up well to close analysis. It is
clearly the case that making an economic difference is making a difference in
learning. If a media makes a particular method financially possible, then this is
an important educational difference, even if that method could theoretically been
achieved in other ways. It would always be theoretically possible to visit Japan
and talk with Japanese people face to face. But this method of learning about
Japanese customs would be not be possible for most children under normal
economic conditions. Conversely, with information technology it would be
possible. Economics make all the learning difference in the world.
Another major point of the media comparison critics is that media should
not be confounded with method. They argue that a study is never just
comparing two media, but is always comparing the media and method.

Therefore, method should remain constant in order to isolate the variable of

media.
But is there a real conceptual distinction between media and method?
Media and method are both ways of doing things, and of accomplishing goals.
They may both arise in context of the question, How are you going to do that?
To this question, I may reply, Im going to do that on my computer, or I might
say, Im going do that by drill-and-practice. Or, we may reply to the question,
How are you going to do that? with a response that includes both method and
media: Im going to do drill and practice on my computer. Thus, a media or a
method may be offered in response to the question, How are you going to do
that?
One further doubt about a sharp separation between media and method is
raised when it is realized we ask ourselves what we mean by media. Media are
thought of as vehicles for delivery of something, usually some sort of
communication. Indeed, we tend to think of media as methods of delivery. We
can ask what is the method for having students do drill-and-practice? And reply
that the method is using the computer. Media are one class of methods; they are
methods that are used in the delivery and transportation of something. Thus,
media are one sort of method, just as a truck is one sort of motorized vehicle. It
would be a mistake to make a sharp distinction between a truck and a motorized

vehicle. Any one teaching and learning procedure will actually consist of
multiple methods methods of delivery (media), methods of attention-holding
and motivating, methods of evaluation, etc.
CONCLUSIONS
This paper has raised several points that may be of help to those trying to
compare various media.
To compare any two things, one must find a comparative framework
built on similarity. The similarity often assumed for media comparison
studies is problematic in assuming similarities among the purposes,
aspects, and standards of comparison of different media. Purposes change
as new media is introduced, objective tests may be impotent to find
differences in things like mood between media, and the ability to
promote intended learning is too narrow an aspect.
Comparing two technologies is also to compare contexts of use. When
attempting to make general comparisons about media, the context of use
must be stipulated. Although it may be difficult to find normal uses of
technology, certain characteristics of technologies make it a theoretical
possibility. To find normal uses one must turn to observations of how
technologies are actually used, and not rely to the intentions of designers.

Analysis of media outside of how they are actually used in a classroom

(apart from the watchful instructions of designers) may be misleading.
Economic facilitation cannot be divorced from learning outcomes. To
make a method economically feasible is to affect learning.
Media cannot be divorced from method media are methods of
delivery.
It is hoped that these points will be of help in developing a more useful
framework for media comparison analysis.