Subjective versus Objective Relevance in Bibliographic Retrieval Systems

Author(s): Don R. Swanson

Source: The Library Quarterly: Information, Community, Policy , Oct., 1986, Vol. 56,
No. 4 (Oct., 1986), pp. 389-398
Published by: The University of Chicago Press

Stable URL: https://www.jstor.org/stable/4308045

REFERENCES
Linked references are available on JSTOR for this article:
https://www.jstor.org/stable/4308045?seq=1&cid=pdf-
reference#references_tab_contents
You may need to log in to JSTOR to access the linked references.

JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide
range of content in a trusted digital archive. We use information technology and tools to increase productivity and
facilitate new forms of scholarship. For more information about JSTOR, please contact support@jstor.org.

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at
https://about.jstor.org/terms

The University of Chicago Press is collaborating with JSTOR to digitize, preserve and extend
access to The Library Quarterly: Information, Community, Policy

This content downloaded from

176.83.9.79 on Tue, 27 Jun 2023 12:56:12 +00:00
All use subject to https://about.jstor.org/terms
 SUBJECTIVE VERSUS OBJECTIVE RELEVANCE IN
BIBLIOGRAPHIC RETRIEVAL SYSTEMS

Don R. Swanson'

The distinction between subjective and objective relevance of a document to a

request, although heretofore given little explicit recognition, is of importance to
information retrieval as well as to certain broader issues in librarianship. Objec-
tive relevance is crucial to the design and testing of bibliographic retrieval
systems, while subjective relevance is paramount in the operation and use of
such systems. The distinction provides a rationale for a trial-and-error mode of
interaction with libraries, online databases, and other information systems. The
concept of objective relevance introduced here is illuminated by earlier work on
logical relevance by Patrick Wilson and by William Cooper.

The Abortive 1953 Test of Two Indexing Systems

What is probably the first major information-retrieval experiment, initi-

ated in 1953, came to grief over the question of "relevance"-that is, the
question of which documents were considered relevant to the requests
put to the system. The retrieval performance of the uniterm system of
Documentation, Inc. (DI) was to be compared with that of the subject
heading system in use at the Armed Services Technical Information
Agency (ASTIA) Reference Center (ARC) [ 1, pp. 325-29]. From
among the requests routinely received by ARC, ninety-eight were se-
lected for testing on a collection of 15,000 documents indexed by both
methods. The staff of ARC, using their own subject-heading system,
retrieved 2,220 documents that they considered relevant. The staff of
DI, using the uniterm system, retrieved 1,560 documents that they

1. Graduate Library School, University of Chicago, 1100 East 57th Street, Chicago, Illinois
60637.

[Library Quarterly, vol. 56, no. 4, pp. 389-398]

X 1986 by The University of Chicago. All rights reserved.
0024-25 19/86/5604-0002$0 1.00

389

This content downloaded from

176.83.9.79 on Tue, 27 Jun 2023 12:56:12 +00:00
All use subject to https://about.jstor.org/terms
 390 THE LIBRARY QUARTERLY

thought were relevant. Only 580 documents were common to the two
retrieved groups. After each team had reviewed all of the retrieved
documents, they jointly agreed that 1,390 documents were relevant to
the ninety-eight requests, but there were an additional 1,577 documents
that were considered relevant by one team but irrelevant by the other, a
rather large arena of disagreement. Unaccountably, the requesters
themselves-the people who originated the ninety-eight requests-were
not consulted to determine which of the retrieved documents they
found relevant or satisfactory. The matter of "relevance," therefore, was
left firmly planted in midair. Nonetheless, the test was not without value;
the two teams' analysis of the reasons for various retrieval failures was
fruitful and led to certain improvements in the indexing systems.
I shall use this primordial experiment to illustrate briefly the distinc-
tion between objective and subjective relevance, terminology that is not
in common use. The rest of this paper will elaborate that distinction.
The ARC and DI teams analyzed and tried to reach agreement on what
I shall call the objective relevance of the retrieved documents to the
ninety-eight written requests. The extent of their disagreement was
massive, but no attempt was made to let the requesters resolve all
differences by their own (subjective) judgment of relevance-that is, a
judgment by fiat of whether the retrieved documents met the original
intent of the requesters, a judgment that would presumably have been
based at least in part on factors not necessarily observable in the written
requests.
Difficulties in defining and in establishing criteria for relevance thus
began at the beginning and have been a matter of dispute for more than
three decades. I believe that the subjective/objective distinction, in-
formed by the work of Karl Popper on objective knowledge [2], can
illuminate important issues in that dispute-for example, the issue of
whether objective relevance is either meaningful or useful.

Subjective Relevance

Unlike the case described above, the relevance to a request of any book,
journal article, document, or other piece of information (loosely re-
ferred to here as a "document") has often been defined by the response
of the requester. That is, whatever the requester says is relevant is taken
to be relevant; the requester is the final arbiter, it is argued, because an
information-retrieval system exists only to serve its users. Relevance so
defined is subjective; it is a mental experience. Bookstein indeed defines
relevance as a relationship between a document and a person; he rejects
as "not very useful" or even meaningless the idea of "relevance to a re-
quest" [3, pp. 269-70].

This content downloaded from

176.83.9.79 on Tue, 27 Jun 2023 12:56:12 +00:00
All use subject to https://about.jstor.org/terms
 SUBJECTIVE VERSUS OBJECTIVE RELEVANCE 391

Subjective relevance has also been called "pertinence," "utility," "use-

fulness," and "psychological relevance" by other writers, as I shall fur-
ther note.
Any particular instance of judging a document against a request is, by
the foregoing definition, immune to criticism; it is unarguable-one can
appeal only to the originator of the request as the sole legitimate author-
ity. Anyone who disagrees with the requester's judgment is necessarily
mistaken. But perhaps we should be suspicious of a definition that
declares as unarguable something that people argue about all the time-
that is, about whether some document is or is not relevant to a particular
request. That such disputes occur suggests that there must be another
kind of relevance-a kind that one can argue about, that is open to
criticism, and that therefore must have some validity or meaning inde-
pendent of the requester-thus, relevance that is objective rather than
subjective.

Objective Relevance

To express an information need in writing is to create a request that, so

to speak, can take on a life of its own-a request that in principle
becomes independent of its creator and therefore objective. Other peo-
ple can interpret and analyze a written request and evaluate responses to
it with or without consulting the requester. A written request can be
considered to inhabit the abstract world of objective knowledge, the
world of problems, theories, and other products of the human mind that
Popper calls "World 3" [2].
Once an information need is objectified as a written request, the
possibility arises that such a request is logically related to some docu-
ment. What is meant by "logically related" is left vague at this point but
will be examined further. Perhaps, for example, the document answers a
question posed in the request. That relationship is then a basis for saying
that the document is objectively relevant to the request. It is relevant
whether or not anyone notices that it is relevant. Because both the
request and the document are products of the human mind, their
relationship (for example, that of "answering a question")-their "rele-
vance" to each other-is also a product of the human mind, even though
it is in general an unintended product. Relevance in this sense, being a
link between a written request and a document, belongs to the world of
objective knowledge.
I do not mean to suggest that the reality of objective relevance, or
objective knowledge, or of World 3 for that matter, is either obvious,
testable, or provable. The reality of World 3 is postulated within a
philosophy of knowledge that one can adopt provisionally to see

This content downloaded from

176.83.9.79 on Tue, 27 Jun 2023 12:56:12 +00:00
All use subject to https://about.jstor.org/terms
392 THE LIBRARY QUARTERLY

whether it is helpful, whether it throws light on problems that otherwise

remain obscure. In an earlier paper I have tried to show that such an
epistemology is indeed rewarding in that it both inspires and illuminates
the idea that new knowledge can in principle be extracted from the body
of existing published knowledge [4]. The unintended logical connec-
tions that form the basis for such new knowledge can be seen as links of
objective relevance between documents, in the sense that one can speak
of one document's being relevant, connected, or logically related to
another.
The idea of objective relevance is perhaps clearest in the case of
requests that leave little room for subjective interpretation-for ex-
ample, requests that entail questions about th-e physical world. A docu-
ment that gives the vapor pressure of mercury at ten different tempera-
tures is objectively relevant to the question, How does the vapor pressure
of mercury depend on temperature? The requester or anyone else who
denied its relevance would be considered not so much to have a different
underlying need for information (though that may be so, of course) but
rather as simply being mistaken in his understanding of either the
document or the question as written or both. The issue is not what the
requester meant to ask but what the request itself actually said.
Any particular case of objective relevance, unlike that of subjective
relevance, is always arguable in principle. That is, people might disagree
about whether a given document is or is not relevant to some request.
Such disagreement does not refute, indeed it supports, the idea of
objective relevance, a relationship that might be perceived differently by
different people, none of whom perhaps fully grasps it. Indeed, it is
difficult to understand what else an argument of that kind could be
about if it is not about something that is presumed to exist indepen-
dently of the arguers. Objective relevance, in any particular instance,
must therefore meet standards of criticism in order to be accepted.

Logical Relevance: Wilson and Cooper

At least two dozen authors during the past thirty years have undertaken
to define, clarify, or analyze relevance. Studies of the concept of logical
relevance by Patrick Wilson and by William Cooper are especially valu-
able [5-7].
Relevance in general, according to Wilson, means essentially "re-
trieval-worthy." It is an intrinsically incomplete notion, for what makes a
document worth retrieving depends on the nature of the request and
the requester [5, pp. 16-18]. So understood, relevance is not a single
notion but many; it can be made specific and precise in a large number
of ways [6, p. 457].

This content downloaded from

176.83.9.79 on Tue, 27 Jun 2023 12:56:12 +00:00
All use subject to https://about.jstor.org/terms
 SUBJECTIVE VERSUS OBJECTIVE RELEVANCE 393

Wilson goes on to define four kinds of relevance: psychological, logi-

cal, evidential, and situational [6, p. 458]. These concepts can be used to
throw further light on the distinction I have drawn between subjective
and objective relevance.
Psychological relevance is related to how people actually use and are
affected by information, how their views change or fail to change conse-
quent on the receipt of information [6]. Expressing a relationship be-
tween information and a person, psychological relevance is an aspect of
subjective relevance.
What Wilson says of logical relevance applies also to what I mean by
objective relevance:

Logical relevance has to do with uses or effects [of information] only insofar as
they correspond to standards of criticism, only as they correspond to what
should happen or be done. We are all of us prepared to say that others have
failed to see the relevance of some piece of information, or that they have
thought relevant what actually was not. We are prepared to say the same of
ourselves; that, for instance, we had the information we needed all along but
failed to see its relevance. In such cases we presume the existence of a relation-
ship that holds whether or not it is noticed, one about whose presence or absence
we can be mistaken, and one that can be investigated by other than purely
psychological means. [6, p. 458]

That description, for Wilson, is a preamble to a more thorough develop-

ment and definition of the idea of logical relevance. He takes as his point
of departure a narrower concept of logical relevance in the framework
of a formalized inferential question-answering system developed by
William Cooper [7].
Cooper, recognizing that an information need is a psychological state
and so not directly observable, presumes that a person with an informa-
tion need can in principle create a linguistic representation of that need
as a basis for formulating a query. By so doing, any need for information
is thus objectified. Cooper goes on to develop a concept of logical
relevance (which is a form of objective relevance, though he does not call
it that) based on logical consequence. In his system, a statement is
relevant to a question if it is a member of a minimal set of premise
statements that logically entail an answer to the question. A relevant
document is any document containing at least one statement relevant to
the question. Relevance so defined does not appeal to the authority of
the requester, and its correctness in any specific instance is not a matter
of opinion; it is a matter of logical consequence.
Such an inferential system-a system based on strict logical conse-
quence-is quite limited in its applicability to practical information-
retrieval systems. Cooper attempts to surmount this limitation by pro-
posing, among other things, to take into account the requester's personal
memory, which is presumed to contain some of the members of any

This content downloaded from

176.83.9.79 on Tue, 27 Jun 2023 12:56:12 +00:00
All use subject to https://about.jstor.org/terms
394 THE LIBRARY QUARTERLY

given premise set. Then a relevant statement would not be required to

meet so strict a requirement as membership in a minimal external
premise set but only membership in a residual premise set-where the
requester's mind supplied the rest of the set, as well as the logical
reasoning necessary for identifying such a set [7, pp. 29-32].
In thus extending his original definition of logical relevance to include
what is in the requester's mind, Cooper attempts at least partially to cope
with subjective relevance in addition to objective relevance. Because in
general private knowledge and human creativity can be brought to bear
on the problem of identifying subjectively relevant documents, it seems
clear that the system itself cannot recognize such relevance without the
direct participation of the requester. In that sense the information-
retrieval process cannot be formalized and so is essentially incomplete.
Cooper has argued that, even in a more restrictive framework, where
relevance depends on membership in premise sets, one arrives at the
same destination-a conclusion that I think is correct, interesting, and
important: "Because no system has direct access to a user's mind, a
reference retrieval system could not, even in principle, ascertain
whether a set of its stored sentences constituted a residual minimal
premiss set or not.... It must be concluded that reference retrieval is
partial and incomplete by its very nature" [7, p. 32]. I have offered
previously a different but not unrelated argument about the limits of
information retrieval and reached a similar conclusion [4, pp. 113-15].
Wilson attempts to build on the foundation of Cooper's inferential
system and to overcome its limitations in two ways. First, he broadens
Cooper's narrowly logical relevance into what he calls "evidential rele-
vance." Evidential relevance is based not on logical consequence but on
the notion of degree of confirmation or probability of conclusions in
relation to a set of premises. Relevant information is that which
strengthens or weakens a case rather than proves a case; irrelevant
information does neither. The point is that an information-retrieval
system should help one arrive at conclusions even when conclusive
arguments cannot be obtained [6, p. 460].
One difficulty with Wilson's argument is the absence of any formal or
even widely accepted system of inductive or probabilistic or plausible
reasoning in support of the idea of degree of confirmation. Wilson
acknowledges such a difficulty but contends that we routinely though
informally operate with inductive concepts. Karl Popper argues persua-
sively against the idea of empirical inductive reasoning and suggests that
all of empirical science proceeds through conjectures (or hypotheses)
plus a systematic attempt to refute such conjectures through a process of
testing and deductive reasoning [8, pp. I 1- 158]. Tests of a theory may
be more or less severe, and so it is appropriate to speak of the degree to

This content downloaded from

176.83.9.79 on Tue, 27 Jun 2023 12:56:12 +00:00
All use subject to https://about.jstor.org/terms
 SUBJECTIVE VERSUS OBJECTIVE RELEVANCE 395

which a theory is corroborated by a test and of the weight of the evidence

in favor of a theory, concepts that Popper examines and develops in
some detail [8, pp. 233-55].
The more important point for the purposes of this paper is my
agreement with Wilson that an information-retrieval system should de-
liver evidence that can support an argument or a theory even if not
prove it. I adopted a similar goal previously [4], in taking as the central
problem of information retrieval the pulling together of all information
that bears on the critical state of a theory. At that time, I addressed also
the question of how multiple weak tests of a theory, if brought together
by an information-retrieval system, could be interpreted as a strong test
[4, p. 111]. In both cases I made use of concepts closely related in their
aims to Wilson's "evidential relevance," albeit apparently differing from
the latter in definition and in certain presuppositions about the philoso-
phy of science.
Wilson takes a second approach to extending Cooper's work by in-
troducing the idea of situational relevance-that is, relevance that takes
into account the requester's individual perspective or situation as he sees
it, insofar as that situation is of concern to his information need or
request. To avoid the intractable problem of subjectivity, Wilson, taking
a cue from Cooper's strategy of objectifying requests, proposes similarly
to objectify the requester's situation-that is, his concerns, interests, and
priorities-by replacing the situation with a description of the situation
[6, p. 461].
If I have correctly understood what Wilson proposes, however, I am
not persuaded that a person's concerns, interests, preferences, and other
situational parameters can in general be formalized, described, or taken
into account to a sufficient degree for the purpose of an information
search. I believe that the problem of accounting for or describing subjec-
tive relevance is essentially intractable. One should therefore design and
operate information-retrieval systems in such a way as to take into ac-
count that intractability.

Design versus Operation of Bibliographic Retrieval Systems

The purpose of an information-retrieval system is, ultimately, to serve its

users; its function is to identify, locate, and deliver books, documents,
journal articles, or similar chunks of recorded information in response
to requests of some kind. The implications of that purpose are somewhat
different for designing a system than for operating it.
In an operational system, subjective relevance to each request, as
assessed by the requester, is clearly an appropriate measure for judging

This content downloaded from

176.83.9.79 on Tue, 27 Jun 2023 12:56:12 +00:00
All use subject to https://about.jstor.org/terms
396 THE LIBRARY QUARTERLY

the success of any specific instance of responding t

earlier, Bookstein accepted such a measure as a definition of relevance
[3]. Cooper calls such a measure "utility" [7, pp. 35-36], Foskett names it
"4pertinence" [9], and Wilson "retrieval worthiness" [5, p. 18].
On the other hand, the design and testing of an information-retrieval
system must depend on objective relevance. Any design is tentative until
it is tested, and on the basis of testing the design can then be revised and
improved. The testing of an information system, insofar as it can in-
fluence the design of the system, must operate with objectified requests
and objective relevance of the responding documents. The requester
may evaluate or judge the degree to which any responding document
meets his (subjective) need, but if he is to explain, for the purpose of
aiding in the design of the system, why such a document succeeds or fails
in meeting that need, he can proceed only by trying to explain the
objective relevance of the document to a stated or written-that is, an
objectified-request. The logic of the situation, in other words, suggests
that the interaction between design and testing must be based on objec-
tive relationships between requests and documents.
The idea that there is a different role for subjective and objective
relevance is not altogether new; it has been noted by at least two
authors-first by Cooper:

Logical relevance is the most important single factor affecting utility so far as the
retrieval system designer is concerned. The system designer cannot do much to
ensure that only the most credible material will be retrieved; such a goal lies
largely beyond the state of the art at present. He cannot gauge the relative
importance to the user of different component statements either; in fact, in
reference retrieval systems the component statements are not even known from
the designer's point of view. Logical relevance is almost the only factor in utility
which the designer does know how to deal with very effectively at present. [7, p.
36]

Kemp criticizes Cooper for stating in an earlier paper that "subjective

utility" (which is identical with what Kemp, following Foskett, calls "per-
tinence") is the only or the principal valid means of determining the
value of information systems: "The point he makes is a valid one so far
as practical, operational information systems are concerned. . . . For
instances where experimental (and perhaps impractical) systems are
being tested, results based on impersonal, objective assessments are
necessary. Relevance [as distinct from pertinence]-based on a consen-
sus of opinion of several assessors-is that kind of a measure" [10, p. 37].
Capricious or arbitrary assessments of relevance, assessments that do not
meet standards of criticism, or that have no anchor in reality, are not
useful in either the design or testing of a system. They may, however,

This content downloaded from

176.83.9.79 on Tue, 27 Jun 2023 12:56:12 +00:00
All use subject to https://about.jstor.org/terms
 SUBJECTIVE VERSUS OBJECTIVE RELEVANCE 397

play an important role in the use of a system, use that is often character-
ized by an imaginative trial-and-error mode of operation.
The designers of an information-retrieval system create a product
with which users can then interact, rationally or irrationally, with or
without help. The operators of the system perform a service in provid-
ing assistance to those who want it. One can see in this distinction certain
broader implications for librarianship-whether, for example, its pres-
ent identity as exclusively a service profession should be accepted or
challenged. So far as research libraries are concerned, the central task of
librarianship is to develop effective routes of access to recorded knowl-
edge-to produce therefore a product or a structure, a bibliographic
apparatus, that users then can in principle explore on their own. In that
process of exploration, trial-and-error methods tend to dominate, for
the system itself cannot account for the user's situation, interests, con-
cerns, or perspective. Or, even if it could do so at any one stage of a
search, that situation or that perspective is likely to shift as the interac-
tion proceeds, and as (subjectively) relevant or interesting material is
found.
In an earlier paper, I developed more fully the idea of a trial-and-
error approach to information retrieval, an idea that would have been
clearer had I introduced at that time a distinction between subjective and
objective relevance [11]. Relevance in what I called "frame of reference
1" is clearly enough subjective, but topic-oriented relevance (frame of
reference 2) can be a mixture; it is largely objective, but not purely so
[1 1, pp. 139-421. For the most part, bibliographic retrieval systems are
topic oriented, and topics are presumed to be objectively defined and
applied. However, the question of whether a document is or is not about
some topic can depend on the observer's point of view, and so a subjec-
tive element may intrude into a topic-oriented relevance judgment. With
that qualification, my frames of reference 1 and 2 can be considered to
represent subjective and objective relevance, respectively.
The potential value of trial-and-error interactive searching is worth
stressing particularly in view of the widespread but mistaken idea that
online searching is interactive by definition and highly effective simply
because a very large bibliography on almost any topic under the sun
can be produced quickly. An online search is not necessarily either
significantly interactive or highly effective [4, p. 116]. Reference re-
trieval, as Cooper notes, is partial and incomplete by its very nature.
That deficit can be made up only by applying human imagination,
creativity, and ingenuity to the search process-qualities that can be
brought to bear only through sufficiently intensive and repeated interac-
tions with a retrieval system, whether online or not. These qualities,
moreover, in some sense signal the wide gulf that separates subjective

This content downloaded from

176.83.9.79 on Tue, 27 Jun 2023 12:56:12 +00:00
All use subject to https://about.jstor.org/terms
398 THE LIBRARY QUARTERLY

relevance from any formal description or definition of relevance and

from the more limited though more logical idea of objective relevance.

REFERENCES

1. Gull, C. D. "Seven Years of Work on the Organization of Materials in the Special

Library." American Documentation 7 (October 1956): 320-29.
2. Popper, Karl R. Objective Knowledge: An Evolutionary Approach. Oxford: Clarendon
Press, 1975.
3. Bookstein, Abraham. "Relevance."Journal of the American Societyfor Information Science
30 (September 1979): 269-73.
4. Swanson, Don R. "Undiscovered Public Knowledge." Library Quarterly 56 (April 1986):
103-18.
5. Wilson, Patrick. "Some Fundamental Concepts of Information Retrieval." Drexel Li-
brary Quarterly 14 (April 1978): 10-24.
6. Wilson, Patrick. "Situational Relevance." Information Storage and Retrieval 9 (August
1973): 457-71.
7. Cooper, William S. "A Definition of Relevance for Information Retrieval." Information
Storage and Retrieval 7 (June 1971): 19-37.
8. Popper, Karl R. Realism and the Aim of Science, edited by W. W. Bartley III. Totowa,
N.J.: Rowman & Littlefield, 1983.
9. Foskett, D. J. "A Note on the Concept of 'Relevance.' " Information Storage and Retnreval
8 (April 1972): 77-78.
10. Kemp, D. A. "Relevance, Pertinence, and Information System Development." Informa-
tion Storage and Retrieval 10 (February 1974): 37-47.
11. Swanson, Don R. "Information Retrieval as a Trial-and-Error Process." Library Quar-
terly 47 (April 1977): 128-48.

This content downloaded from

176.83.9.79 on Tue, 27 Jun 2023 12:56:12 +00:00
All use subject to https://about.jstor.org/terms