This article was downloaded by: [University of Cambridge]

On: 04 January 2015, At: 09:06

Publisher: Routledge
Informa Ltd Registered in England and Wales Registered Number: 1072954
Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

The Reference Librarian

Publication details, including instructions for authors and
subscription information:
http://www.tandfonline.com/loi/wref20

Evaluating Reference Services and

Reference Personnel:
a
Charles A. Bunge
a
Professor, School of Library and Information Studies,
University of Wisconsin-Madison, Madison, WI, 53706
Published online: 26 Oct 2008.

To cite this article: Charles A. Bunge (1994) Evaluating Reference Services and Reference
Personnel:, The Reference Librarian, 20:43, 195-207, DOI: 10.1300/J120v20n43_16

To link to this article: http://dx.doi.org/10.1300/J120v20n43_16

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information
(the “Content”) contained in the publications on our platform. However, Taylor
& Francis, our agents, and our licensors make no representations or warranties
whatsoever as to the accuracy, completeness, or suitability for any purpose of the
Content. Any opinions and views expressed in this publication are the opinions and
views of the authors, and are not the views of or endorsed by Taylor & Francis. The
accuracy of the Content should not be relied upon and should be independently
verified with primary sources of information. Taylor and Francis shall not be liable
for any losses, actions, claims, proceedings, demands, costs, expenses, damages,
and other liabilities whatsoever or howsoever caused arising directly or indirectly in
connection with, in relation to or arising out of the use of the Content.
 This article may be used for research, teaching, and private study purposes. Any
substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing,
systematic supply, or distribution in any form to anyone is expressly forbidden.
Terms & Conditions of access and use can be found at http://www.tandfonline.com/
page/terms-and-conditions
Downloaded by [University of Cambridge] at 09:06 04 January 2015
 PART VI:
EVALUATION
Downloaded by [University of Cambridge] at 09:06 04 January 2015

Evaluating Reference Services

and Reference Personnel:
Questions and Answers
from the Literature
Charles A. Bunge

SUMMARY. There is a large lilerature on the evaluation of refer-

ence services and reference personnel. This bibliographic essay dis-
cusses a wide variety of items from that literature, organized around
a series of questions that might be considered a model for the refer-
ence evaluation process.

Reference Librarians and reference managers have available to

them a wide range of literature that can be helpful in evaluating
their services and their own performance. This article will outline a

Charles A. Bunge is Professor, School of Library and Information Studies,

University of Wisconsin-Madison, 600 N. Park S t , Madison, WI 53706.
[Haw& cc-indexing enuy mtel: "Evaluating Rcfcrcncc Services md Rcfaencc Personnel: Qucs-
lions and Answus horn h e Litcraturc." Bunge. Charks A. Co-published simulramously in The Refer-
ence Librrmri~(The H a w h Ross. Inc.) No. 43. 1994, pp. 195-207; and: ReferenceServias Planning
i n d e 90s (cd: Gail Z. Eckwright, and L a i M. K c e m ) The Hawath Rcss, Inc.,. 1994. pp. 195-207.
Multiple copies of this ar(iclc1chapw may be purchased from llu Hawonh Dallmcnt Dclivwy Cc&r
[I-BOO-3-HAWORTH; a.m. 9:00 p.m. (EST)].
- 500
O 1994 by The Haworth Press, hc.All rights reserved. 195
 196 REFERENCE SERVICES PLANNING M THE 90s

model or a series of steps for the evaluation process, citing literature

that will be helpful at each step. While the emphasis will be on
literature from the past decade or so, the literature reviews that are
cited will provide access to earlier literature, as well. The first part
of the article will discuss the literature of reference evaluation,
especially evaluation of reference question answering. This will be
followed by a briefer discussion of the literature on evaluation of
reference personnel.
Downloaded by [University of Cambridge] at 09:06 04 January 2015

OVERVIEWS AND WTERATURE REVIEWS

A number of authors have provided overviews of the literature of
reference evaluation, some with emphasis on implications for refer-
ence practice and others with emphasis on using the literature to
conduct evaluation locally. Louise Green's 1988 &clel is fre-
quently cited as an accessible and useful discussion of designing
and implementing reference evaluation. A more recent survey of
the literature by ~ u l t mentions
s~ current developments and issues to
be considered in developing a reference evaluation program. Recent
editions of two reference textbooks contain overview chapters on
reference evaluation that discuss various issues and methods and
include references to further reading3, PP. 4, pp. 223-40Baker
and Lancaster's standard text also has a useful chapter on evalua-
~ .229-71 as does Lancaster's
tion of reference question a n s ~ e r i n g ,PP.
smaller, practice oriented work.6*P P 108-26 crews7 and powel18
provide useful discussions of the research literature on reference
evaluation, including comprehensive coverage of studies from sev-
eral decades.
The 1980s and 1990s have seen several collections of articles or
papers that provide useful ideas and guidance for evaluation of
reference senices. A 1984 issue of The Reference ~ i b r a r i a nis~
dedicated to reference evaluation and contains a variety of helpful
articles on issues, methods, and specific shdies. A more recent
issue of the same journal uses the context of the current emphasis
on outcomes assessment as a framework for discussing a variety of
aspects of reference evaluation.1° The 1990 AUerton Park Institute
of the University of Illinois Graduate School of Library and In-
formation Science was on the subject of evaluation of public ser-
 Evaluation 197

vices and public services personnel. The published proceedings of

this conference contain overview and conceptual papers, discus-
sions of a variety of ap roaches and methods, and reports of specif-
ic studies and projects.G
MODELS AND A MODEL
Literature reviews and collections of articles can provide useful
Downloaded by [University of Cambridge] at 09:06 04 January 2015

access to ideas, techniques, and findings. However, the resulting

welter of encouragements, cautions, examples, and suggestions can
be overwhelming, without some way of sorting them for step-by-
step planning and application. Luckily, the literature contains some
pieces that are useful in this regard, as well. In fact, such an orga-
nizational framework is probably one of the aspects of Green's
article1 that causes it to be cited so frequently. Likewise, ~ancastefl
has a useful overview chapter on evaluation of library services in
general, giving different ways to look at the process and its compo-
nents. Linda Olson proposes a model for evaluating reference ser-
vices that sets out objectives to be accomplished b the evaluation
1
process and strategies for meeting the objectives.' Cronin applies
principles of performance measurement to the evaluation of public
services, using a program planning model.13 Robbins-Carter (now
Robbins) and Zweizig provide a useful framework for evaluation,
along with specific application to reference service.14
Most authors agree that evaluation of reference services should be
guided by the reference service's overall philosophy, mission, goals,
and objectives. For example, Cronin13 has this as an important com-
ponent of her model, and recent manuals from divisions of ALA
place measurement of outputs within a planning conte~t.'~. l6 In a
1984 article, I tried to provide practical suggestions for planning and
arriving at goals and objectives for the reference department.17
Robbins' and Zweizig's process consists of a series of steps that
are placed within the context of a planning process. The steps begin
with deciding on the specific target area for evaluation and proceed
through setting the target level to be aimed at, gathering data, ana-
lyzing and interpreting the data, and taking action based on the
results. Robbins and Zweizig make the point that the evaluation
process should not be one that asks "How good are we?" but rather
 198 REFERENCE SERVICES PLANNnvG LN THE 90s

one that attempts to answer the question, "Are we there yet?" I

have found their model very useful for conducting reference evalu-
ation and have adapted it to create a process consisting of answering'
a series of questions:
1. What do we want to know about?
2. Where do we want to be?
3. How will we know if we are getting there?
4. How close are we?
Downloaded by [University of Cambridge] at 09:06 04 January 2015

5. So what?
6. What next?
The next few sections of this article will elaborate briefly on these
questions and will mention items in the literature that can be helpful
in answering them.

THE TARGET
The question, "What do we want to know about?" reminds the
reference librarian or manager that one cannot usefully evaluate all
aspects of reference service at the same time. The most effective
evaluation focuses on a specific aspect that is of particular interest
or concern. Such interest or concern can arise from a variety of
sources. For example, the reference staff may become aware of
goals or objectives for which accomplishment seems wide of the
mark; patron comments or complaints might indicate target areas
for investigation; previous or ongoing measurement or evaluation
efforts might be examined to identify areas for further attention.
Certain very important aspects (e.g., answering reference questions)
may be recurring target areas for evaluation.
Having chosen a target area, the next step is to ask, "Where do
we want to be?" in that target area. Toward what objective or
standard are we aiming? What criteria will be used to determine
success at achieving the target? Answers to these questions can
come from a variety of sources, including published standards,
comparisons with achievements in other libraries, expectations of
library staff and administrators, user expectations, and the desire for
improvement in the current level of achievement.
 Evaluation 199

Reference librarians are well aware that our field is not well sup-
plied with useful standards. The Reference and Adult Services Divi-
sion of ALA has published standards in the form of guidelines that
cover various aspects of reference and information services in gener-
al terms.18 Some state library agencies have published standards,
especially for public and school libraries, that contain targets for
reference services. Whitlach's brief discussion of the use of stan-
dards makes the point that different groups involved in the evaluation
process will use different types of standards.1g Mary Goulding pro-
Downloaded by [University of Cambridge] at 09:06 04 January 2015

vides an interesting discussion of the use of minimum standards as a

first step in reference evaluation in a library system.20
Published studies and statistics can be used to establish norms or
to set reasonable targets for achievement locally. Reviews such as
those by crews7 and ow ell^ that suinmarize findings of numerous
studies can be quite useful in this regard. The results of nationwide
use of ALA/PLA's output measures process, including reference fill
rates, are published and can be used in this way, as well.2' The
Wisconsin-Ohio Reference Evaluation Program has produced ques-
tion-answering success rates from a large number of academic and
public libraries that can be used to guide the establishment of local
targets.Z2
Because one desired outcome of evaluation is usually improve-
ment in service, establishing the current status or level of accom-
plishment can be an important part of answering the question,
"Where do we want to be?" In fact, an important purpose of the
"first round" in an ongoing evaluation process can be to establish
benchmarks that can be used in establishing objectives or targets for
the future.

GATHERING DATA

Deciding the target area and specifymg the target level lead to the
third question in the model, "How will we know if we are getting
there?" (i.e., approaching the target). Answering this question is a
matter of determining the data to be gathered, sources for the data,
methods of gathering data, and related matters. Helpful information
on these matters can be found in a very wide variety of sources, .
including basic research methods texts, the overview articles on
 200 REFERENCE SERVICES PLANNING IN THE 90s

reference evaluation mentioned above, articles on specific issues

and techniques, and manuals that suggest strategies and provide
sample applications and forms.
Sometimes answering the "How will we know?" question stans
with definitions of terms and units of measurement. The field of
reference services has come a long way in this regard, and tools like
the ANSI standards can be very helpful in this regard.23
Von Seggern, using work of the ALAIRASD Evaluation of Ref-
Downloaded by [University of Cambridge] at 09:06 04 January 2015

erence and Adult Services Committee, has provided a useful anno-

tated bibliography of publications, with an emphasis on methodolo-
gy and data-gathering i n s t ~ m e n t s Whitlach
.~~ discusses various
strategies for collecting data in her 1992 article.lg Zweizig provides
a useful discussion of how to identify or design the kinds of mea-
sures that are needed for effective reference evaluation, including
criteria for identifying useful measures.25 Van House and her col-
leagues have suggested strategies for measuring the output of refer-
ence services in public and academic libraries, including sample
data-gathering fonns.15. PP. 65-71; 16. QP. 95-108. 156-61 Librarians in
two library systems in Illinois have produced manuals for gathering
information on various aspects of reference services, including
sample forms.26.27 The Medical Library Association has published
a "DocKit" that contains forms that have been found useful for
measuring elements of information services.28At the 1993 Midwin-
ter meeting of ALA the Evaluation of Reference and Adult Services
Committee of ALNRASD moved into the final stages of publish-
ing a "reference assessment manual" that will provide reference
librarians, managers, and researchers with access to a wide range of
evaluation instruments for measuring reference service effective-
ness. Progress on this project can be followed by checking 1993 or
1994 issues of RASD Update.
Aaicles on specific issues and techniques abound in the litera-
ture. One example of such an issue is the research orientation or
type of data gathering method to be used, such as qualitative versus
quantitative approaches. Westbrook provides a guide to the use of
qualitative evaluation methods.29 Another issue is the choice of
sources for data for evaluating question-answering effectiveness.
As sources of such data, both librarians and patrons have strengths
 Evaluadon 201

and weaknesses, as I have discussed in a aper that describes a

project that gathers data from both sources.!2
One problem with reference evaluation is that reference service
does not automatically leave records or evidence that can be used in
measurement, as do other aspects of library operations. This means
that the data-gathering process can be cumbersome and can intrude
on the service that is being evaluated. In such situations sampling is
an attractive strategy, but sampling reference queries has h own
complexities. A number of articles in the literature discuss methods
Downloaded by [University of Cambridge] at 09:06 04 January 2015

for determining appropriate sample size, the usefulness of cluster

sam ling of reference queries, and other sampling strategies.15*
pp. lg21; 30, pp. 104:18; 31; 32; 33; 34.pp. 31-32
Answering the fourth question, "How close are 'we?" involves
actually usini the methods'and techniques chosen to gather data and
- - the data to compare real accomplishment with the
then analyzing
target level or standard. This is largely a matter of following the
chosen methods with care and rigor, making appropriate adjust-
ments as the data-gathering process plays out, &d being cateful not,
to generalize from the data beyond their ability to suppolt such
generalization (especially if compromises, e.g., in sample size, be-
came necessary g the process developed). The use of computers in
gathering and analyzing data can make the process easier for all
involved. My 1991 paper discusses an example of an evaluation
project that uses computer-scannable data-gathering forms and
computer analysis of resulting data22

WHAT DOES IT ALL MEAN?

So you have lots of data and it says that some aspect of your
reference service is not yet at the target level. How can you turn this
information into a strategy for improving your service? This brings
up the "So what?" question. Answering this question involves
identifying factors in the situation that might be related to your level
of accomplishment. If one considers reference service to be a pro-
cess wherein inputs are transformed into desirable outputs, one can
look at both input factors and process factors for potential answers
to the "So what?" question. Reference librarians h o w that a poor-
ly answered reference question might be explained by a lack of
 202 REFERENCE SERVICES PL4NMNG IN THE 90s

materials (input), a faulty reference interview (process), or a variety

of other factors (often acting in combination). Recognizing such
potential factors in advance and gathering data on them in the
evaluation process can make the interpretation phase easier, but the
search for potential factors requires an open mind and detailed
knowledge of reference resources and processes. Articles that dis-
cuss the findings and applications of past evaluation studies can be
helpful in this regard. For example, Lancaster lists and discusses
factors that might affect question-answering success.35 The articles
Downloaded by [University of Cambridge] at 09:06 04 January 2015

by Crews7 and Powell8 discuss factors that have been studied in

relation to reference effectiveness. Dyson reports on an interesting
group of studies that show the effect of reference librarian behav-
iors on question-answering su~cess.3~
Having gained some ideas about what might be causing your
service to fall short of your targets, what can or should you do about
it? This "What next?" question is crucial to turning evaluation into
improvement. The rather obvious answer to the "What next?"
question is to make some changes in the identified factors or vari-
ables. In this day of limited or shrinking resources, the desire to
increase inputs (e.g., information sources, staff time) might have to
be addressed through resource reallocation. "Doing more with
less" can bring close scrutiny of process factors, so that efficiency
is at its optimum. In any case, the changes made in answer to the
"What next?" question become the target area for future evaluation
efforts, underscoring the cyclical nature of the planning and evalua-
tion process.

REFERENCE PERSONNEL
The evaluation of the personnel component of reference services
can be viewed as an application of the process outlined above.
Surely, each of the six questions mentioned should be asked in the
design and implementation of a system for evaluating reference
staff. However, the literature on this specific topic is large and
distinct enough to merit its discussion in the next few paragraphs.
Rubin discusses performance evaluation of library staff members
within the broader context of human resources management in li-
braries, combining theory and practice of various systems and cit-
 Evaluation 203

ing useful literature from outside l i b r a r i a n ~ h iIn

~ .his
~ ~ 1991 article
he extends this discussion to the evaluation of reference personnel
in particular, including various evaluation techniques, recommen-
dations to managers on their use, and useful references to the litera-
t ~ r eWeech
. ~ ~ also provides a useful summary of the literature of
reference personnel evaluation and proposes a model for the pro-
cess.39
Performance evaluation based on accomplishment of goals and
Downloaded by [University of Cambridge] at 09:06 04 January 2015

objectives has been a frequently suggested approach. Cohen pro-

vides practical information on following this appr~ach."~ Adam
and ~ u d d and
~ l Vrncelette and pfisteP2 also discuss the use of goals
and objectives in reference personnel evaluation, though both ar-
ticles describe methods for moving from goals and objectives to
librarian behaviors that should be the focus for evaluation.
Many authors argue that behaviorally anchored approaches pro-
duce the most effective evaluation of reference personnel. Tyckoson
discusses methods for behavioral evaluation and provides a useful
checklist of desirable behaviors, along with a listing of positive and
negative indicators for those beha~iors."~ While Young covers vari-
ous methods of evaluating the performance of reference librarians,
he emphasizes behaviorally anchored approaches and cites exam-
ples of their use in the l i t e r a t ~ r eRubin
. ~ ~ includes a sam le behav-
iorally anchored scale from a public P
P P 126-2 ~ y s o n ~ ~
provides a checklist of "model reference behaviors" that could be
used in evaluating reference librarians.
Another approach to reference personnel evaluation that is receiv-
ing increasing attention is peer review. Kleiner discusses the applica-
tion of the peer review process in a university library setting, includ-
ing a usefui checklist for implementing the process &d references to
the literah~e?~ King and Mahmoodi describe an innovative and
interesting applicatiG of the peer review process in a public li-
b r a ~ yThe
. ~ ~process they discuss includes librarian self-evaluation
and an opportunity for librarians to assess the effect of various orga-
nizational factors on their ability to do their jobs. Their paper in-
cludes competency lists and copies of forms used in the process.
Perhaps the most promising process for evaluating reference staff
members combines peer evaluation and behaviorally anchored
scales. An article by Schwartz and ~ a k i describes n ~ ~ the develop-
 204 REFERENCE SERVICES PLANNLNG IN THE 90s

ment of such a scale and its use in peer evaluation of reference

librarians. The ARUOMS SPEC Kit on performance appraisal in
reference services48includes the form discussed by Schwartz and
Eakin, along with other behavioral and peer review forms.

CONCLUSION
Reference librarians and managers are increasingly well served
Downloaded by [University of Cambridge] at 09:06 04 January 2015

by the literature on the evaluation of reference services and refer-

ence personnel. However, this literature can be intimidating without
a way of organizing it and making it useful in specific situations.
Because reference librarians are so accustomed to answering ques-
tions, organizing this literature and the evaluation process around a
series of questions can make the design and implementation of
evaluation easier that it might otherwise be. Indeed, one might think
of the evaluation process as conducting a sort of extended reference
interview. If the main question is "Are we there yet?" the questions
in this reference interview are:
What do we want to know about?
Where do we want to be?
How will we know if we are getting thew?
How close are we?
So what?
What next?
And, as so often happens with our favorite patrons, the informing,
learning, and improving process never ends; it just keeps evolving
and challenging us to do our best at helping our patrons be their best.

REFERENCES
1. Green, Louise K. "Assessing the Effectiveness of Reference Services: A
Difficult but Necessary Process." Catholic Library World 59 (January-February
1988): 168-71.
2. Hults, Patricia. "Reference Evaluation: An Overview." The Reference
Librarian No. 38 (1992): 141-50.
 Evaluation 205

3. Allen. Bryce. "Evaluation of Reference Services." In Reference and In-

formation Services: An Infroduction, edited by Richard E. Bopp and Linda C.
Smith. Englewood, CO: Libraries Unlimited, 1991.
4. Katz. William A. "Evaluation of Reference Services and Librarians." In
his Infroduction to Refemnce Work.Vol. 11. New York: McGraw-Hill, 1992.
5. Baker, Sharon L.. and Lancaster, F. Wilfred. "Evaluation of Reference Ser-
vices: Question Answering." In their TheMeasurement and Evaluation of Libmry
Services. Arlington, VA: Information Resources Pr.. 1991.
6. Lancaster, F,W. If You Want to Evalunfe Your Libmry. Urbana-Champaign:
Downloaded by [University of Cambridge] at 09:06 04 January 2015

University of Illinois Graduate School of Library and Information Science. 1958.

7. Crews, Kenneth D. "The Accuracy of Reference Services: Variables for
Research and Implementation." Libmry and Infomafwn Science Research 10
(July 1988): 331-5.5.
8. Powell, Ronald R. "Reference Effectiveness: A Review of Research." Li-
brary and InJbrmationScience Research 6 (January-March 1984): 3-19.
9. Katz, Bill, and Fraley, Ruth, eds. "Evaluation of Reference Services." The
Reference Libmrian No. 11 (FalW~nter1984): entire issue.
10. Blandy, Susan G.. Martin, Lynne M., and Strife. Mary L., eds. "Assess-
ment and Accountability in Reference Work." The Reference Libmrim No. 38
(1992): entire issue.
11. Allen, Bryce. ed.Evaluation of Public Services and Public Services Per-
sonnel. Urbana-Champaign: University of Illinois Graduate School of Library and
Information Science, 1991.
12. Olson, Linda M. "Reference Service Evaluation in Medium-Sized Aca-
demic Libraries: A Model." Journal ofAcdemic Librarianship 9 (January 1984):
322-29.
13. Cronin. Mary J. Perjbnnance Measumment for Public Services in Academ-
ic and Research Libraries. Association of Research Libraries, Office of Manage-
ment Studies, 1985 (Occasional Paper no. 9).
14. Robbins-Carter. Jane, and Zweizig, Douglas L. "Are We There Yet?"
American Libraries 16 (October, December 1985): 624-27.780-84.
15. Van House, Nancy A. et al. Output Measures for Public Libraries. 2d ed.
Chicago: ALA, 1987.
16. Van House, Nancy A., Weil. Beth T.,and McClure, Charles R. Measuring
Academic Library Perfo m c e : A Pmctical Approach. Chicago: ALA, 1990.
17. Bunge, Charles A. "Planning, Goals, and Objectives for the Reference De-
partment." RQ 23 (Spring 1984): 306-15.
18. "Information Services for Information Consumers: Guidelines for Provid-
ers." RQ 30 (Winter 1990): 262-65.
19. Whitlach, Jo Bell. "Reference Services: Research Methodologies for As-
sessment and Accountability." The Reference Libra& No. 38 (1992): 9-19.
20. Goulding, M a q "Minimum Standards as a F M Step Toward Evaluation of
Reference Services in a Multitype System" In Evaluation of Public Services. . .
[ref. 11, pp. 103-231.
 206 REFERENCE SERVICES PLANNING RV THE 90s

21. Public Library Data Service. "Statistical Report." Chicago: PLAIALA,

1988- .
22. Bunge, Charles A. "Gathering and Using Patron and Librarian Perceptions
of Question-Answering Success." In Evaluation of Public Services . . . [ref. 11,
pp. 59-83].
23. American National Standards Institute. American National Sfandard for
Library and Information Sciences and Related Publishing Practices-Library Sta-
tistics. New York: ANSI, 1983.
24. Von Seggem, Marilyn. "Assessment of Reference Services." RQ 26 (Sum
mer 1987): 487-96.
Downloaded by [University of Cambridge] at 09:06 04 January 2015

25. Zweizig, Doglas L. "Tailoring Measures to Fit Your Service." The Refer-
ence Librarian No. 11 (Falmnter 1984): 53-61.
26. Reference Evaluation Manual for Public Libraries. Oak Lawn, IL: Subur-
ban Library System n.d. [1986].
27. Reference Evaluation Project. Wheeling. 1L: Reference Librarians
Association. North Suburban Library System 1991.
28. Marshall. J o ~ M G..~comp. Evaluarion h ~ r u m e n t s f oHealth
r Sciences Li-
braries. Chicago: Medical Library Association. 1990 (MLA DocKit #2).
29. Westbrook, Lynn. Qualitative Evaluation Methods for Reference Services:
An Intmductory Manual. Washington. DC: Office for Management Studies.
Association for Research Libraries. 1989.
30. Hemon, Peter, and McClure, Charles R. Evaluation and Library Decision
Making. Norwood, NJ: Ablex, 1990.
31. Maxstadt, John M. "A New Approach to Reference Statistics." College
and Researrh Libraries News 49 (February 1988): 85-88.
32. Edinger, Joyce andFalk, Stephen. "Statistical Sampling of Reference Desk
Inquiries." RQ 20 (Spring 1981): 265-68.
33. Halperin, Michael. "Cluster Sampling Reference Transactions." RQ 17
(Summer 1978): 328-32.
34. White, Howard D. "Measurement at the Reference Desk." Drexel Library
Quanerly 17 (Winter 1981): 3-35.
35. Lancaster, F. W. "Factors Influencing the Effectiveness of Question-An-
swering Service in Libraries." 7he Reference Librnrian No. 11 (FallMnter
1984): 95-108.
36. Dyson, Lillie S. "Improving Reference Services: A Maryland Training
Program Brings Positive Results." Public Libraries 31 (SeptemberlOctober
1992): 284-89.
37. Rubin, Richard E. "Performance Evaluation." In his Human Resoume
t Libraries: Theory and Practice. New York: Ned Schuman, 199 1.
M a n a g e ~ n in
38. Rubin. &chard. "Evaluation of Reference Personnel." In Evaluation of
Public Services. . . [ref. 11. pp. 147-651.
39. Weech. Terry L. "Who's Giving All Those Wrong Answers? Direct Ser-
vice and Reference Personnel Evaluation." The Reference Librarian No. 11 (Fall/
Winter 1984): 109-22.
 Evaluation 207

40. Cohen, Lucy R. "Conducting Performance Evaluations." Library T ~ n d s

38 (Summer 1989): 40-52.
41. Adam. Mignon S., and Judd, Blanche.. "Evaluating Reference Librarians:
Using Goal Analysis as a First Step." The Rderence Librariun No. 11 ( F a l m n -
ter 1984): 131-45.
42. Vincelette. Joyce P.. and Pfister, Fred C. "Improving Performance Ap-
praisal in .Libraries." Library and Informntion Science Research 6 (April-June
1984): 191-203.
43. Tyckoson, David "Wrong Questions, Wrong Answers: Behavioral vs.
Factual Evaluation of Reference Service." The Reference Libmrian No. 38
Downloaded by [University of Cambridge] at 09:06 04 January 2015

(1992): 151-73.
44. Young, William F. "Methods for Evaluating Reference Desk Perfor-
mance." RQ 25 (Fall 1985): 69-75.
45. Kleiner, Jane P. "Ensuring Quality Reference Desk Service: The Introduc-
tion of a Peer Process." RQ 30 (Spring 1991): 349-61.
46. King, Geraldine B..and Mahmoodi, Suzanne H. "Peer Performance Ap-
praisal of Reference Librarians in a Public Library." In Evaluation of Public Ser-
vices . . . [ref. 11. OD. 167-2031.
47. Sc'hwartz,'Ijfiane G., and Eakin. Dottie. "Reference Service Standards. Per-
formance Criteria, and Evaluation." Journal of Academic Libmricarhip 12
(March 1986): 4-8.
48. Performance Evaluation in Reference Services in ARL Libraries. Washing-
ton, DC: Office of Management Studies. Association of Research Libraries, 1987
(SPEC Kit 139).