496 July-August 2003 Family Medicine

Development and Content Validation of Family Practice

Residency Recruitment Questionnaires
Lorraine Silver Wallace, PhD; Gregory H. Blake, MD, MPH;
Jon S. Parham, DO, MPH; Ruth E. Baldridge, MD

Background and Objectives: The residency recruitment process involves a substantial time and financial
commitment on the part of medical students and residency programs. This paper describes the develop-
ment and content validation process of two written questionnaires designed to assess the application and
interview process at our family practice residency program. Methods: Two written questionnaires were
developed after completion of a literature review and from areas deemed important by our academic
faculty. Drafts of each questionnaire were sent to nine jurors to assess content validity. Content reviewers
provided both a qualitative and a quantitative assessment of each questionnaire. Results: The inclusion of
both open- and closed-ended questions/items was deemed necessary and appropriate by the panel of
content jurors. Assessing faculty, residents, curriculum, program’s reputation, geographic location, and
spouse/family influence were considered the most important factors to include on the questionnaires when
assessing a family practice residency program. Conclusions: With increasing pressure to fill positions
across many family practice residency programs, it is important for faculty involved in the recruitment
process to recognize that both factors within and out of their control contribute to the selection process.

(Fam Med 2003;35(7):496-8.)

Fourth-year medical students face the difficult task of
selecting a medical specialty consistent with their per-
sonal and professional goals. After deciding on a medi-
cal specialty, selecting a residency program is the next
crucial step. To date, there have been a limited number
of studies published that assess specific reasons for se-
lecting a particular residency program. A national sur-
vey of first-year residents found that program location,
reputation, faculty, and current residents were the most
important factors in the selection process.1 Similarly,
interpersonal issues, education, location, and program
reputation were the most influential factors in select-
ing an internal medicine residency program.2
Recruitment of medical students for residency pro-
grams involves a substantial time and financial com-
mitment. Department chairs, residency program direc-
tors, and faculty have a vested interest in recruiting the
best-suited medical students for their programs. With
increasing pressure to fill family practice residency
positions,3 it is important for those involved in the re-
cruitment process to recognize that factors within and
out of their control contribute to the selection process.
For these reasons, we designed two questionnaires to
better understand the application and interview process.
This article describes the process used to assure con-
tent validity of these questionnaires.
Methods
Collecting data with established validity and good
psychometric qualities enables those reviewing the data
to do so with confidence. Content validity refers to the
extent to which an instrument reflects the specific in-
tended content.4 Recently, McKenzie and colleagues5
described four specific phases in establishing content
validity of a questionnaire. These steps include (1) cre-
ating an initial draft of the questionnaire, (2) establish-
ing/selecting a jury/panel of reviewers to evaluate the
questionnaire, (3) having jurors conduct a qualitative
review of the questionnaire, and (4) having jurors con-
duct a quantitative review of the questionnaire.

Phase I: Creating an Initial Draft of the Instruments

Initial drafts of the questionnaires were developed
From th e Department of Family M edicin e, Univ ersity of Tennessee, based on three distinct stages as outlined by Lynn.5
Knoxville. These include identifying the domain of content, item
Residency Education
generation, and questionnaire construction. The first
questionnaire assessed perceptions of applicants inter-
viewing, but not matching with, our residency program.
The second questionnaire assessed views of incoming
interns (PGY-1) to our residency program.
Content domain was identified after a thorough lit-
erature review and areas deemed important by the study
authors. The following content domains were estab-
lished: program personnel, academic factors, program
facilities, geographic location, personal factors, and
community considerations. Next, items were generated
to assess each specific content domain examined. Third,
questionnaires were constructed to include instructions
for completion and item placement.
Phase II: Selecting a Panel of Reviewers
The Standards for Educational and Psychological
Testing6 emphasizes the necessity of relevant training,
experience, and qualifications in selecting content ex-
perts to review instruments. We identified nine content
jurors and invited them to serve as panelists.
The following individuals composed the content ju-
ror panel: two fourth-year medical students, two fam-
ily practice chief residents, three physician (MD) fac-
ulty members, and two nonphysician (PhD) faculty
members. The medical students and family practice
chief residents were selected based on their familiarity
with the residency selection interview process. The MD
faculty members selected had extensive experience with
recruitment procedures of family practice residency
programs and had been employed at various residency
programs throughout the United States. Both PhD fac-
ulty members had extensive experience with instrument
design and question construction, as well as being fa-
miliar with the family practice residency recruitment
process.
Phase III: Qualitative Assessment
Content jurors were mailed a packet, which included
both questionnaires and instructions to assess content
and organization. Table 1 shows an outline of qualita-
tive assessment criteria. All of the jurors’ comments
were reviewed, and appropriate changes were made to
improve the overall quality of the questionnaires.
Phase IV: Quantitative Assessment
After completion of revisions of the qualitative re-
views, a quantitative assessment packet was mailed to
each juror. Jurors were asked to rate the appropriate-
ness of each item/question by stating if each item was
“essential,” “useful but not essential,” or “not neces-
sary.”7 Table 2 presents an outline of the quantitative
assessment items.
After receiving each juror’s ratings, values were en-
tered into a spreadsheet using the Statistical Package
for the Social Sciences (SPSS+) for Windows ® Version
Vol. 35, No. 7 497
Table 1
Qualitative Review Criteria Provided to Jurors
Please provide feedback regarding the directions, instrument items, and
overall ins trument. Please feel free to write your comments on the
instrument(s) or use additional paper as needed.
Directions
• Are the directions concise? If no, please explain.
• Are the directions clear? If no, please explain.
• Are the directions complete? If no, please explain.
Instrument items
• Are the items appropriate? If no, please explain.
• Are the items clear? If no, please explain.
• Would you revise any item(s)? If yes, please explain.
• Do you recommend deleting an item(s)? If yes, please explain.
• Do you recommend adding an item(s)? If yes, please explain.
• Other comments?
11.5. The content validity ratio (CVR) was calculated
for each item/question based on the formula developed
by Lawsche:8
CVR=n e-N/2
N/2
where: n e= number of panelists (jurors) indicating “es-
sential” and N= total number of panelists (jurors).
Calculated CVRs were then compared to the levels
required for statistical significance.8 A minimum CVR
value of .75 was necessary for statistical significance
at P<.05 based on nine panelists (jurors).
Results
Table 3 highlights significant questions/items on both
questionnaires. Open-e nded items (eg, program
Table 2
Example of Items: Quantitative Assessment
(Questionnaire One)
Appropriateness of Each Item/Question Rating*
During your interview at the ___ Family Practice
Residency Program, what did you perceived as
our strengths? 1 2 3
On the following scale, please rate the ___ Family
Practice Residency Program with the program where
you matched.
Faculty 1 2 3
Residents 1 2 3
Geographic location 1 2 3
* 1=essential, 2=useful but not essential, 3=not necessary
498 July-August 2003 Family Medicine

Discussion
We described the steps necessary to develop and
Table 3 “content validate” written questionnaires to evaluate
the residency recruitment process. Although these ques-
Content Validity Ratios (CVRs) tionnaires were developed and content validated for our
for Instrument Questions/Items individual family practice residency program, many of
the items can be used in other residency programs.
Questionnaire One* Questionnaire Two** For many fourth-year medical students, the applica-
Question/Item CVR Question/Item CVR tion and interview process constitute the very first ex-
perience they have with an individual family practice
Open-ended Open-ended
Program strengths 1.00✝✝ Application process 1.00✝✝ residency program. To improve recruitment and reten-
Program weaknesses 1.00✝✝ Interview process 1.00✝✝ tion of the best residents possible, it is important that
Factors affecting ranking .78 ✝ family practice residency programs continually exam-
Closed-ended Closed-ended ine the overall application and interview process. The
Faculty 1.00✝✝ Faculty 1.00✝✝ importance of using content-valid data collection tools
Residents 1.00✝✝ Residents 1.00✝✝ is imperative to evaluate and, therefore, identify pro-
Other 1.00✝✝ Other 1.00✝✝
Hospital facilities .78 ✝ Curriculum .78 ✝ gram strengths and areas for improvement. Important
FP clinic facilities .78 ✝ Experience as med student .78 ✝ improvements can be planned and made based on in-
Curriculum .78 ✝ Geographic location .78 ✝ put from fourth-year medical students interviewing at
Program’s reputation .78 ✝ Program’s reputation .78 ✝
Geographic location .78 ✝ Spouse/family .78 ✝ a particular family practice residency program.
Spouse/family .78 ✝ Hospital facilities .56
Department facilities .56 FP clinic facilities .56 Acknowledgment: This project was supported by the Department of Family
Experience as med student .56 Curriculum .56 Medicine, University of Tennessee, Knoxville.
Recreational opportunities .56 Cultural opportunities .33
Intensity of program .33 Intensity of program .33 Corresponding Author: Address correspondence to Dr Silver Wallace, Uni-
Cultural opportunities .11 Recreational opportunities .11 versity of Tennessee, Department of Family Medicine, 1924 Alcoa High-
Religious opportunities .11 Religious opportunities .11 way, U-67, Knoxville, TN 3792 0. 865 -544-9352. Fax: 865-544-653 2
lwallace@mc.utmck.edu.
* Distributed to medical students who interviewed but did not match with
our program
** Distributed to incoming interns at our program
REFERENCES
✝ Eight of nine jurors considered this to be an “essential” question/item
✝✝ All jurors (n=9) considered this to be an “essential” question/item 1. Mayeaux EJ, Arnold J. Why first-year family practice residents choose
their residency programs. Fam Med 1993;25(4):253-6.
Italicized items were not statistically significant and were eliminated from 2. Flynn TC, Gerrity MS, Berkowitz LR. What do applicants look for
the final instruments. when selecting internal medicine residency programs? A comparison
of rating scale and open-ended responses. J Gen Intern Med 1993;8:249-
54.
3. Natio nal Res iden t M atching Pro g ram. Selected d ata tab les .
www.nrmp.org/res_match/data_tables.html. Accessed September 12,
2002.
4. McKenzie JF, Wood ML, Kotecki JE, et al. Establishing content valid-
strengths and weaknesses, application and interview ity: using q ualitative and quantitativ e steps. Am J Health Behav
process) were considered to be very important by ju- 1999;23:311-8.
rors on both questionnaires. The jurors deemed nine of 5. Lynn MR. Determination and quantification of content validity. Nurs
Res 1986;35:382-5.
the 16 closed-ended items on Questionnaire One sig- 6. American Educational Research Association, American Psychological
nificant. Half (n=8) of the 16 closed-ended questions Association and National Council on Measurement in Education. Stan-
on Questionnaire Two were considered significant by dards for educational and psychological testing. Washington , DC:
American Psychological Association, 2002.
the jurors. These questions are important to assess when 7. Veneziano L, Hooper J. A method for quantifying content validity of
evaluating the application/interview process at any resi- health-related questionnaires. Am J Health Behav 1997;21:67-70.
dency program. (Final instruments are available from 8. Lawshe CH. A quantitative approach to content validity. Personnel Psy-
chology 1975;28:563-75.
the authors on request.)