Download as pdf or txt
Download as pdf or txt
You are on page 1of 10

Evaluating Reflective Writing

Fostering and Evaluating Reflective Capacity


in Medical Education: Developing the REFLECT
Rubric for Assessing Reflective Writing
Hedy S. Wald, PhD, Jeffrey M. Borkan, MD, PhD, Julie Scott Taylor, MD, MSc,
David Anthony, MD, MSc, and Shmuel P. Reis, MD, MHPE

Abstract
Purpose determination of interrater reliability, reflection and learning and confirmatory
Reflective writing (RW) curriculum reevaluation and refinement, and learning. ICC ranged from 0.376 to
initiatives to promote reflective capacity redesign. Rubric iterations were applied 0.748 for datasets and rater
are proliferating within medical in successive development phases to combinations and was 0.632 for the final
education. The authors developed a new Warren Alpert Medical School of Brown REFLECT iteration analysis.
Downloaded from http://journals.lww.com/academicmedicine by BhDMf5ePHKav1zEoum1tQfN4a+kJLhEZgbsIHo4XMi0hCywCX1AWnYQp/IlQrHD3i3D0OdRyi7TvSFl4Cf3VC1y0abggQZXdtwnfKZBYtws= on 01/28/2021

evaluative tool that can be effectively University students’ 2009 and 2010 RW
applied to assess students’ reflective narratives with determination of Conclusions
levels and assist with the process of intraclass correlations (ICCs). The REFLECT is a rigorously developed,
providing individualized written feedback theory-informed analytic rubric,
to guide reflective capacity promotion. Results demonstrating adequate interrater
The final rubric, the Reflection Evaluation reliability, face validity, feasibility, and
Method for Learners’ Enhanced Competencies acceptability. The REFLECT rubric is a
Following a comprehensive search and Tool (REFLECT), consisted of four reflective analysis innovation supporting
analysis of the literature, the authors reflective capacity levels ranging from development of a reflective clinician via
developed an analytic rubric through habitual action to critical reflection, with formative assessment and enhanced
repeated iterative cycles of development, focused criteria for each level. The rubric crafting of faculty feedback to reflective
including empiric testing and also evaluated RW for transformative narratives.

Editor’s Note: Commentaries on this article appear complexity that is inherent to clinical example, Mann and colleagues19 define
on pages 5 and 8. practice, potentially influencing the reflective capacity as “critical analysis of
choice of how to act in “difficult or knowledge and experience to achieve
F ostering reflective capacity within
morally ambiguous circumstances.”12 In
this vein, the development of reflective
deeper meaning and understanding.”
Theoretical pillars of reflective capacity
medical education helps develop critical practice has been associated with include Schon’s20 progression from
thinking skills,1,2 inform clinical enhancing an individual’s character or knowing-in-action, to surprise,
reasoning,3 and enhance professionalism4 “virtue,” fostering a “habit of mind,”13 reflection-in-action (“thinking on our
among trainees. Reflection—the “dispositional tendency,”14 or “medical feet”21), experimentation, and, finally,
expertise-enhancing, metacognitive, tacit morality”15 with which to approach reflection-on-action (postexperience
process5,6 whereby personal experience clinical reasoning and ethical or values- reflection), and Boud and colleagues’22
informs practice7—is integral to core related16 dilemmas that may arise. It also emphasis on addressing feelings in the
professional practice competencies.8,9 helps in developing “phronesis”— reflective process. Moon23 introduces the
Development of reflective capacity has adaptive expertise or practical wisdom to component of meaning making to
been highlighted as necessary for effective guide professionally competent clinical reflection in learning, and Mezirow24
use of feedback in medical education10,11 practice.13,17 Failure to reflect on one’s links premise reflection with
and is an essential aspect of self-regulated own thinking processes, including critical transformative or confirmatory learning,
and lifelong learning.5,10 Reflection can examination of one’s assumptions, bringing additional depth and breadth to
guide practitioners as they encounter the beliefs, and conclusions, was recently reflection conceptualization. Mann and
described as a cognitive component of colleagues19 describe two overarching
“physician overconfidence,” a dimensions in models of reflection:
Please see the end of this article for information contributing cause of diagnostic error in iterative and vertical. The iterative
about the authors.
medicine.18 In line with this, research has dimension of reflection is one triggered
Correspondence should be addressed to Dr. Wald, offered promising new evidence of an by experience, producing a new
Warren Alpert Medical School of Brown University,
Department of Family Medicine, 111 Brewster St.,
association between analytical reflective understanding; the vertical dimension
Pawtucket, RI 02860; telephone: (781) 424-2711; reasoning and improved diagnostic combines surface (descriptive) and
fax: (866) 372-7918; e-mail: hedy_wald@brown.edu. accuracy in challenging cases.1 deeper (analytic) levels of reflection.
Acad Med. 2012;87:41–50.
First published online November 18, 2011 Definitions of reflective capacity abound, Reflection is not necessarily intuitive,
doi: 10.1097/ACM.0b013e31823b55fa though they generally include review, especially in students at initial stages of
Supplemental digital content for this article is interpretation, and understanding their medical careers. Thus, medical
available at http://links.lww.com/ACADMED/A68. experiences to guide future behavior. For educators strive to implement innovative

Academic Medicine, Vol. 87, No. 1 / January 2012 41


Evaluating Reflective Writing

educational methods to promote students’ levels of reflection and the assessment of level of reflection as an
development of reflective capacity early development of reflective skills within indicator of professional development of
in the training process. The use of RW pedagogy. Publications on the utility medical students and best teaching
reflective writing (RW) to facilitate of RW in medical education have been practices,35 we set out to design an
reflective practice is well documented.25–29 largely anecdotal or based on student empirically tested, concise, “user-
Curricula have included RW groups for self-report. Although some suggest friendly” evaluative paradigm stemming
students in clerkships and residencies, assessing students’ levels of reflection to from our review of existing qualitative
journaling, portfolios, video essays, and evaluate reflective learning outcomes,44,45 and quantitative measures and
what we have termed “interactive” RW— a recent comprehensive review concluded frameworks for reflective capacity.
integration of written feedback from that measurement of reflection is at an
faculty to foster learners’ development of early stage of development and that The Reflection Evaluation for Learners’
more sophisticated reflection skills.29 qualitative and exploratory research Enhanced Competencies Tool
Pedagogic goals of professional approaches are appropriate for achieving (REFLECT), a new rubric for evaluating
development, insights into the process of deeper understanding of this essential medical students’ levels of reflection and
patient care, and practitioner well-being construct.19 the development of those levels within
have been addressed through the small- RW pedagogy, is an innovative approach
group RW process.30 RW, a subset of There are significant limitations and to assessing reflection that includes
narrative medicine, cultivates self- challenges in applying available coding multiple fundamental domains of
awareness and builds narrative systems for analyzing written reflective reflection. In this report, we describe the
competence for clinical encounters journals to determine the extent and level development of the rubric, present
through the processes of attending, of reflection. Proposed criteria for reliability and validity data, and discuss
representing, and affiliating that are “grading” physiotherapy students’ the rubric’s application and potential use
shared between RW and clinical reflective journals,46 for example, lacked for enhancing the educational effects of
practice.25 RW embodies the clear explication,47 and a reliable reflective narratives in medical education.
“interpretative and narrative”31 qualities structured worksheet for assessing
of practical medical reasoning. Narrative reflection level48 focused on depth to the
Method and Results
competence and emotional self-reflective exclusion of breadth of reflection.47 Plack
ability, which may be cultivated through and colleagues7 applied a modified Preliminary stage: Literature and model
RW, can bolster resilience to emotionally Bloom’s taxonomy to determine review
challenging situations32 and promote achievement of higher-order thinking in The development of the REFLECT rubric
capability in challenging communication reflective journals, yet they only began in early 2008 with a comprehensive
encounters, such as breaking bad news.33 indirectly assessed reflection per se. analysis of the literature, including
Identification and coding of textual theoretical models of reflection, RW
Mentors who skillfully support and elements of journals for levels of pedagogy, elements of reflective practice,
challenge learners through noticing the reflection using Boud and colleagues’22 and existing assessment modalities in
reflective moment, making sense of the model was described as relatively difficult health professions education. By October
experience (including emotional and not achieving sufficiently reliable 2008, we concluded literature searches in
responses), tolerating uncertainty (or outcomes.44,46 Plack and colleagues47 the PubMed database for relevant articles
“messiness” of clinical practice at the broadened coding schema for reflective for the years 1995 to 2008 using key
“heart of professional expertise”20), and journals by including Schon’s,20 Boud words such as “reflection,” “reflective
using new insights5,34 are an essential and colleagues’,22 and Mezirow’s 24 practice,” reflective writing,” “reflection
component to developing reflective theoretical frameworks; however, the in medicine,” “reflection in medical
capacity. Their guided written feedback schema did not integrate criteria within education,” and “reflection in health
about reflective narratives can promote a reflective levels, and the authors professions education.” We then
more in-depth reflective process.35,36 At identified the need for further refinement conducted ongoing subsequent literature
the Warren Alpert Medical School (AMS) of some operational definitions. In searches until late 2010 (to inform the
of Brown University, students receive addition, our review of available criteria writing of this article) with “reflection,”
guided, individualized feedback about for assessing level of reflection revealed “reflective writing,” and “reflection
their RW from interdisciplinary that existing criteria did not include assessment” used as key words, though
faculty29,37–39 during the Doctoring Mezirow’s24 transformative or articles from 2008 to 2010 were not
course40,41 and family medicine confirmatory learning schemata49; in fact, included in the literature review for the
clerkship.42 Faculty use a rigorously we encountered a critique of his development of REFLECT.
developed tool (the Brown Educational reflection levels (as used in current
Guide to the Analysis of Narrative assessment formats) as inadequately We then used snowball technique to
[BEGAN]) to enhance the educational describing the process of reflective extend the literature search from
impact of their written feedback about thinking.46 Some recently published retrieved articles to other relevant
reflective narratives.43 rubrics for reflective narrative analysis are sources. The snowball technique for
limited either in scope50 –52 or in building sampling is a method whereby existing
The proliferation of RW curricula locally a validity argument.53 Lastly, the factorial study participants suggest, recruit, or
and internationally has created the need validity of at least one self-report reflection assist in recruiting future subjects from
for a valid, reliable evaluative tool that instrument has been questioned.54 In light among their acquaintances or contacts.55
can be effectively applied to assess of the increased interest in formal In this case, it refers to careful review of

42 Academic Medicine, Vol. 87, No. 1 / January 2012


Evaluating Reflective Writing

the bibliographies of articles found from seemed to be the best choice for the broad range of possible elements, we
database searches to detect other relevant assessment of reflective levels because reached consensus on five levels of
articles that may have been otherwise they are based on a theoretical framework reflection with associated criteria based
missed. From our review of the literature, and can be tailor-made for specific on the theories of Schon,20 Boud and
we identified four existing modalities of purposes. An instructional rubric colleagues,22 Moon,23 and Mezirow.24
reflection assessment: (1) scales (“paper delineates the various dimensions or This rubric included the following levels:
and pencil” forms with responses scored levels of an assessed construct, defining Level 1: Nonreflective: Habitual Action;
by respondents), (2) thematic coding benchmarks for each, and can yield Level 2: Nonreflective: Thoughtful
(qualitative analysis that codes themes in quantitative scores.57,58 The rubric Action; Level 3: Reflective; Level 4:
the narratives), (3) qualitative analysis format—used for both formative and Critically Reflective; and Level 5:
(more elaborate qualitative analysis summative purposes—may vary, though Transformative Learning. We developed
moving beyond themes into models), and common features include quality level criteria or dimensions for each level (e.g.,
(4) analytical instructional rubrics gradations on a continuum of strong to descriptive versus reflective stance,
(theory-based delineation of dimensions weak work product, as well as a relatively attending to emotions) based on a
or levels of an assessed construct). complex list of criteria or “what synthesis of literature descriptors. A
counts” in completing a project or session aimed at standardization of
We next examined these four approaches mastering a skill.59 Our close scoring on three RW samples followed.
for their utility in the assessment of examination of the four existing Within this session, we presented
medical students’ RW. Our deliberations approaches led us to select an analytical rationale for scoring, discussed and
were based on both theoretical and instructional rubric as the evaluative resolved scoring discrepancies, and
functional premises. We used paradigm for our own tool.
reached consensus about scoring.
anonymized analogical datasets of
medical student RW exercises—sampled Iterative development of the initial We obtained full formal institutional
anew with each iteration—from the 2009 rubric review board approval from the
and 2010 Doctoring course and family
medicine clerkship as anchors for the Once we had determined which approach Memorial Hospital of Rhode Island prior
deliberation. Although our literature to use, we began the process of to narrative analyses to allow cycles of
search uncovered an existing scale for developing an actual analytical empirical testing on actual examples of
measuring “personal reflection,”54 we did instructional rubric to assess students’ randomly selected medical students’ RW.
not use it for our analysis given its reflective narratives. This was We applied the initial rubric to a dataset
intended purpose for students’ self- accomplished through an accepted of all 93 second-year students’ self-
reported reflective capacity rather than methodology of thorough model review, selected “best” RW “field notes” collected
for assessment of the construct within listing criteria, designating quality levels, for evaluation (2008 –2009). Three raters
RW. Thematic coding26,27 with sole creating a rubric draft, and revising the applied the initial reflection rubric to
emphasis on extraction of themes was also draft.59 Several iterative cycles of code subsets of these field notes, with an
inadequate for our evaluative aims because development were required. overlap of 10 randomly selected notes for
students’ reflective levels within RW could reliability calculation, and interrater
not be determined with such a method. The first iterative cycle: Initial reflection reliability was determined on these 10
Similarly, qualitative analysis was deemed rubric. In the first cycle, we constructed overlapping notes using intraclass
insufficient because of its inability to an initial reflection rubric based on our correlation (ICC; see Table 1). The
provide focused differentiation of reflective comprehensive analysis of relevant distribution of students in each reflection
levels. The fourth approach, the analytical theoretical models of reflection level, according to our coding, was as
instructional rubric,56 is specifically used for and existing reflection measure follows: Level 1 ⫽ 0, Level 2 ⫽ 17, Level
assessment. Analytical instructional rubrics instruments.60 After considering a 3 ⫽ 38, Level 4 ⫽ 28, and Level 5 ⫽ 10.

Table 1
Interclass Correlation (ICC) Estimates Computed for Each Iteration of the
REFLECT (Reflection Evaluation for Learners’ Enhanced Competencies Tool)
Rubric in Five Pilot Tests of the Rubric, Developed at Warren Alpert Medical
School (AMS) of Brown University, Providence, Rhode Island, 2009 –2010

Date of pilot Rubric Number of ICC single Cronbach


test iteration Sample raters measures alpha
1: July 2009 1 10 narratives from the Doctoring course, year 2, AMS 3 0.748 0.899
...................................................................................................................................................................................................................................................................................................................
2: December 2009 2 10 narratives from the Doctoring course, Year 2, and 3 0.455 0.715
family medicine clerkship, years 3–4, AMS
...................................................................................................................................................................................................................................................................................................................
3: January 2010 2 10 narratives from the general surgery clerkship, 3 0.376 0.644
University of Alberta
...................................................................................................................................................................................................................................................................................................................
4: February 2010 3 10 narratives from the family medicine clerkship, 3 0.508 0.756
years 3–4, AMS
...................................................................................................................................................................................................................................................................................................................
5: April 2010 3 60 narratives from the Doctoring course, year 2, AMS 4 0.632 0.774

Academic Medicine, Vol. 87, No. 1 / January 2012 43


Evaluating Reflective Writing

The second iterative cycle: The two raters, and we computed ICCs for ordinal/continuous data for interrater
REFLECT rubric. Next, we set out to the six combinations. reliability for two or more raters when
modify the rubric on the basis of insights data may be considered interval. It may
gained from further literature review Present iteration. The present iteration also be used to assess test–retest
(including review of literature gleaned of the REFLECT was informed by reliability. An ICC may be conceptualized
from the original search, plus new search methodological consultation with as the ratio of between-groups variance
results), application of the initial additional content and psychometric to total variance. In single-measure
reflection rubric to students’ reflective experts and further close reading of the reliability, individual ratings constitute
narratives, and feedback obtained when relevant literature. Our aim was to more the unit of analysis (i.e., single-measure
we presented our initial findings at precisely calculate interrater reliability reliability provides the reliability for a
conferences. We reached consensus about data and to deliberate the role of the single judge’s rating). Single-measure
definitions for four reflection levels REFLECT rubric in formative versus ICC is the more conservative estimate
retained from initial rubric and two summative assessment. Given our and can represent how much agreement
possible outcomes of the reflective primary emphasis on analyzing quality of one rater will have with other raters. We
process, as well as more precise reflection within RW in a developmental chose to use ICCs because the levels in
delineation of criteria presented as a context, we decided to omit assigned rubric iterations are ordinal data where
continuum of development. The four numbers for reflection “levels” to gradations are interpretable, with no
levels carried over from the initial rubric encourage use of the rubric for formative “natural zero.” Each application of the
were Nonreflective: Habitual Action; rather than summative purposes developing rubric involved at least three
Nonreflective: Thoughtful Action; (Appendix 1). raters.
Reflective; and Critically Reflective. The
two possible learning outcomes require REFLECT rubric application As demonstrated in Table 1, we observed
achievement of the Critically Reflective The process of applying the REFLECT variation in the ICCs. The noted decrease
level and were defined as transformative rubric to a reflective narrative consists of between iterations 1 and 2 may be
learning and confirmatory learning. four steps: attributed to insufficient training of the
raters and/or lack of clarity in definitions
1. Read the entire narrative.
We refined and elaborated criteria for of levels and criteria. Some of the ensuing
mastering each of the four levels: voice 2. Fragmentation: Zoom in to details variation may be due to the use of
and presence, description of conflict or (phrases/sentences) of the narrative to different samples of field notes, each of
disorienting dilemma (insight and assess the presence and quality of all which may have had different qualities, as
reflection), attending to emotions, and criteria (see Appendix 1). Determine well as the small sample sizes in iterations
critical analysis and meaning making. We which level each criterion represents. 1 to 4. In addition, further variation may
also identified attention to assignment as be attributed to the alterations in the
3. Gestalt: Zoom out to consider overall
an optional “minor” criterion to be criteria for the rubric’s rating scale, which
gestalt of the narrative (while taking into
addressed when relevant. During this occurred as part of the iterative process of
consideration the detailed analysis of
iteration, we named the rubric REFLECT. scale development. The current iteration
Step 2). Determine which level the
is likely a more stable ICC because it
narrative as a whole achieves. If the
Using three raters, we applied the second includes 60 field notes, though this is still
Critical Reflection level is achieved,
iteration rubric to a sample of 10 new a relatively small sample. Internal
determine whether either or both
reflective narratives from the second-year consistency measured by Cronbach alpha
learning outcomes (transformative or
Doctoring course and the family medicine is also reported in Table 1 and ranges
confirmatory learning) were also
clerkship and a sample of 10 field notes from 0.644 to 0.899.
achieved.
from a general surgery clerkship and again
determined interrater reliability using ICC 4. Defend the assignment of level and
learning outcomes with examples Discussion
(see Table 1).
from the text. Do not “read between RW initiatives within medical education
Third iteration. After improving the tool the lines.” have prospered as medical educators are
and retesting it during the second called on to prepare students to become
iteration, we further reevaluated, refined, A sample reflective narrative and reflective clinicians.3,62 Increasing use of
and redesigned the REFLECT in a third REFLECT rubric analysis is presented in such pedagogy has led to interest in
iteration. To empirically test the tool and Appendix 2. Another example can be formal assessment of achieved level and
determine its interrater reliability, we seen in Supplemental Digital Appendix 1, qualities of reflection within narrative.
applied the rubric to a sample of 10 http://links.lww.com/ACADMED/A68. The rationale for conducting theory-
family medicine clerkship reflective informed evaluation of RW includes
narratives. We then applied the rubric to Statistical analyses obtaining a deeper understanding of the
all 92 second-year Doctoring course We applied single-measure ICCs61 to all professional development of students,
students’ self-selected “best” reflective datasets and computed ICCs for each designing best teaching practices, and
narratives (2009 –2010). We scored all iteration of the REFLECT in the pilot evaluating curriculum outcomes and
narratives independently, and then four developmental phases (Table 1). We used effectiveness. Although written essay
raters independently scored 60 narratives, SPSS version 11.0 (IBM Corporation, methodology may tap into important
randomly split into batches of 10. Each Armonk, New York) to calculate ICCs. competencies such as empathy, personal
narrative was scored independently by An ICC is used to measure reflection, and professionalism, effective

44 Academic Medicine, Vol. 87, No. 1 / January 2012


Evaluating Reflective Writing

assessment of RW can be challenging.32 colleagues50 used a similar statistical with larger samples will be required to
We obtained encouraging results in ease approach in the development of their establish robust internal structure
of application and interrater reliability reflection rubric, yet this rubric does not validity. In addition, we recommend
with the REFLECT rubric. include various reflection domains. testing this rubric against other validated
Kember and colleagues52 introduce a reflection evaluation tools.
We deliberately chose an analytic rubric “transitional” phase between each of four
evaluation paradigm because it promotes reflection categories, though these We propose the use of the REFLECT rubric
a theory-informed evaluation of RW and categories are not elaborated. McNeill as a developmental tool within medical
supports learning and metacognition and colleagues51 offered a relatively education. It is designed to help guide our
(“the act of monitoring and regulating cursory grading system without clear learners toward achieving greater breadth
one’s thinking”).59 The content validity of reference to theoretical underpinnings, and depth of reflective capacity within the
the resulting framework is sound given and Devlin and colleagues’53 rubric is developmental trajectory of becoming
the iterative process of instrument described as a feedback rubric, based on reflective practitioners.62 Such formative
development we employed. Additionally, one typology. In general, we propose that assessment and feedback may help foster
the components of the rubric (levels of the REFLECT rubric achieves a more expertise, promoting more effective self-
reflection, criteria defining each level, and comprehensive assessment than these evaluation64 and self-directed learning,65
outcomes) are grounded in the reflection recent rubric design efforts, increasing its as well as more thoughtful approaches to
literature. Rubric levels capture credibility within an increasing pool of patient care.66 Although our efforts at
developmental progression from habitual instruments for a similar purpose. standardization have yielded promising
action to critical reflection. Criteria for psychometric properties, we recommend
each level are based in theory and clearly The process of rubric development using the REFLECT rubric for formative
explicated. Fundamental, core processes involved refining a pilot rubric through rather than summative assessment. In
of the reflection construct, including further immersion in the literature, contrast to “quantifying” or “grading,”
presence, recognizing “disorienting” application of the rubric to various which may risk a lack of reflective
dilemmas, critical analysis of datasets, and discussion until consensus authenticity by encouraging more
assumptions, attending to emotions, and was reached on specific criteria. The ICC formulaic approaches to reflection,67 we
deriving meaning from the exercise, are scores at the present iteration envision the REFLECT rubric as
all assessed with the rubric. An additional demonstrate acceptable interrater providing qualitative anchors to help
distinguishing feature of the REFLECT is reliability. Feasibility of scoring and educators both assess development of
the two possible learning outcomes of acceptability to both raters and students reflective capacity dimensions and
critical reflection—new understanding are promising based on feedback from formulate constructive, individualized
(transformative learning) and/or faculty development workshops and use feedback to students’ reflective
confirming one’s frames of reference or in student instruction. We have received narratives. At this time, we counter
meaning structures (confirmatory positive feedback about the REFLECT calls for rubrics to be used for
learning). Both of these delineated rubric for formative assessment of quantitative and summative assessment
outcomes have relevance for gaining students’ RW from faculty development of learners.68 We urge caution in this
insight to guide present and future workshops locally, nationally, and regard because such use may prove
behavior. internationally. Further investigations, counterproductive, potentially
including feedback queries for students inhibiting the development of reflective
The REFLECT rubric is currently used and faculty at AMS and multiinstitutional capacity within interactive RW.
within AMS for structured RW collaboration, are planned. The
paradigms within the Doctoring course generalizability of the REFLECT rubric is We plan to study the use of the REFLECT
and family medicine clerkship, though we potentially limited, given its development rubric to enhance the educational impact
could envision its application for and testing within a single institution, but of RW feedback. We hope to examine
products of spontaneous in-class RW we are currently undertaking efforts to both faculty’s and students’ perspectives
assignments as well. Written feedback improve generalizability by using the on the effectiveness of rubric application
about students’ RW is currently standard rubric within various health professions for feedback formulation and promotion
within the Doctoring course and the curricula at multiple institutions. We of reflective capacity. Given the current
family medicine clerkship curricula, and hope to soon complete and distribute a emphasis in medical education on
faculty can use the BEGAN43 and/or rubric “codebook” containing illustrative measurable objectives, future research to
REFLECT rubric tools to formulate this examples of rubric application to determine the extent to which what is
written feedback. Faculty assess overall narratives to enhance feasibility and being measured in text is a valid indicator
“level” of reflection for research purposes, promote generalizability. Future of reflective activity and how this predicts
but students do not receive this information directions include assessments of or correlates with professionalism issues
as feedback. Faculty do not assess quality of longitudinal reflective narratives at is of interest. Further research is needed
writing, in keeping with recent evidence of a various stages of the professional life to explore concurrent validity through
lack of significant relationship between quality cycle and analysis of variables such as the use of reflection scales,54 thematic
of writing and reflective content.63 writing prompt design on rubric results. analyses,26 and/or measures of reflective
practice outcomes. We propose the
Recently reported rubrics for “grading” We note some limitations to our work. inclusion of our rubric paradigm within
RW exhibit similarities and differences Although we provide ample content such an approach as a means of
with REFLECT. O’Sullivan and evidence, further support from studies enhancing “state of the art” reflection

Academic Medicine, Vol. 87, No. 1 / January 2012 45


Evaluating Reflective Writing

assessment. The study of medical schools Ethical approval: This study was reviewed and 14 Guillemin M, McDougall R, Gillam L.
that teach reflective practice has been approved by the institutional review board of the Developing “ethical mindfulness” in
Memorial Hospital of Rhode Island. continuing professional development in
suggested to determine whether they are healthcare: Use of a personal narrative
more likely to produce physicians who Dr. Wald is clinical associate professor of family approach. Camb Q Healthc Ethics. 2009;18:
are able to improve patient care.69 Thus, medicine, Warren Alpert Medical School of Brown
197–208.
University, Providence, Rhode Island.
the connection between medical 15 Huddle TS. Viewpoint: Teaching
education modalities such as RW- Dr. Borkan is professor and chair, Department of professionalism: Is medical morality a
Family Medicine, Warren Alpert Medical School of competency? Acad Med. 2005;80:885–891.
enhanced reflective capacity and quality Brown University, Providence, Rhode Island. 16 Bolton G. The art of medicine. Writing
clinical outcomes69 warrants further values. Lancet. 2009;374:20 –21.
Dr. Taylor is associate professor of family medicine
investigation. and director of clinical curriculum, Warren Alpert
17 Epstein RM, Hundert EM. Defining and
assessing professional competence. JAMA.
Medical School of Brown University, Providence,
Rhode Island. 2002;287:226 –235.
18 Berner ES, Graber ML. Overconfidence as a
Conclusions Dr. Anthony is assistant professor of family cause of diagnostic error in medicine. Am J
medicine and director of medical student education, Med. 2008;121(5 suppl):S2–S23.
RW and its assessment may enhance our Warren Alpert Medical School of Brown University, 19 Mann K, Gordon J, MacLeod A. Reflection
understanding of the professional Providence, Rhode Island. and reflective practice in health professions
development of physicians and help Dr. Reis is professor and immediate past chair, education: A systematic review. Adv Health
guide pedagogic initiatives aimed at Section of Family Medicine and Department of Sci Educ Theory Pract. 2009;14:595–621.
Medical Education, Ruth and Bruce Rappaport Faculty 20 Schon DA. The Reflective Practitioner: How
supporting this process. Metacognitive Professionals Think in Action. New York,
of Medicine, Technion-Israel Institute of Technology,
skills including reflection as well as Haifa, Israel, and adjunct clinical professor of family NY: Basic Books; 1983.
dimensions of professionalism in medicine, Warren Alpert Medical School of Brown 21 Smith MK. Donald Schon: Learning, practice,
effective patient care (such as self- University, Providence, Rhode Island. and change. Infed: The Encyclopaedia of
awareness, empathy, and insight), and Informal Education. http://www.infed.org/
thinkers/et-schon.htm. Accessed September
physician well-being can potentially be References 28, 2011.
fostered through RW exercises.30 We are 1 Mamede S, Schmidt HG, Penaforte JC. 22 Boud D, Keogh R, Walker D, eds. Reflection:
hopeful that longitudinal investigations Effects of reflective practice on the accuracy Turning Experience Into Learning. London,
of RW exercises using the REFLECT of medical diagnoses. Med Educ. 2008;42: UK: Kogan Page; 1985.
468 –475. 23 Moon J. A Handbook of Reflective and
rubric will assist educators as well as Experiential Learning. London, UK:
learners as they reflect on the efficacy of 2 Driessen E, van Tartwijk J, Dornan T. The self
critical doctor: Helping students become Routledge; 1999.
such curriculum initiatives. In essence, more reflective. BMJ. 2008;336:827–830. 24 Mezirow J. Transformative Dimensions of
the use of the REFLECT rubric as part of 3 Plack MM, Greenberg L. The reflective Adult Learning. San Francisco, Calif: Jossey-
the assessment tool kit has the potential practitioner: Reaching for excellence in Bass; 1991.
practice. Pediatrics. 2005;116:1546 –1552. 25 Charon R. Narrative Medicine—Honoring
to broaden the question of “How do the Stories of Illness. New York, NY: Oxford
4 Stern DT, Papadakis M. The developing
doctors think?”70 to “How can we help physician—Becoming a professional. N Engl University Press; 2006.
doctors to think?” As interdisciplinary J Med. 2006;355:1794 –1799. 26 Levine RB, Kern DE, Wright SM. The impact
interest continues to grow in RW and the 5 Sandars J. The use of reflection in medical of prompted narrative writing during
education: AMEE guide no. 44. Med Teach. internship on reflective practice: A qualitative
role of reflective capacity in health care study. Adv Health Sci Educ Theory Pract.
2009;31:685–695.
practice, increased rigor in theory 6 Quirk M. Intuition and Metacognition in 2008;13:723–733.
building, curricula implementation, Medical Education. New York, NY: Springer 27 Brady DW, Corbie-Smith G, Branch WT.
assessment, and outcome research is Publishing Co.; 2006. “What’s important to you?”: The use of
7 Plack MM, Driscoll M, Marquez M, narratives to promote self-reflection and to
called for in order to demonstrate
Cuppernull L, Maring J, Greenberg L. understand the experiences of medical
authenticity and sustainability of such Assessing reflective writing on a pediatric residents. Ann Intern Med. 2002;137:220 –
constructs. Such efforts can help realize clerkship by using a modified Bloom’s 223.
the promise of RW as a vehicle for taxonomy. Ambul Ped. 2007;7:285–291. 28 Kumagai AK. A conceptual framework for the
promoting reflective capacity and its role 8 Accreditation Council for Graduate Medical use of illness narratives in medical education.
Education. The ACGME learning portfolio. Acad Med. 2008;83:653–658.
in building professional identity, as well 29 Wald HS, Davis SW, Reis SP, Monroe AD,
http://www.acgme.org/acWebsite/portfolio/
as for guiding development of medical learn_alp_aboutalp.asp. Accessed September Borkan JM. Reflecting on reflections:
expertise, leading to the formation of 28, 2011. Enhancement of medical education
mindful, compassionate, and competent 9 General Medical Council. Tomorrow’s curriculum with structured field notes and
Doctors: Outcomes and Standards for guided feedback. Acad Med. 2009;84:830 –
practitioners.
Undergraduate Medical Education. London, 837.
UK: General Medical Council; 2009. 30 Shapiro J, Kasman D, Shafer A. Words and
Acknowledgments: The authors gratefully
10 Sargeant JM, Mann KV, Vleuten CP, wards: A model of reflective writing and its
acknowledge Dr. Jonathan White for use of uses in medical education. J Med Humanit.
Metsemakers JF. Reflection: A link between
narratives in determination of rubric reliability, receiving and using assessment feedback. Adv 2006;27:231–244.
Drs. David Cook and William McGaghie for their Health Sci Educ. 2009;14:399 –410. 31 Avrahami E, Reis SP. Narrative medicine. Isr
comments on statistical design, and Dr. Rita 11 Bing-You RG, Trowbridge RL. Why medical Med Assoc J. 2009;11:216 –219.
Charon for insightful dialogue on rubric educators may be failing at feedback. JAMA. 32 Kuper A. Literature and medicine: A problem
application. The authors also gratefully 2009;302:1330 –1331. of assessment. Acad Med. 2006;81(10 suppl):
acknowledge Tina Charest, MD, and Eugene 12 Bryan CS, Babelay AM. Building character: A S128 –S137.
Cone for the use of their reflective narratives. model for reflective practice. Acad Med. 2009; 33 Meitar D, Karnieli-Miller O, Eidelman S. The
84:1283–1288. impact of senior medical students’ personal
Funding/Support: None. 13 Epstein R. Reflection, perception and the difficulties on their communication patterns
acquisition of wisdom. Med Educ. 2008;42: in breaking bad news. Acad Med. 2009;84:
Other disclosures: None. 1048 –1050. 1582–1594.

46 Academic Medicine, Vol. 87, No. 1 / January 2012


Evaluating Reflective Writing

34 Paterson BL. Developing and maintaining 46 Williams RM, Sundelin G. Assessing the presentations: A pilot study. J Surg Res. 2007;
reflection in clinical journals. Nurse Educ reliability of grading reflective journal 142:304 –307.
Today. 1995;15:211–220. writing. J Phys Ther Educ. 2000;14:23–26. 58 Newman LR. Developing a peer assessment of
35 Chretien K, Goldman E, Faselis C. The 47 Plack MM, Driscoll M, Blissett S, McKenna lecturing instrument: Lessons learned. Acad
reflective writing class blog: Using technology R, Plack TP. A method for assessing reflective Med. 2009;84:1104 –1110.
to promote reflection and professional journal writing. J Allied Health. 2005;34:199 – 59 Andrade H. Using rubrics to promote
development. J Gen Intern Med. 2009;23: 208. thinking and learning. Educ Leadersh. 2000;
2066 –2070. 48 Pee B, Woodman T, Fry H, Davenport E. 57:1–7.
36 Wald HS. Guiding our learners in reflective Appraising and assessing reflection in 60 Wald HS, Reis SP, Borkan JM. Really good
writing. Lit Med. In press. students’ writing on a structured worksheet. stuff: Development of a reflection rubric to
37 Wald HS, Reis SP. A piece of my mind. Brew. Med Educ. 2002;36:575–585.
evaluate students’ reflective writing. Med
JAMA. 2008;299:2255–2256. 49 Kember D, Jones A, Loke A, et al.
Educ. 2009;43:1110 –1111.
38 Wald HS. Teaching and learning moments. A Determining the level of reflective thinking
61 Shrout PE, Fleiss JL. Intraclass correlations:
reflective moment. Acad Med. 2009;84:633. from students’ written journals using a
39 Wald HS. I’ve got mail. Fam Med. 2008;40: coding scheme based on the work of Uses in assessing rater reliability. Psychol
393–394. Mezirow. Int J Lifelong Educ. 1999;18:18 –30. Bull. 1979;86:420 –428.
40 Monroe A, Frazzano A, Ferri F, Borkan J, 50 O’Sullivan PS, Aronson L, Chittenden E, 62 Epstein R. Mindful practice. JAMA. 1999;282:
Macko M, Dube C. Doctoring: Course Niehaus B. Reflective ability rubric and user 833–839.
Syllabus. Providence, RI: Warren Alpert guide. MedEdPortal. August 26, 2010. 63 Aronson L, Niehaus B, DeVries CD, Siegel JR,
Medical School of Brown University; 2005– 51 McNeill H, Brown JM, Shaw NJ. First year O’Sullivan P. Do writing and storytelling skill
2007. specialist trainees’ engagement with reflective influence assessment of reflective ability in
41 Taylor J, Frazzano A, Macko M. Doctoring: practice in the e-portfolio. Adv Health Sci medical students’ written reflections? Acad
Course Syllabus. Providence, RI: Warren Educ. 2010;15:547–558. Med. 2011;33:220 –225.
Alpert Medical School of Brown University; 52 Kember D, McKay J, Sinclair K, Wong FKY. 64 Sadler DR. Evaluation and the improvement
2008 –2011. A four-category scheme for coding and of academic learning. J Higher Educ. 1983;54:
42 Wald HS, Anthony DA. Fostering Reflective assessing the level of reflection in written 60 –79.
Capacity With Interactive Reflective Writing. work. Assess Eval Higher Educ. 2008;33:369 – 65 Nicol DJ, Macfarlane-Dick D. Formative
Alpert Medical School Family Medicine 379. assessment and self-regulated learning: A
Clerkship Faculty and Student Guides. 53 Devlin MJ, Mutnick A, Balmer D, Richards model and seven principles of good feedback
Providence, RI: Warren Alpert Medical BF. Clerkship-based reflective writing: A practice. Stud Higher Educ. 2006;31:199 –218.
School of Brown University: 2009 –2011. rubric for feedback. Med Educ. 66 Kumagai AK. Forks in the road: Disruptions
43 Reis SP, Wald HS, Monroe AD, Borkan JM. 2010;44:1143–1144. and transformation in professional
Begin the BEGAN (The Brown Educational 54 Aukes LC, Geertsma J, Cohen-Schotanus J, development. Acad Med. 2010;85:1819 –1820.
Guide to the Analysis of Narrative)—A Zwierstra RP, Slaets JPJ. The development of
67 Wald HS, Reis SP. Beyond the margins:
framework for enhancing educational impact a scale to measure personal reflection in
Reflective writing and development of
of faculty feedback to students’ reflective medical practice and education. Med Teach.
reflective capacity in medical education.
writing. Patient Educ Couns. 2010;80:253– 2007;29:177–182.
259. 55 Goodman LA. “Snowball sampling.” Ann J Gen Intern Med. 2010;25:746 –749.
44 Wallman A, Lindblad AK, Hall S, Lundmark Math Stat. 1961;32:148 –170. 68 Aronson L. Twelve tips for teaching reflection
A, Ring L. A categorization scheme for 56 Boulet JR, Rebbecchi TA, Denton EC, at all levels of medical education. Med Teach.
assessing pharmacy students’ levels of McKinley DW, Whelan GP. Assessing the 2011;33:200 –205.
reflection during internships. Am J Pharm written communication skills of medical 69 Chen FM, Baucher H, Burstin H. A call for
Educ. 2008;72:1–10. school graduates. Adv Health Sci Educ outcomes research in medical education.
45 Chimera KD. The use of reflective journals in Theory Pract. 2004;9:47–60. Acad Med. 2004;79:955–960.
the promotion of reflection and learning in 57 Musial JL, Newman LR, Lown BA, Jones RN, 70 Montgomery K. How Doctors Think: Clinical
post-registration nursing students. Nurse Johansson A, Schwartzstein RM. Developing Judgment and the Practice of Medicine.
Educ Today. 2007;27:192–202. a scoring rubric for resident research Oxford, UK: Oxford University Press; 2006.

Academic Medicine, Vol. 87, No. 1 / January 2012 47


48
Appendix 1
Evaluating Reflective Writing

The REFLECT (Reflection Evaluation For Learners’ Enhanced Competencies


Tool) Rubric

Level Axis II for critical reflection


Transformative
Habitual action Thoughtful action reflection and Confirmatory
Criterion (Nonreflective) or introspection Reflection Critical reflection learning learning
Writing spectrum Superficial descriptive Elaborated descriptive Movement beyond reporting Exploration and critique Frames of reference or Frames of reference or
writing approach (fact writing approach and or descriptive writing to of assumptions, values, meaning structures are meaning structures are
reporting, vague impressions without reflecting (i.e., attempting to beliefs, and/or biases, transformed. Requires confirmed. Requires
impressions) without reflection understand, question, or and the consequences critical reflection critical reflection
reflection or analyze the event) of action (present and Integration of new learning
introspection future) into one’s identity,
informing future
Presence Sense of writer being Sense of writer being Sense of writer being largely Sense of writer being perceptions, emotions,
partially present partially present or fully present fully present
................................................................................................................................................................................................................................................................................................. attitudes, insights,
Description of conflict No description of the Absent or weak Description of the Full description of the meanings, and actions.
or disorienting disorienting dilemma, description of the disorienting dilemma, disorienting dilemma, Conveys a clear sense of a
dilemma conflict, challenge, or disorienting dilemma, conflict, challenge, or issue conflict, challenge, or breakthrough
issue of concern conflict, challenge, or of concern issue of concern that
issue of concern includes multiple
perspectives, exploring
alternative explanations,
and challenging
assumptions
.................................................................................................................................................................................................................................................................................................
Attending to emotions Little or no recognition Recognition but no Recognition, exploration, Recognition,
or attention to exploration or attention and attention to emotions exploration, attention
emotions to emotions to emotions, and gain
of emotional insight
.................................................................................................................................................................................................................................................................................................
Analysis and meaning No analysis or meaning Little or unclear analysis Some analysis and meaning Comprehensive analysis
making making or meaning making making and meaning making
.................................................................................................................................................................................................................................................................................................
Optional minor Poorly addresses the Partial or unclear Clearly answers the Clearly answers the
criterion: Attention to assignment question addressing of assignment question or, if assignment question or,
assignment (when and does not provide a assignment question; relevant, provides a if relevant provides a
relevant) compelling rationale does not provide a compelling rationale for compelling rationale for
for choosing an compelling rationale for choosing an alternative choosing an alternative
alternative choosing an alternative

Academic Medicine, Vol. 87, No. 1 / January 2012


Evaluating Reflective Writing

Appendix 2
REFLECT (Reflection Evaluation for Learners’ Enhanced Competencies Tool)
Rubric Applied to a Reflective Narrative From a Third-Year Student in the Family
Medicine Clerkship at Warren Alpert Medical School of Brown University,
Providence, Rhode Island

Reflective Narrative

Writing Prompt
“Sick people need physicians who can understand their disease, treat their medical problems, and accompany them through their illness.” Rita
Charon, MD, PhD
Reflect on a patient care experience(s) in which you learned something new about the role of a primary care physician. Include a description of the
experience(s) of the patient encounter. Some dynamics to consider:
● Longitudinal physician–patient relationship
● Being there for the patient, alleviating or sharing the suffering, preserving our empathy over time
● Responsibility and service
Were any of your assumptions challenged or validated? Did you gain any insights about yourself (cognitive and/or emotional)? How might your
experience(s) change your practice of medicine?
Student Narrative
Sitting on a small green stool about a foot away from the patient, the doctor furrowed his brow. It had been a particularly long day, filled with
patients asking for early refills on pain medications. Now, Donna* had come in having an asthma attack and we were trying to discuss her
medications.
“I don’t know the name of it,” Donna said. “It’s round and green.”
“Are you sure it’s green?”
“Oh yeah, I’m sure it’s green. I can see it in my head right now. Round and green. Dark green.”
“Are you sure it’s not purple? And does it look like a flying saucer?”
The doctor was sure that his patient was on Advair which came in a dispenser that could be described as round, but was definitely not green. Donna
was sitting on the exam table, her face and her eyes, teary since the attack, were a matching shade of red. The records indicated that Donna was
taking Advair on a daily basis and Albuterol for acute attacks. Donna, however, was describing a green round device that didn’t sound like anything
that the doctor and I were familiar with.
“I take the green circle thing whenever I have an attack and that hasn’t been for awhile. Then I take the other medicine every day.”
“Hold on,” said the doctor reaching for the door. He rustled through his closet outside the exam room for awhile and then returned, holding an
Advair discus. “Do you take anything that looks like this?” he questioned.
“Ohhhhh, yeah!!! That’s it, that’s the thing. That’s the thing I take whenever I have an attack.”
“This … is the green circle?” I asked incredulously.
“Oh, well … I guess it’s purple.”
Oh, Jesus, I thought. This was ridiculous. This lady has no idea what medications she’s taking and it sounds like she’s been taking them completely
wrong. There’s no way the doctor didn’t explain to her that Advair was for daily use and Albuterol was for acute attacks. And the discus was
obviously not green! How did she not bring her medications list in with her, or better yet, the medications themselves so she could tell us how she
took them?
My mentor discussed how to take her medications with her again and then scheduled a follow-up appointment with her so that they could make
sure she understood.
“Wow, I really dropped the ball on that one,” he said as we were walking back to his office.
“Excuse me?” I said, unable to hold in my disbelief. He dropped the ball? It was the patient who had dropped the ball! She had fairly serious asthma
and didn’t know what medications she was on!
“I obviously didn’t communicate well with her the first time she was here. She really didn’t know much about her medications.”
“Well, didn’t you tell her that the Advair was for daily use and then Albuterol for whenever she had an attack?”
“Of course, but obviously something I said didn’t register with her. It’s the job of a family physician to not only tell the patients what medications to
take, but to make sure you’re communicating in a way that is effective. To be honest, I don’t think she’s going to do a much better job taking the
medications after today. It’s hard to remember anything when 10 minutes before you couldn’t breathe. That’s why I’m having her come back so
soon.”
I thought about that patient on my drive home. I usually consider myself to be sensitive to the needs of my patients. I think I’m pretty darn good at
talking to them, empathizing with them, and expressing myself in a way that they understand. But that day, after dealing with what felt like drug
seeker after drug seeker, I had been frustrated and completely unable to relate to this woman. I don’t know if I would have thought to have her
return in a calmer state to go over her medications. I’m embarrassed to say that I might have written her off as someone who just didn’t care.
From this experience I was reminded of just how complicated the job of a family physician truly is. Sure, many times the diagnosis isn’t difficult, but
there is so much more to family medicine than diagnosis. Communication is so crucial to doing the job right. While I may be a good communicator
and try hard to empathize with patients, there is obviously so much left for me to learn. Even my mentor, who has been practicing for almost 25
years, is still improving. Patients are as individual as their diseases. If a physician is not continuously working on communicating more effectively, no
matter how brilliant he/she may be, the patient is being done a disservice.
(Appendix continues)

Academic Medicine, Vol. 87, No. 1 / January 2012 49


Evaluating Reflective Writing

Appendix 2, Continued
REFLECT Rubric Application Process
Writing Spectrum: The learner is reflecting on herself in the situation as well as the mentor, demonstrating Reflection on Action. There is clear
“movement beyond reporting or descriptive writing to reflecting, i.e., attempting to understand, question, or analyze the event” for Reflection level.
The narrative describes grappling with a more nuanced view of a family physician. The writer appears to be on the cusp of critical
reflection–transformative learning level. The importance of “communication,” for example, is identified and described, though some more
elaborated concrete examples of how this could be realized and integrated in future practice might have been helpful, possibly contributing to more
comprehensive meaning making. Overall Level: Reflection.
Individual Criteria
Presence: An authentic voice permeates the writing and there is a sense of bringing the full self to the situation. Thus, the narrative fully conveys
“being there.” The reader is brought into the exam room through provision of details and then into the writer’s “head.” The writer engages the
reader in a powerful, meaningful way. Level: Critical Reflection.
Description of conflict or disorienting dilemma: The disorienting dilemma regarding perceived responsibility for such a medication mishap
poignantly emerges (“unable to hold in my disbelief, my mentor dropped the ball? It was the patient who had dropped the ball!”). The potential
conflicts within a developing professional identity (i.e., the “expert” not always getting it right, exuding competence while remaining open to
improving with humility in approach, considering broader communications issues and issues of responsibility) are impressively identified, though the
challenging of assumptions could be further elaborated. The dilemma of preserving clinical empathy within “dealing with what felt like drug seeker
after drug seeker” is implied. Level: Reflection.
Attending to Emotions: “I had been frustrated” (“and completely unable to relate to this woman”) is an opening phrase, a reflective trigger.
Critical analysis might include (1) considering how feelings of frustration or anger toward patients could arise out of one’s own vulnerability and/or
(2) how self-awareness of emotional state can help maintain provision of quality care, potentially preventing/minimizing emotional distancing. “I’m
embarrassed to say that I might have written her off as someone who just didn’t care”—self-reflective and authentic revelation. There could be
further consideration of (attending to) patient’s emotional state (e.g., emotional upheaval, such as anxiety, in the clinical encounter potentially
disrupting information processing). Level: Reflection.
Critical Analysis and Meaning Making: Salient themes include importance of individualized communication, humanizing of mentor,
dedication to lifelong learning within the profession. Enhanced appreciation of “staying on one’s toes,” reflecting in action to ascertain patient
“being on board” is described, and assumptions are beginning to be challenged. Though there is room for further elaboration of “communication”
for more comprehensive meaning making, the student has introduced several notable elements and appears to have examined the dilemma on
several levels. Level: Reflection–Critical Reflection.
* Patient’s name has been changed.

50 Academic Medicine, Vol. 87, No. 1 / January 2012

You might also like