Portfolio Assessment:

A literature review

Philip Smyth
English Centre
The University of Hong Kong
Overview

 Historical perspectives
 Defining the portfolio

 Purposes of portfolios

 Issues in portfolio assessment

 Research in portfolio assessment

 My own possible research avenues

Historical perspective

 Portfolios widely used for many years

 Late 80s interest in portfolios for
assessment (Belanoff and Dickson
1991)
 90s saw advent of eportfolios

 A shift in emphasis away from

assessment to learning?
Definitions

 “collection of student work that

demonstrates achievement or
improvement” (Stiggins 1994)
 “a portfolio is a collection of evidence
that is gathered together to show a
person’s learning journey over time
and to demonstrate their abilities” (Butler
2006)
Definitions

 “…student writing over time, which

contains exhibits showing the stages
in the writing processes a text has
gone through and the stages of the
writer’s growth as a writer, and
evidence of the writer’s self-reflection
on her/his identity and progress as a
writer” (Hamp-Lyons 1996)
Definitions

 portfolios are “…prepared with a

particular audience in mind”, “…are
selective” and “call for judgments” (Calfee
and Freedman 1996)
Definitions

 “…a purposeful collection of student

work that illustrates efforts, progress,
and achievement in one or more
areas [over time]. The collection must
include: student participation in
selecting contents, the criteria for
selection, the criteria for judging merit,
and evidence of self-reflection” (The
Northwest Evaluation Association cited in Barret 2005)
Definitions – main
characteristics
 They are collections of work, different from
a single timed impromptu essay or a class
essay carried out over a semester.
 They are purposeful in that they
“demonstrate”, “exhibit” or provide
“evidence” of “achievement”,
“improvement”, “the writer’s self reflection”,
“the writing process” and “the writer’s
growth”.
 The degree to which these characteristics
are evidenced in portfolios largely depends
on their purpose.
Types of portfolio
 a process portfolio  A positivist portfolio
 a showcase portfolio  A constructivist
 an assessment portfolio
portfolio  A personal portfolio
 A dossier portfolio  A structured portfolio
 A reflective portfolio  An employment
 A classroom portfolio portfolio
 A working portfolio
Where are portfolios used?
 Primary and secondary classrooms
 In tertiary settings:
 Teacher education
 Medicine
 Nursing
 Engineering
 Dentistry
 Psychology
 ESP/EAP writing classes
How are portfolios used?

 In a class
 Across more than one class

 Statewide

 Across a university curriculum

Purposes

 Accountability; evaluating program or

curriculum effectiveness
 Evaluating individual student
progress; grading, certifying student
accomplishment
 Diagnosing students’ needs; informing
classroom instructional planning
Purposes
 Encouraging teacher efficacy;
encouraging reflective practice at the
school and classroom levels;
supporting teachers’ professional
development
 Encouraging student efficacy;
promoting student self-assessment;
motivating student performance
(Herman, Gearhart and Acshbacher,
1996)
Issues

 Different audiences (who is the

portfolio for?)
 Grading – who grades? (fairness)

 Learning and reflection get lost in

drive to measure competency
(Herman and Winter 1994)
Issues

 Gap between psychometrics and

collaborative nature of the revision
process (Song and August
2002,Hamp-Lyons and Condon 2000 )
 Time needed for both teachers and
students (Callahan 1995, Herman and
Winters 1994)
Common problems

 Purposes – clear to teacher and

student? (Callahan 1995)
 Mismatch between assessment
criteria and goals of programme of
study
 Student anxiety and confusion (Butler
2006)
How do portfolios function best?
Research in portfolio
assessment

A slim collection?
Research in portfolio
assessment
 Validity and reliability
 Fairness

 Impact
Validity

 Vermont program
 Correlation ranging from .47 to .58
between writing portfolio scores and
direct writing assessments
 Similar correlation between portfolio
scores and multiple-choice maths test
scores
(Koretz 1993 cited in Herman and Winter 1994)
Validity
 Gearhart and others (1993) cited in Herman
and Winter (1994) found:
 No relationship when comparing writing
portfolios with standard writing assessments
 Two thirds of students classified as
“masters” on the portfolio assessment would
not have been so classified on the standard
assessment.
Validity
 Gearhart and others (1993) cited in Herman
and Winter (1994) also found:
 When portfolios were scored in two different
ways (holistic and individual pieces scored)
correlations were in the .6 range
 Half the students who would have been
classified as masters on the single portfolio
score would not have been so classified
when individual pieces were averaged
Which assessment best represents an
enduring capability?
Validity
 CUNY (Song and August 2002)
 2 groups, 1 assessed by portfolio and writing
test, the other only a writing test
 Students twice as likely to move to the next
course when evaluated by portfolio
 At the end of the next course the pass rate
and grade distribution for the two groups
were nearly identical
Reliability

 Vermont interrater reliability of .28

to .60
 Pittsburgh portfolio system ranged
from .6 to .7
 Herman et al. (1993)found
correlations of .82 in an elementary
school portfolio containing final drafts
of writing
Reliability

 Little work on other sources of

portfolio reliability
 Score stability over time
 Stability across different rater groups
 The portfolio set in which a particular
portfolio is rated
(Herman and Winter 1994, pg. 51)
Reliability

 Heller, Sheingold and Myford (1998)

think-aloud protocol on portfolio raters
to see if they fit process model of
portfolio rating
 Found score validity was threatened
when a major process was omitted or
extraneous assessment criteria were
applied
Reliability
 Nystrand, Cohen and Dowling (1993)
found reliability could be significantly
improved if:
 Raters scored each task in response
to a prompt before moving to the next
task and
 Raters read several examples
together to decide how they were to
be rated
Impact

 Herman and Winter (1994) based on

self-reports from teachers and others
implementing portfolios appears to
have positive effects on instruction
 Vermont principals affirmed that the
portfolio assessment program had
beneficial effects on curriculum and
instruction
Impact
 Aschbacher’s (1993) action research cited
in Herman and Winter 1994 suggests
teacher’s instructional practices and their
attitudes towards students changed.
 Reported ways they thought about their own
teaching
 Two-thirds of teachers expected higher level
of performance from students
Impact
 Hirvela and Sweetland (2005) used 2 case
studies showing the 2 students did not
strongly endorse the portfolios as used in 2
different courses.
 Seemed to need more explanations of what
portfolio approaches were meant to achieve
 Even with a 5% final course grade students
saw the portfolio as essentially summative in
nature
Impact
 Richardson (2000) study involved
classroom observations teacher and
student interviews and examination of
student writing and teacher response.
 Found that students regard teacher
responses as directives. Were not
prepared to make independent
judgments largely because of the
threat of grades
The future?

 More technical quality

 Issue of fairness needs to be
addressed
 More research on impact

 A move away from psychometric

measurement?
Potential research

 Equity of portfolio assessment

 Validity v reliability

 Nature of feedback

 Portfolios for employers

References
 Abrami, P. C., & Barrett, H. (2005). Directions for Research and
Development on Electronic Portfolios. Canadian Journal of
Learning and Technology, 31(3).
 Belanoff, P., & Dickson, M. (Eds.). (1991). Portfolios : process
and product. Portsmouth, N.H.: Boynton/Cook Publishers.
 Calfee, R. C., & Freedman, S. W. (1996). Classroom Writing
portfolios:Old, New, Borrowed, Blue. In R. C. Calfee & P.
Perfumo (Eds.), Writing Portfolios in the Classroom. MahWah,
N. J.: L. Erlbaum Associates.
 Callahan, S. (1995). Portfolio expectations: Possibilities and
limits. Assessing writing, 2(2), 117-151.
 Case, S. H. (1994). Will mandating portfolios undermine their
value? Educational Leadership, 52(2), 46-47.
References
 Desmet, C., & Cummings, R. (2004). Negotiating the Teaching-
Assessment Cycle in Writing Programs with XML. Paper
presented at the World Conference on E-Learning in Corporate,
Government, Healthcare, and Higher Education 2004
Washington DC.
 Frederiksen, J. R., Sipusic, M., Sherin, M., & Wolfe, E. W.
(1998). Video Portfolio Assessment: Creating a Framework for
Viewing the Functions of Teaching. Educational Assessment,
5(4), 225-297.
 Hamilton, S. J. (2006). A Principle-Based ePort Goes Public
(and Almost Loses its Principles). In A. Jafari & C. Kaufman
(Eds.), Handbook of Research on ePortfolios (pp. 434-446).
Hershey: Idea Group Reference.
 Hamp-Lyons, L. (1990). Second language writing: assessment
issues. In B. Kroll (Ed.), Second language writing : research
insights for the classroom. Cambridge: Cambridge University
Press.
References
 Hamp-Lyons, L., & Condon, W. (2000). Assessing the Portfolio:
Principles for Practice Theory and Research. Cresskill, NJ:
Hampton Press.
 Heller, J. I., Sheingold, K., & Myford, C. M. (1998). Reasoning
about Evidence in Portfolios: Cognitive Foundations for Valid
and Reliable Assessment. Educational Assessment, 5(1), 5-40.
 Herman, J. L., Gearhart, M., & Aschbacher, R. (1996). Writing
portfolios in the classroom : policy and practice, promise and
peril. In R. C. Calfee & P. Perfumo (Eds.), Writing portfolios in
the classroom : policy and practice, promise and peril
 (pp. x, 374 p.). Mahwah, N.J.: L. Erlbaum Associates.
 Herman, J. L., Gearhart, M., & Baker, E. L. (1993). Assessing
writing portfolios: Issues in the validity and meaning of scores.
Educational Assessment, 1(3), 201-224.
References
 Herman, J. L., & Winters, L. (1994). Portfolio research: A slim
collection. Educational Leadership, 52(2), 48-55.
 Hirvela, A., & Sweetland, Y. L. (2005). Two case studies of L2
writers' experiences across learning-directed portfolio contexts.
Assessing writing, 10, 192-213.
 Holt, D., & Baker, N. W. (1991). Portfolios as a follow-up option
in a proficiency-testing program. In P. Belanoff & M. Dickson
(Eds.), Portfolios: process and product. Portsmouth NH:
Boynton/Cook.
 Jafari, A., & Kaufman, C. (Eds.). (2006). Handbook of Research
on ePortfolios. Hershey: Idea Group Reference.
 Murphy, S., & Camp, R. (1996). Moving towards systemic
coherence: A discussion of conflicting perspectives on portfolio
assessment. In R. C. Calfee & P. Perfumo (Eds.), Writing
portfolios in the classroom: policy and practice, promise and
peril. Mahwah, N.J.: L. Erlbaum Associates.
References
 Nystrand, M., Cohen, A. S., & Dowling, N. M. (1993).
Addressing reliability problems in the portfolio assessment of
college writing. Educational Assessment, 1(1), 53-70.
 O'Brien, K. (2006). ePortfolios as Learning Construction Zones.
In A. Jafari & C. Kaufman (Eds.), Handbook of research on
ePortfolios (pp. 74-82). Hershey: Idea Group Reference.
 Ostheimer, M. W., & White, E. M. (2005). Portfolio assessment
in an American college. Assessing writing, 10, 61-73.
 Pullman, G. (2002). Electronic Portfolios Revisited: The efolios
Project. Computers and Composition, 19, 151-169.
 Richardson, S. (2000). Students' conditioned response to
teachers' response: Portfolio proponents, take note! Assessing
Writing, 7, 117-141.
References
 Song, B., & August, B. (2002). Using Portfolios to Assess the
Writing of ESL Students: A Powerful Alternative? Journal of
Second Language Writing, 11, 49-72.
 Spalding, E., & Cummins, G. (1998). It was the best of times. It
was a waste of time: University of Kentucky students' view of
writing under KERA. Assessing Writing, 5(2), 167-199.
 Stiggins, R. J. (1994). Student-centred classroom assessment.
New York: Merrill.
 Wagner, M., & Lamoureaux, E. (2006). Implementing an
Outcome-Based Assessment ePortfolio. In A. Jafari & C.
Kaufman (Eds.), Handbook of Research on ePortfolios (pp. 539-
550). Hershey: Idea Group Reference.
 Weigle, S. C. (2002). Assessing writing. Cambridge, UK
Cambridge University Press.
References
 Yancey, K. B. (1996). Portfolio as genre, rhetoric as reflection:
Situating selves, literacies, and knowledge. WPA 19(3), 55-69.
 Yao, Y., Thomas, M., Nickens, N., Downing, J. A., Burkett, R.
S., & Lamson, S. (2008). Validity evidence of an electronic
portfolio for preservice teachers. Educational Measurement:
Issues and Practice, Spring 2008, 10-24.