Rubric-based

Assessment of Critical
Thinking Skills
Guy Hoelzer and Christie Howard
Department of Biology
UNR
or
More than Just the Facts,
Maam:
Assessment of Critical Thinking
for Biology Majors and Programs
Guy Hoelzer and Christie Howard
Department of Biology
UNR
Our Motivating Factors
1. Critical thinking (CT) has been identified by
the Biology faculty as one of the key things we
want our majors to be able to do well upon
graduation
2. CT is notoriously difficult to teach and assess
Not part of our normal course content
3. Emphasis on CT in the capstone course for
Biology majors leaves the impression that
they are not very good at CT and dont really
understand what we mean by it
4. Indeed, they dont seem to
trust the instructors
explanation of CT
It seems too subjective, so the target is the
mind of the instructor rather than anything of
value to the student
It seems that every instructor means
something different by CT, so it doesnt seem
like a real or particular kind of skill
Instructors too often pay lip-service to CT
without giving it weight in grading
5. Generic CT Skills vs.
Discipline-Specific Skills?
We would like to generate majors who
think like a biologist
On the other hand, CT skills ought to be
generally useful in any intellectual
endeavor
6. Student Assessment Of CT
Skills Is Hard Enough - Can
We Do Programmatic
Assessment Of CT?
We would like to engineer a curriculum that
builds on student CT skills in an integrated
and coherent fashion
We Propose To
emulate the rubric-based institutional model
developed at Washington State University

[see http://wsuctproject.wsu.edu/]
The CT Rubric for Biology
Students
Will include the key components of CT
identified by the Biology faculty
Will serve as a template that can be modified
to meet the needs of specific courses
Will be a primary tool for assessing student
work at two or more points within the majors
curriculum
Track development of the CT skills of individual
students
Focus on an Intro course (Howard) and the
capstone course (Hoelzer)
Developing the Rubric
H&H will design a 1
st
draft of the rubric based
on the relevant assignments in their respective
courses
Students are asked to criticize scientific papers in
each course
H&H will do a pilot study by using the1
st
draft of
the rubric to assess 20-30 sample student
papers from each course
Does the rubric effectively discriminate student
performance over the full range encountered in both
courses?
A Faculty Retreat
Introduce the rubric to the Biology faculty (from
both UNR and TMCC) and solicit feedback at a
faculty retreat
TMCC will be included to facilitate articulation
for transfer students
Ideally, the rubric will be useful and ultimately used
in TMCC courses as well as UNR courses
Outside evaluators (experts) will be present to
contribute to the discussion and report to H&H
on the efficacy of our process
Evaluators, like John Mahaffy, will be consulted
throughout the duration of the project
Consistency in Rubric Scoring
H&H will design a second draft of the rubric
based on feedback from the retreat
Clarify and calibrate the rubric with an Inter-
rater reliability (IRR) study
Each member of the IRR team will score the
same set of 20-30 pieces of student work
Meet to discuss differences among raters and
possible causes of those differences
H&H will again revise the rubric based on the
IRR results to maximize repeatability and
tune the scoring scale to student variances
Another Faculty Retreat
Explain the revised rubric to the faculty
Workshop in which all faculty will use the
rubric to score student work
Faculty training
Another round of faculty input into rubric
development
Effectively another (informal) IRR study
Fostering faculty buy-in on use of the rubric
A Novel Student IRR Study
It is just as important that the meaning of the
rubric be clear to students (consistently
interpreted among students) as to the faculty
Student interpretations should be the same as the
intended meaning
Students should recognize a consistent thread
regarding CT across courses and instructors
A team of students will use the rubric to score
anonymous student work in an IRR study
Applying The Rubric
Sample work from the Intro and Capstone
courses will be collected and assessed
As the project ages, we will collect repeated
samples from individual students where
possible
We will aim to codify a comparable use of the
rubric in one or more 300-level courses from
the Biology majors curriculum and encourage
use of our CT rubric wherever appropriate
throughout the Biology curriculum
Work by the Assessment Team
Team will meet after each semester to assess at least
30 papers from each course
Each meeting will start with a small IRR study to align
the scoring practices of team members
Each piece of student work will be scored by two team
members, and discrepant scores will be resolved by a
third team member
We have proposed to pay all participants in this
project for 3 years
We hope that use of the rubric by the faculty for both student
and programmatic assessment will become valued and
routine, and easily incorporated into ordinary departmental
activities when the grant has ended
Some Planned Data Analyses
Track patterns of growth in student CT skills
Compare performance in CT skills with other
variables
E.g., GPA, number of CT-type Biology courses
taken, number of CLA courses taken
Compare 1
st
generation college student
performance with others
Compare transfer with non-transfer students
Compare assessed CT skills with the
perceptions of our alumni and their
employers
Refine Our Curricular Map
Obtain a more detailed understanding of
the extent to which CT skills are taught
and/or assessed in particular courses
Are there gaps in the curriculum allowing
some students to miss regular attention to
CT skills?
Use the results of assessments and
analyses to suggest curricular
modifications addressing problems
revealed by the process
Our Guiding Principles:
Faculty Input and Training
Workshops and retreats focused on
defining and assessing CT
Discussion involvement and presentations
by outside experts
Well paid faculty will participate in this
project from start to finish
Improved quality control
Faculty buy-in regarding use of the rubric and
the value of programmatic assessment
Our Guiding Principles:
Including TMCC
Engaging TMCC Biology faculty in this sort
of project should:
Improve our understanding of pedagogy at
TMCC
Improve understanding by TMCC faculty of
our curricular expectations
Facilitate the transition of transfer students
into the UNR Biology major
Our Guiding Principles:
Communicate Our Process and
Outcomes to Others
Align efforts with the UNR Core Curriculum plans to
assess general (not discipline-specific) CT skills
Inform Biologists at all other UCCSN institutions of
our project, our rubric, and our results
Once our rubric is adopted by the department for
use throughout the curriculum, it will be published
on the departmental website
We will publish our process and analytical results in
a journal specializing in educational research
E.g., The Journal of Research in Science Teaching