HEM 423 Evaluation

Education in Nursing in Teaching Process

 Class Objectives

After completing this class, you will be able to:

►Define the terms measurement, assessment and evaluation.
►Compare assessment and evaluation
►Identify purposes of evaluation.
►Discuss barriers to evaluation.
►Examine methods for conducting an evaluation
“Ever tried. Ever failed. No matter.
Try Again. Fail again. Fail better.”

Samuel Beckett
Remember teaching process
 Share your thoughts: (3 minutes)

► Which evaluation methods do you routinely use in

your teaching to measure learning?
(Think of a particular patient or mentee teaching
experience).
► How is evaluation different from assessment?
 WHAT IS ASSESSMENT?
► …is the purposeful, systematic and ongoing collection of
evidence and its use in making judgments about student
demonstrations of learning outcomes
► Processes for assessment require inference about what individuals
do in relation to what they know.
For example, inferences can be drawn about how students are applying theory to
practice from instructor observations of students implementing client care, from
student self-assessments, and from peer assessments.
► …to decide a direction for action
 PURPOSES OF ASSESSMENT

► To provide information about learning needs is diagnostic;

teachers use that information to decide on the appropriate
content and instructional activities for students to meet the
learning outcomes (Brookhart & Nitko, 2008).
► determine relevant instructional strategies to meet students’
learning needs and help them improve performance
► to provide valid and reliable data for determining students’
grades.
 MEASUREMENT

► is the process of assigning numbers to represent student

achievement or performance according to certain rules,
► for instance answering 85 out of 100 items correctly on a test
► …involves systematic quantification of data (collected in an
organized manner according to rules)
 WHAT IS EVALUATION?
► … is the process of making judgments about student learning
and achievement, clinical performance, employee
competence, and educational programs, based on
assessment data.
► ..to determine the extent to which an action was successful
Sample evaluation questions:
“How well did the student perform?
“Is the student competent in clinical practice?
 EVALUATION
► …makes judgments about value or worthiness in
relation an objective, goal or outcome.

► …needs information from a variety of different

sources and at different times.
 EVALUATION

In terms of educational programs,

►…includes collecting information prior to developing
the program, during the process of program
development to provide a basis for ongoing revision,
and after implementing the program to determine its
effectiveness.
FEEDBACK
Feedback differs from assessment and evaluation.
Assessment requires instructors to make inferences
and evaluation requires them to make judgments.
Feedback is nonjudgmental and requires instructors
to provide learners with information that facilitates
improvement
FEEDBACK
it should be specific, and it should be directly linked
to learners’ personal goals
Periodic, timely constructive feedback that
recognizes both strengths and areas for
improvement should be given
Certainly “provides prompt feedback” is
particularly relevant.
 GRADING

► indicates the degree of accomplishment

achieved by a learner
FUNCTIONS OF EVALUATION
Class Activity (5 minutes)

Why do we evaluate students/ learners?

► suggest the three most important reasons

► and share with the class
 WHY WE EVALUATING TEACHING?

Two broad purposes;

► Evaluation for improvement

► Evaluation for accountability
WHY WE EVALUATING TEACHING?

Incentive to learn (motivation)

Feedback to student
Modification of learning activities
Selection of students
Success or failure
Feedback to teacher
Protection of society (certification of competence)
 EVALUATION
In planning any evaluation,
► the first and most crucial step is to determine the
focus of the evaluation.

► Usefulness and accuracy of the results of an

evaluation depend heavily on how well the
evaluation is initially focused.
 Evaluation focus includes five
basic components:
audience, purpose, questions, scope, and resources (Ruzicki,
1987).

To determine these components, ask the following questions:

►For whom is the evaluation being conducted?
►Why is the evaluation being conducted?
►What questions will be asked in the evaluation?
►What is the scope of the evaluation?
►What resources are available to conduct the evaluation?
CHARACTERISTICS OF EFFECTIVE EVALUATION

► Evaluates what we teach – and what we expect

students to learn.
► Actively involves both teachers and students.
► Uses multiple and varied measures.
► Provides information for improving learning.
Effective Evaluation...

► Provides useful, timely feedback to those being

assessed and those most affected.
► Is an intrinsically educational activity – one that
reinforces and furthers the teaching and learning
goals on which it focuses.
Processes Involved in Evaluation

1. Setting the evaluation criteria

2. Providing evidence in relation to those criteria
3. Making a judgment on the evidence provided
4. Communicating the results of the judgment
5. Making use of the results of the evaluation
 Guiding Questions

► What is the purpose? What do you hope to learn from this evaluation?
► What will be evaluated? (learning objective)
► Who will be evaluated?
► How will it be evaluated?
► In what setting will the evaluation be conducted?
► How will the results be analyzed?
► How will the results be used?
► How will the results be helpful to you and your learners?
► To whom will the results be communicated?
Principles of Evaluation

• Clear purpose – central purpose is to improve student learning

• An integral part of the curriculum design and planning
• Relevant criteria
• Appropriate methods
• Clear communication and transparent
• Consistent procedures
• Valid and practicable tasks
• Realistic workload
• Provision of constructive feedback to facilitate and improve
learning and teaching
Assessing Student/learner Learning Outcomes
(Class Activity- 20 minutes)

Problems with assessing learning outcomes

► Discuss and share any problems you have encountered or

concerns that you have in evaluating the learning outcomes
of your patients
► Discuss how you solved these evaluation problems.
Problems/barriers with assessing learning
outcomes

► Language
► Education level – literacy level
► Patient disabilities
► Time
► Environment
► Educator’s attitudes, body language, gestures ….
 Barriers to Evaluation

► Lack of clarity
► Lack of ability
► Fear of punishment or loss of self esteem
Approaches and Models of Evaluation

► Formative / Summative
► Criterion-referenced / Norm-referenced
► Measurement model / Standards model
► Teacher-centred / Student-centred
► Teacher-Peer-Self-evaluation-360Degree
evaluation
Formative Evaluation
Continuous: On-going during learning / teaching
Diagnostic: Provides feedback to students and teachers on
-Strengths and weaknesses
-Difficulties
-Misconceptions
Remedial: Provides opportunities for modification / improvement
The purpose of formative evaluation is to determine where
further learning is needed.
Formative Evaluation
In the classroom, formative information may be collected by teacher:
observation and questioning of students, diagnostic quizzes,
small-group activities, written assignments, and other activities that
students complete in and out of class.

In clinical practice, formative evaluation is an integral part of the

instructional process.
The teacher continually makes:
observations of students as they learn to provide patient care,
questions them about their understanding and clinical decisions,
discusses these observations and judgments with them, and guides
them in how to improve performance.
Summative Evaluation

Terminal; At the end of learning / teaching

Descriptive; How well materials / knowledge / skills
have been learned
For ranking and selection; No provision for
modification / improvement
Summative Evaluation

Strategies used commonly for summative evaluation in

the classroom and online courses are:
tests, term papers, and other types of projects.

In clinical practice:
rating scales, written assignments, portfolios, projects
completed about clinical experiences, and other
performance measures may be used.
 Formative and Summative Evaluation

• Both formative and summative evaluation are

essential components of most nursing courses.
• However, because formative evaluation represents
feedback to learners with the goal of improving
learning, it should be the major part of any nursing
course.
Approaches and Models of Evaluation

► Formative / Summative
► Criterion-referenced / Norm-referenced
► Measurement model / Standards model
► Teacher-centred / Student-centred
► Teacher-Peer-Self-evaluation-360Degree
evaluation
Criterion-Referenced Interpretation

► involves interpreting scores based on preset

criteria, not in relation to the group of
learners.
 Norm-Referenced Interpretation

► In norm-referenced interpretation, test scores and

other assessment data are compared to those of a
norm group.
► Norm-referenced interpretation compares a student’s
test scores with those of others in the class or with some
other relevant group.
► The student’s score may be described as below or
above average or at a certain rank in the class.
Approaches and Models of Evaluation

► Formative / Summative
► Criterion-referenced / Norm-referenced
► Measurement model / Standards model
► Teacher-centred / Student-centred
► Teacher-Peer-Self-evaluation-360Degree
evaluation
 Measurement Model of Evaluation

• Based on the theory of individual differences

• Student performances are reduced to
quantifiable scores for comparison purpose
• Focuses on
- reliability of scores
- discriminating and ranking of students
 Standards Model of Evaluation
Based on the assumption that
✔ standards can be achieved by most students, and reflected in their
performance
✔ student performance can be fairly assessed against the set
standards
Focuses on
✔ identifying the essential performances in the given discipline
✔ establishing clear criteria defining the quality of the standards of
student performance
✔ obtaining evidence of the achievement of standards
Approaches and Models of Evaluation

► Formative / Summative
► Criterion-referenced / Norm-referenced
► Measurement model / Standards model
► Teacher-centred / Student-centred
► Teacher-Peer-Self-evaluation-360Degree
evaluation
Teacher-centred / Student-centred

Depends on who is involved in the various processes involved

in evaluation
1. Setting the evaluation criteria
2. Providing evidence in relation to those criteria
3. Making a judgment on the evidence provided
4. Communicating the results of the judgment
5. Making use of the results of the evaluation
Approaches and Models of Evaluation

► Formative / Summative
► Criterion-referenced / Norm-referenced
► Measurement model / Standards model
► Teacher-centred / Student-centred
► Teacher-Peer-Self-evaluation-360Degree
evaluation
Teacher / Peer / Self Evaluation
360 degree evaluation

Depends on who is involved in making a

judgment on the evidence provided
360 degree evaluation Self

Patient/patient family
member
Teacher

Student

Nurses Classmates
 Evaluating Student Learning Outcomes

Evaluation should not be labelled as “good” or

“bad”, but rather “appropriate” or “inappropriate”.

The important issue is how the evaluation is designed

and implemented to appropriately address the
learning objectives
 Portfolio Evaluation

Based on Constructivist Theory

• Meaning is created by the learner rather than being imposed
or transmitted by direct instruction
• Students are required to provide evidence to convince the
teacher that the best kind of learning in relation to the
objectives has been achieved
• A move towards a more student-centred approach to
assessment & evaluation
Constructivism is ‘an approach to learning that holds that people actively construct or
make their own knowledge and that reality is determined by the experiences of the
learner’ (Elliott et al., 2000, p. 256).
 Portfolio Evaluation
Essential characteristics:

Collection of work produced by the student

Assembled/created for particular purpose
Includes students’ reflection on their work in relation
to the achievement of objectives
 Portfolio Evaluation
Students’ responsibility is to:

✔ provide evidence that learning relevant to course

objectives has taken place
✔ choose and justify the portfolio items
✔ provide a holistic case of an integrated and usable
body of knowledge
Portfolio Evaluation
Teachers’ responsibility is to:

✔ inform students about course objectives

✔ provide guidance for preparation of the portfolio
e.g. philosophy of portfolio assessment, implementation
procedures,
✔ examples of some possible portfolio items
✔ provide on-going support and formative feedback
✔ make a holistic evaluation of the portfolio
✔ provide feedback
Collecting evidence for evaluation

► Techniques
► Observation
► Consultation
► Analysis
Observation…
► …involves observing students as they participate in
teaching, learning and evaluation activities.
► Teacher observation occurs continually as a natural part of
the learning and teaching process and can be used to
gather a broad range of information about student
learning, teaching and the learning environment.
► Teacher observations can also be structured to gather
particular kinds of information in relation to learning
outcomes
Consultation…

► …involves discussing student learning, teaching and aspects of

the learning environment with students, colleagues, parents,
carers and others.
► The varying perspectives of the participants in consultations
can help enrich the evidence gathered.
► Consultation can be used to verify the evidence gathered
using other techniques.
► Some consultations may reveal a need for more detailed
investigation.
Analysis…

► … involves examining in detail student

learning, own or others’ teaching and other
aspects of the learning environment.

► (Adapted from Position and Guidelines on Assessment and Reporting for Years 1 to 10:
http://www.qsa.qld.edu.au/research/qscc/pdf/gl_asses.pdf)
Collecting evidence - records

► Records of evaluation may include:

► sample teaching artefacts (eg unit documentation,
resources developed, audio or video recording of teaching)
► own reflection documents (eg learning log or detailed
planning documents)
► samples of student demonstrations of learning outcomes
► samples of student reflection (eg learning logs)
Collecting evidence – records (cont.)

► samples of self and peer assessment sheets

► student observation records (eg checklists)
► feedback sheets from supervising teacher
► notes of discussions with students, teachers or
administrators
► analysis sheets (based on any of the above)
What do we do with evaluations?

► celebrating success
► further investigation of some findings
► consultation with colleagues
► reporting findings to others
► maintaining aspects of current practice
► altering aspects of current practice
► professional development
► planning for subsequent learning
Other considerations

► Guidelines for ethical conduct of evaluations

EVALUATION METHODS

► Evaluation methods include those actions that are

undertaken to carry out the evaluation according
to the design structure.
► All evaluation methods deal in some way with data
and data collection.
 EVALUATION METHODS

Answers to the following questions will assist in selection of the most

appropriate, feasible methods for conducting a particular
evaluation in a particular setting and for a specified purpose:
►What types of data will be collected?
►From whom or what will data be collected?
►How, when, and where will data be collected?
►By whom will data be collected?
Evaluation Methods

► Methods of evaluation must be selected according

to educational objectives and domains to be
evaluated .
 Cognitive Domain
The cognitive domain includes all knowledge and those skills that
require thought rather than practical action. There are many ways
of testing for cognitive learning
Written test Oral test
Objective type
Subjective type
• Matching
• Long Answer,
• Multiple Choice
• Essays,
• True & False
• Problem-solving Exercises
• Short Answers
• a Project Execution
Affective Domain

The affective domain is concerned with attitudes and emotions.

This is a harder area to test for learning, but there are some
options:

►interviews
►direct observation
►role play
►attitude rating scale
►simulation
►observational rating scale
►checklist
Psychomotor

► The psychomotor domain covers all practical skills. There's usually

only one way to test for practical skills and that's to have them
performed, although in certain circumstances they can be
simulated (many pilots only ever get to use a flight simulator before
flying a new plane for real).

►Direct observation
►Real practical tests
►Simulated practical tests
► Observation
► Tutors, coaches or the learner's supervisor can watch out
for the way in which learning is being applied.
► They can also provide positive reinforcement where the
results are successful and constructive feedback and
encouragement where they are not.
► Automatically
► In some cases, where the learner is working with
electronic equipment, it may be possible for
evidence of behaviour to be gathered
automatically.
► Self-reporting
► Learners will know better than anyone whether
they are applying their new learning.
► You may ask learners to keep counts of what
they do, complete checklists or questionnaires.
 Clinical Education Evaluation Methods
Observations
► Anecdotal notes
► Incident Notes Oral Presentations
► Rating Scales Simulations
► External Raters Self-Evaluations
► Skills checklists
► Written work
► Observation guides
► Process recordings
► Nursing care plans
► Documentation
SUMMARY-Evaluation

► Evaluation is the process of determining the

effectiveness of teaching. There are two major
concerns of evaluation:
► Learner evaluation: did the learner learn what he
or she was supposed to learn?
► Teaching evaluation: was the teaching
presented in an effective manner?
Levels of Learner Evaluation
About evaluating teaching...

“Use multiple sources of data if you are serious about

improving teaching
Cashin, p.93, 1992
NOW
Open your Instructional plan

► Fill in the fourth Column.

 Exercise

►Define the terms measurement, assessment and evaluation.

Please match the terms given below remembering learning/teaching process

Terms Definition

Evaluation …..
Measurement
Assessment …..

…..

…..
 Exercise

►Compare assessment and evaluation

►What are the differences and similarities between assessment and

evaluation?
 Exercise

►Identify purposes of evaluation.

►All of the following are the purposes of evaluation EXCEPT

a) Incentive to learn (motivation)
b) Feedback to student
c) Modification of learning activities
d) Selection of best teacher
 Exercise

►Discuss barriers to evaluation.

Statement Agree Disagree

It is not important to patient literatcy level for patient education

I always ask to patients their own previous knowledge related disease

Nurse educator should re-organize environment before patient education

session
I always organize patient education session basen on patient comfortable
time
 Exercise

►Examine methods for conducting an evaluation

MIND MAPPING

COMMENTS
&
QUESTIONS