LINGOs PMD Pro 1 e-Learning

Formative Evaluation
Preliminary Results
and Recommendations

By Julaine Fowlin and Wendy Gentry
April 26, 2012
Client: Dr. Katherine Cennamo, Virginia Tech

Evaluand: e-Learning Module
As preparation for the Project Management for
Non-Governmental Organizations Certification Exam (PMD Pro 1)









Project Overview
The purpose of the formative evaluation is to identify the strengths and
weaknesses of a self-directed online project management module. The
evaluation will collect data to determine: module prior to release to LINGOs, its
member organizations and of the module

1. the extent to which the learning objectives are achieved,
To what extent does the design of the instruction facilitate
achievement of the learning objectives?

2. the usability of the learning interface, and
To what extent is the user interface design usable?


3. the level of the participants engagement in completing the instruction.
To what extent are the learners engaged during the instruction?


tResults are to be used to modify the module prior to release
o be used to modify the module prior to release
Purpose and Alignment to Key Questions
Instrumentation and Participants
1
Complete
Introductory
Questionnaire
2
Review
Online
Module
3
Complete
Knowledge
Questionnaire
4
Complete
Module
Survey
7 Participants after omissions
Gender: 4 females, 3 males
Nationalities: 3 Americans, 1 Cameroon, 2 Jamaicans, 1 South African

1. Learning Objective Achievement: Overall Scores
Introductory (Pre-Test) Knowledge (Post-Test)
Mean 15.3 83.7
Median 14.3 92.9
Mode 0 92.9
1. Learning Objective Achievement: Breakdown
1. Learning Objective Achievement: Survey Results
4
Strongly Agree
3
Agree
2
Disagree
1
Strongly Disagree
Mean 3.0 - 3.3 and Mode 3.0

It was made clear what I was expected to learn in the module

I know the factors that influence data collection

I know the factors and tools for data analysis

Exception

2.7 Mean, 3.0 Mode - I know the features of the project logical framework
1. Learning Objective Achievement: Survey Results
4
Strongly Agree
3
Agree
2
Disagree
1
Strongly Disagree

Strengths related to presentation of content ( Mean 3.4 - 3.6 and Mode 4.0):


Scenario based approach supported learning

Practice reinforced learning

Feedback from practice was relevant

Confidence in answering the assessment questions

The practice
opportunities
helped reinforce my
learning.
I felt like someone
was actually speaking
to me
1. Learning Objective Achievement
Identified Weaknesses:

Number of practices
Grammatical errors
Time for completion unknown


The number of practice
opportunities could
have been increased.

there were a few
grammatical errors.

It will be helpful to know
how long is the module,
so I suggest time
allotment for the module
and/ or practices.
1. Learning Objective Achievement
Recommendations for Improvement:

Increase the number of practices in the Data Analysis section




Correct grammatical errors

Include an estimated time to complete the module

Have the assessment validated**


2. Usability of Interface Design
4
Strongly Agree
3
Agree
2
Disagree
1
Strongly Disagree
Strengths ( Mean 3.5 - 3.6 and Mode 4.0):

Overall usability
Navigation allowed freedom to explore
Instructions were easy to follow
Screen content was easy to read

The interface design
was well organized
and easy to read and
follow.
2. Usability of Interface Design
Identified Weaknesses:

Knowledge of where the learner is within the lesson
(one participant)
Navigational arrows not noticeable
(one participant)


I wasnt sure where I
was in the lesson.

2. Usability of Interface Design
Recommendations for Improvement:

Provide an explanation at the beginning of the module about
the use of the numbers on the map
Change the shape of the navigational buttons to arrows


3. Learner Engagement during Instruction
4
Strongly Agree
3
Agree
2
Disagree
1
Strongly Disagree
Strengths (Mean 3.3 and Mode 3.0 -4.0)

Motivated to complete the module (strongest category mode)
Module was enjoyable and interesting
Content was meaningful

3. Learner Engagement during Instruction
4
Strongly Agree
3
Agree
2
Disagree
1
Strongly Disagree

Mixed Reactions:

2.9 Mean, 3.0 Mode Connection to characters and interest in the storys progression
(weakest category mean)


I also thought the body/facial
gestures were engaging,
specific the 'light bulb'
moments in gestures were
timed appropriately with myself
learning and forming those
associations with the material.
The characters weren't integral to
my interest in the lesson so I would
have continued whether or not there
were characters and a storyline.
3. Learner Engagement During Instruction
Recommendations for Improvement:

Give learners the option to view the content without the
characters, example print content outline

Review how technical material is handled and determine if it is
possible to reduce the cognitive load

My motivation changed
during the lesson especially
when there was a lot of
technical information.