184
2, MAY 2011
Improvements to an Electrical Engineering Skill
Audit Exam to Improve Student Mastery
of Core EE Concepts
David W. Parent, Member, IEEE
Abstract—The San José State University Electrical Engineering
(EE) Department implemented a skill audit exam for graduating
seniors in 1999 with the purpose of assessing the teaching and the
students’ mastery of core concepts in EE. However, consistent low
scores for the first years in which the test was administered sug-
gested that students had little incentive in reviewing for passing
the test. To promote the concept that there is a set of basic skills
every graduating senior should master, improvements were imple-
mented that included creating an online exam used for review and
requiring students to earn a 70% or higher score on an in-class
exam that is a requirement for graduation. After the improvements
were made, all students demonstrated at least 70% mastery of the
core EE concepts as measured by the improved in-class skill audit
exam. The details of these improvements are presented. Anony-
mous survey results of graduating seniors indicate that students
feel that preparing for the skill audit exam was a good use of their
time and that the exam should be made more rigorous for future
students.
Index Terms—ABET, capstone engineering experience, elec-
trical engineering education, exit exam, learning management
system.
I. INTRODUCTION
A S PART OF ABET 2000 [1], [2], the EE Department at
San José State University, San José, CA, added the re-
quirement for all students enrolled in the department’s capstone
senior design class [3] to take a skill audit exam [4]. The orig-
inal purpose of the exam was to measure the department’s se-
niors’ mastery level of core EE concepts and use this data to
improve the delivery of these core concepts in the department’s
core classes. The faculty stated that an exam under their control
would give better feedback than using the Fundamental Exam
(FE) [5]–[7]. In addition, the faculty did not want to implement
a qualifying exam as described in [8] and [9].
The original skill audit exam consisted of 50 items in a free-
response format. The students took the exam during a 3-h pe-
riod under the supervision of a proctor. The students were not
given any information about what was to be on the exam be-
cause the faculty wanted to measure what the students remem-
bered without preparation. The items, for the most part, were
intended to measure student mastery levels without preparation.
Typical items for each area included the following.
Manuscript received September 02, 2009; manuscript revised November 18,
2009. Date of publication March 15, 2010; date of current version May 04, 2011.
The author is with the Electrical Engineering Department, San José State Uni-
versity, San José, CA 95192-0084 USA (e-mail: dparent@email.sjsu.edu).
Digital Object Identifier 10.1109/TE.2010.2042451
0018-9359/$26.00 © 2010 IEEE
IEEE TRANSACTIONS ON EDUCATION, VOL. 54, NO.
• Digital: Use a Karnough map (K-map) to reduce the
number of logics gates required to implement a Boolean
logic function.
• Solid State: Explain what channel length means.
• Electromagnetics: Using Transmission Line Simulator
data, determine the wavelength of a signal.
• Circuits: Determine the gain of an inverting or noninverting
OPAMP.
• Systems: Given transient current and voltage data from a
system, determine the level of damping.
The student’s exam score counted for 10% of the student’s
final grade in the senior project class. The grading was done by
the faculty expert who wrote the items for the area.
After collecting data about student mastery of core EE con-
cepts (as measured by the skill audit exam), it was agreed by
the faculty that the mastery demonstrated by the students was
too low and that improvements needed to be made to improve
student mastery of core EE concepts. The means of the scores
for each core topic area of the skill audit exam by semester can
be seen in Table I. The overall means and standard deviations
for the Digital, Solid State, Electromagnetics, Circuits, and Sys-
tems areas for the 14 semesters shown were (46.5%, 6.7), (50%,
11.9), (34.4%, 11.4), (47%, 13). and (52.6%, 11.6), respectively.
Given that the faculty stated the exam items were important for
the students to master, these low scores were cause for concern.
II. IMPROVEMENTS TO THE SKILL AUDIT EXAM
Several changes were made to the skill audit exam to ad-
dress faculty concerns about the students’ mastery levels. These
changes had two independent goals. The first was to determine
in a non-resource-intensive manner what students, unprepared,
remembered from the curriculum. From this data, necessary im-
provements to the curriculum could be determined. The second
goal was to help students improve their mastery of these core
concepts before they graduated.
The improvements1 made to improve student mastery of core
EE concepts consisted of breaking the skill audit exam into
two parts. The first part consisted of an online exam with mul-
tiple-choice items that the students used to review core EE skills
and that measured student mastery without preparation. The
second part consisted of a 10-item in-class exam with questions
similar to, but different than, the online version of the exam.
The in-class exam measured student mastery levels after prepa-
ration. An additional component of the intervention was that stu-
dents had to earn a passing score on the 10-item in-class exam
(in this case 70%) to receive a grade in the senior project class
(EE198B). This made the exam into a Go/No Go (pass/fail) exit
1These were made with department faculty input.
PARENT: IMPROVEMENTS TO EE SKILL AUDIT EXAM
TABLE I
MEAN SCORES OF THE SKILL AUDIT EXAM FROM SPRING 1999 TO SPRING 2007 BY CONTENT AREA
Missing data.
exam [10]. The in-class exam was reduced from 50 items in the
original exam to 10 items from the original exam so that the
cost of administrating the test would be reduced. With the cost
of offering the test reduced, students were allowed to retake the
in-class exam until they passed.
The online exam was created with an automatic quiz gener-
ator, using the Blackboard software [11], to allow the students
to review the skills they were being asked to master and to mea-
sure student mastery levels without preparation. Students were
able to take the exam until they achieved the mastery level they
were comfortable with. Another advantage to offering the exam
online was that student performance by item could be easily cap-
tured and displayed. This data could then be used to study item
validity and improve the test.
The 10-question pass/fail in-class exam measures student
mastery of core EE skills after preparation, and the pass/fail na-
ture of the exam ensured that students will study until they pass
the exam. The underlying assumption that drove the decision
to make passing the in-class exam a graduation requirement
was that even though the students may have known the core EE
skills at one time, the students needed to be motivated to put
in the effort required to master these skills. Even though some
programs encourage motivation (buy-in) from the student by
setting the exit exam to be worth 66% of the total grade in a
project course [12], the department faculty determined it not to
be appropriate to give an exam so much weight in a hands-on
project course.
III. RESULTS OF THE IMPROVEMENTS
Mean scores for each content area from the online exam for
the first semester the intervention was implemented (Fall 2007)
and the last two semesters can be seen in Fig. 1. The sample
size for each semester was greater than 40. The results of the
Fall 2007 online exam match very closely with the results from
the previous 14 semesters using the 50 question in-class exam
for which the students were not able to prepare.
The average mastery level (as measured by the means of on-
line test scores) in all the core content areas increased in subse-
quent years since the online exam was first offered. After careful
185
Fig. 1. Student mastery levels for core content areas measured by the online
exam for three semesters.
analysis of the raw data provided by the automated quiz gener-
ator, the increased mastery level shown by the online exam re-
sults was likely due to the students taking the online exam more
often until a higher level of mastery was achieved. If the con-
tent areas were to be scored only taking into account the first
attempt of students in recent semesters, the means of the con-
tent areas would drop to those encountered in semesters before
the improvements were carried out.
In Fig. 2, item mean (closed boxes), and discrimination
(closed circles) are plotted against the item identifier. The
discrimination factor is an estimate of how well a particular
item can predict the performance of a student on the whole
exam. The item mean suggests the level of difficulty of the
item. In Fig. 2, it can be seen that the item “Systems 7” had a
discrimination factor of zero, which means that the item does
not predict the overall performance of the student on the exam.
This item also had one of the highest means. This indicates that
the item might be too easy and that it should be removed from
the exam.
The in-class exam results after the improvements were made
were very positive. All the students achieved at least 70% mas-
tery in the in-class exam; 76% of the students demonstrated
mastery on the first try, 23% on the second try, and 1% on the
third try. The trend of a few students taking more than once to
186
Fig. 2. Item mean (closed boxes) and discrimination (closed circles) from the
Spring 2009 semester. The items are arranged from the lowest discrimination
factor to highest.
demonstrate mastery in the in-class exam has continued to the
most recent semester. The students who did not pass the in-class
exam were the students who did not complete the online version
of the exam.
Prior to the intervention, there was some student resistance
to the idea of an exit exam due to the sensitive nature of high-
stakes exams. This resistance was assumed to be from a fear
of failure. The student resistance to the exit exam was reduced
by allowing the students to take and pass the exam early (after
taking the online exam). Interestingly, once they had passed the
exam, many students stated that the passing score should be set
at 90% or higher.
IV. MANAGING A GO/NO GO (PASS/FAIL) EXIT EXAM
Given that the skill audit exam was changed into a pass/fail
exit exam, and that exit exams such as the California High
School Exit Exam [13] can be controversial [14], efforts were
made to ensure that no one group of students was penalized by
implementing the pass/fail exit exam. Additional concerns with
respect to high-stakes exit exams are having the exam become
an end in itself (teaching to the test), maintaining a secure pool
of items and making sure that high pass rates are not achieved
by lowering the required mastery level.
Evidence that the exam has not become an unreasonable
burden on minority students is that all students, regardless of
minority status, have passed the exam since the changes were
made in Fall 2007.
Exam security is always a concern and is handled in several
ways. First, the motivation to cheat is removed because the on-
line exam is only used by the student to review; it does not count
for passing the in-class exam. Second, the mean scores for the
semester are also monitored. For instance, if the cheating was
occurring on the online exam, one would expect the means of the
student’s first attempt of the exam to increase. Third, items on
the in-class exam are modified to ensure the students are not re-
warded for memorizing answers. Memorization of answers has
been detected because students have been found to select the an-
swer that was correct on the online version of the exam but was
not the correct answer for the exam of record.
Even though the pass/fail exit exam intervention did increase
student performance, the passing score of 70% may be too low
IEEE TRANSACTIONS ON EDUCATION, VOL. 54, NO. 2, MAY 2011
[15]. The 70% pass rate was selected with the idea that students
might select the wrong answer on an exam by accidentally cir-
cling the wrong answer out of nervousness. Given that only one
student earned a 70% on the exam without preparation in all the
years the exam was offered, 70% seemed a high goal to achieve.
The problem with the 70% mastery level is that a student can
show a complete lack of mastery (fail) a content area of the exam
and still earn a passing score. Data analysis showed that students
who failed the in-class exam were those who did not take the on-
line exam. Most of the wrong answers were not a “typo,” but a
memorization of the wrong answer, or solving the problem in-
correctly. (In the in-class exam of record, students write on the
exam paper, so what they were thinking can sometimes be de-
duced.) Given this data, the passing score of the exam should
probably be raised.
V. STUDENT PERCEPTIONS OF THE SKILL AUDIT EXAM
A survey was given to students enrolled in the senior design
project both to determine if they felt that the effort required to
prepare for the skill audit exam was an effective use of their time
and to determine if the exam should be improved by increasing
the score required to pass it or by making the questions more
realistic.
The students answered the survey voluntarily and anony-
mously at the end of their senior project. The survey and student
responses can be seen in Table II. The survey was kept short and
consisted of three questions concerning the worth of preparing
for the exam and two questions concerning improvements to
the exam.
The students were almost evenly split between those who
thought that preparing for the skill audit exam was at least some-
what helpful and those who thought preparing for the exam was
not helpful (Item 1, Table II). Given that some student projects
are at a high level of systems integration, and most students pass
the exam two months prior to the start of their project, it is not
surprising that many students did not feel that the preparation
was not worth the time as it related to their project. These results
could indicate that more systems questions need to be included.
A majority of students felt that preparing for the skill audit
exam was at least somewhat helpful for job interviews (Item 2,
Table II). This result indicates that further refinements to the
skill audit exam should be made to prepare students for the
workforce, even though it is too soon to have feedback from
local industry professionals as to a difference in SJSU EE stu-
dents.
Interestingly, when the students were asked simply if
preparing for the skill audit exam was worth it (Item 3,
Table II), a large majority answered “yes” by a higher margin
than for the first two items measuring the skill audits exam’s
worth. This probably means that preparing for the skill audit
exam is good for the students who have not been captured by
the survey.
The results from items 4 and 5 (Table II) support the idea
that students feel the skill audit exam should be made more dif-
ficult to pass. A majority of students felt the questions should
be made more realistic and the passing score should be at least
80%. Given that items 2 and 3 strongly indicated that students
feel that preparing for the skill audit exam was worth spending
time on,2 and that items 4 and 5 indicated that the exam should
2with Item 3 weakly supporting the worth of the exam
PARENT: IMPROVEMENTS TO EE SKILL AUDIT EXAM
TABLE II
SKILL AUDIT EXAM SURVEY ITEMS AND RESULTS FOR GRADUATING SENIORS
be more rigorous, the exam requirements were changed for the
Fall 2009 semester. The passing rate was changed to 80% given
that this was the most cost-effective improvement that could be
made.
VI. CONCLUSION
The improvements made to the skill audit exam (which in-
cluded breaking up the old in-class exam into two parts: an on-
line exam for review and an in-class 10-item exit exam) has al-
lowed the students to increase their mastery level of core EE
concepts immediately before they enter the workforce or grad-
uate school while still allowing the department to gather infor-
mation on how well the students remember core EE concepts.
The online exam allows the gathering of information in a non-re-
source-intensive manner. At the time of writing, two years after
the improvements were implemented, there is no detectable stu-
dent resistance to the idea that they should demonstrate mastery
though an exit exam.
More improvements to the skill audit exam need to be made.
First, the item bank should be expanded. Second, the item level
could be used to develop items with higher discrimination scores
and diverse difficulty levels. Each item’s discrimination factor
and mean can be used to develop exams with high discrimina-
tion factors and diverse difficulty levels. Third, a higher passing
score should be set to encourage mastery of EE core concepts.
ACKNOWLEDGMENT
M. Adams is gratefully acknowledged for his help setting up
the Blackboard software, as is T. Hudson for her constructive
editing help.
REFERENCES
[1] “ABET,” May 11, 2009 [Online]. Available: www.abet.org
187
[2] M. Mendelson and R. Noorani, “Improving undergraduate engineering
education in the US and abroad using EC 2000,” in Proc. Int. Conf. Eng.
Educ., Taipei, Taiwan, Jan. 15, 2000.
[3] “EE 198 senior design project resource home page,” May 11, 2009
[Online]. Available: www.engr.sjsu.edu/dparent/ee198
[4] “Assessment for curricular improvement,” May 11, 2009 [Online].
Available: www.engin.umich.edu/teaching/assess_and_improve
[5] “FE exam,” May 11, 2009 [Online]. Available: http://www.ncees.org/
Exams/FE_exam.php/
[6] M. Rao, S. Junaidu, T. Maghrabi, M. Shafique, M. Ahmed, and K.
Faisal, “Principles of curriculum design and revision: A case study in
implementing computing curricula CC2001,” in Proc. 10th Annu. Conf.
Innov. Tech. Comput. Sci. Educ., Portugal, 2005, pp. 256–260.
[7] K. Sanders and R. McCartney, “Program assessment tools in computer
science: A report from the trenches,” in Proc. 34th SIGCSE Tech. Symp.
Comput. Sci. Educ., Reno, NV, 2003, pp. 31–35.
[8] J. Bishop, “The effect of curriculum-based external exit exam systems
on student achievement,” J. Econ. Educ., pp. 171–182, 1998.
[9] J. Bishop, “Are national exit examinations important for educational
efficiency?,” Swedish Econ. Policy Rev., vol. 6, pp. 349–398, 1999.
[10] “Soldier’s Manual of Common Tasks—Warrior Skills Level 1,” May 11,
2009 [Online]. Available: http://www.scribd.com/doc/2472749/Army-
stp21-1-Soldiers-Manual-of-Common-Tasks-Warrior-Skills-Level-1
[11] Blackboard May 11, 2009 [Online]. Available: http://www.blackboard.
com/
[12] J. Liu, J. Marsaglia, and D. Olson, “Teaching software engineering to
make students ready for the real world,” J. Comput. Sci. Coll., vol. 18,
pp. 43–50, 2002.
[13] K. J. Sullivan, “In new study, high school exit exam gets a failing
grade,” Apr. 21, 2009 [Online]. Available: http://storybank.stanford.
edu/stories/in-new-study-high-school-exit-exam-gets-a-failing-grade
[14] B. Jacob, “Getting tough? The impact of high school graduation
exams,” Educ. Eval. Policy Anal., pp. 99–121, 2001.
[15] J. Betts, R. Costrell, H. Walberg, M. Phillips, and T. Chin, “Incentives
and equity under standards-based reform,” Brookings Papers Educ.
Policy, vol. 2001, pp. 9–74, 2001.
David W. Parent (S’96–M’99) received the Ph.D. degree in electrical engi-
neering from the University of Connecticut, Storrs.
Currently, he is an Associate Professor with the Electrical Engineering De-
partment, San José State University (SJSU), San Jose, CA, where he teaches
CMOS processing and device physics and conducts bio-interfacing research.