2nd International Conference on Electrical and Information Technologies ICEIT’2016
Towards an automatic assessment system in
introductory programming courses
Soundous Zougari, Mariam Tanana, Abdelouahid Lyhyaoui
Lab. LTI ENSA of Tangier, University Abdelmalek Essaadi, Tetouan, Morocco
Abstract—Nowadays, automatic assessment of programming
assignments has become an important topic in academic
research. In fact, the increasing number of enrolled students in
programming courses created the need of a system providing
immediate feedback to the learners and saving teachers from
manually managing all the students’ solutions. This paper
presents an attempt to conduct programming exercises
assessment both dynamically and statically to ensure a reliable
and objective evaluation job. While dynamic analysis is based on
xUnit framework, the static analysis is performed on the graph
representation of the programs.
Keywords—programming assessment; computer aided
assessment; graph representation
I. INTRODUCTION
The research work related to the TEL “Technology
Enhanced Learning” gathers in a transdisciplinary scientific
field, and wonders about the way of developing computing
tools to support, improve or even increase the learning, whether
it is face-to-face or remote [1]. The handled issues concern for
example the design, the implementation, or the learners’
assessment in the TEL. In this sense, this work presents a
reliable and objective method of assessing learners’
productions that not only will reduce the workload for teachers
but also provide useful feedbacks to students throughout their
learning process. Concerning the practical domain, we opted
for introductory programming courses for several reasons.
Besides the fact that these courses are the core of any
engineer's training, this is a domain where assessment is of a
great complexity, mainly because it is characterized by the
multitude of solutions to a given problem.
This contribution is organized as follows: Section 2
presents necessary background information. Afterwards, we
give an overview of the main methods adopted by program
analysis systems in section 3. Section 4 describes the proposed
approach, with a special emphasis on the programs’
transformation into intermediate graph structures. Section 5
concludes the paper and outlines directions for future work.
II. THE RESEARCH STATEMENT
To learn and master a new programming language, students
need to solve a large number of exercises in order to practice
the new syntax and semantics of the language. The comments
and feedback from teachers about the mistakes they made are
crucial to improve their knowledge. However, it is difficult for
teachers to manually manage all students’ solutions. Indeed,
correcting manually programming exercises can involve a lot
978-1-4673-8469-8/16/$31.00 ©2016 IEEE
of work and consume much time. It is often a complex and
daunting task since each program must be tested and its source
code analyzed. In addition, the correction process is prone to
errors or omissions due to the fatigue and the repetitive nature
of the task [2].
Furthermore, the advantages of the automatic assessment
are especially appreciated in the context of e-learning [3].
Several universities worldwide offer numerous online courses
and the number of students enrolled in these courses is on the
order of thousands. In online courses, the teaching process is
carried out via the computer, with minimal or no contact with
the teacher. Therefore, fast and reliable automated assessments
are particularly desirable. All these reasons and so many more
have motivated many researchers to be interested in automating
the process of assessing learners’ productions. The first
reference comes from Hollingsworth who published on the
subject in 1960 [4]. The idea caught on quickly and several
assessment systems have been developed [5]. Unfortunately,
these systems are neither generic nor configurable and most of
them are not available to the general public. That is why we
seek to develop our own assessment system.
In what follows we will focus on the approaches mainly
adopted by these systems, namely dynamic and static.
III. PROGRAM ANALYSIS METHODS
The validation of computer programs is a crucial part in
the cycle of their development. Two verification and
validation techniques have stood out in recent years: dynamic
analysis and static analysis. The main difference between
these two approaches is that the dynamic analysis requires the
execution of the program to check its accuracy, unlike the
static analysis that examines a program without executing it.
This section aims to present them:
A. Dynamic Analysis
Dynamic analysis involves running the code to verify its
accuracy. This is achieved by using different and varied test
cases allowing maximum path coverage of the programs. It is
intended to detect errors by comparing the gotten results with
those expected by the specification. This method is adopted by
most of the automatic programming assessment systems such
as Ceilidh [6], TRY [7], BAGS [8], Kassandra [9]. There are
two different approaches in the dynamic analysis method:
• Black-box testing: testing is carried out without
knowledge of the internal structure and the objects
handled by the system. We are only interested in the
 2nd International Conference on Electrical and Information Technologies ICEIT’2016
inputs and outputs of the system. Functional testing or
manual testing, for example, belongs to this category.
• Grey-box testing: consists in validating the results of
each function in the program and generating the final
score compared to these results.
1) Advantages of the dynamic analysis:
• The dynamic analysis is easy to implement, and tester
can be non-technical.
• It makes possible evaluating any program performance
in terms of the generated results compared with the
expected outputs in the test case.
• It allows analysis of applications in which we do not
have access to the actual code.
2) Disadvantages of the dynamic analysis:
• Risks related to the execution of the source code, e.g. a
buffer overflow that can cause a sudden breakdown of
a server and put the data at risk.
• The major drawback of this method is that if the
program does not compile, then it cannot be assessed.
• The generated feedback is limited to the expected
outputs of the test case.
• We cannot check the conformity of the program as
for the requirements defined by the instructor.
B. Static Analysis
The static program analysis is a family of techniques
allowing to collect information on the program without having
to run it and therefore eliminate the risks associated with its
execution. Among the tools that rely on static analysis lint [10]
and AutoLEP [11]. We can distinguish various methods within
the approach of static analysis:
• Style analysis: analyses program readability
(significant variable names, the presence of comments,
the indentations, the scope of variables, etc.) so that the
program can be understood by other users. Among the
systems using program style analysis we find Style ++
for C ++ [12].
• Error detection: the errors detected through this method
are imperceptible during compilation time, but cause
problems when running (division by zero, infinite
loops, etc.). These errors or these bugs may contribute
to a blockage or a malfunction in the system.
• Metric analysis: measures the program properties to
test its complexity and reliability (average size of
instructions, frequency of comments, number of
instructions in a function, number of classes, number of
functions, etc.).
• Keyword analysis: examines the source code in order
to determine the presence of the words required by the
evaluator in the source code, for example a program
wherein the evaluator requires the use of the switch
structure and which can be written using a set of if.
This method gives the possibility to check whether the
program meets the requirements designated by the
evaluator.
• Structural analysis: measures the degree of similarity
by comparing the assessed program to the solutions
expected by the teacher or expert. This method as a
static analysis method consists in measuring and
calculating the degree of equivalence between
programs through their graphical representation.
1) Advantages of the static analysis:
• It can find weaknesses in the code at the exact location.
• It takes into account all possible execution paths.
• The ability to analyze the program even if the code
contains errors as opposed to dynamic analysis.
2) Disadvantages of the static analysis:
• The limitations of structural analysis since there are a
variety of solutions for the same problem.
• Intricate and difficult to apply in the context of
complex programs.
• Automated tools can produce false positives and false
negatives.
IV. OUR PROPOSITION
We can deduce that the strengths and weaknesses of the
two approaches are complementary. We therefore propose an
original combination of these two techniques. In this
combination, the dynamic analysis reports errors at runtime,
then the static analysis evaluates the structural properties of
the programs.
To perform dynamic analysis, we suggest the use of a
dedicated framework to automate and conduct tests in a given
language. This not only allows separating the test code from
the code, making possible testing it and thus facilitates its
reuse, but also to do it without a manual intervention and a
human interpretation. Finally, the analysis of the gotten results
could be automated since each test result has a status,
generally ok or error [13]. This is addressed in section A.
On the other hand, to evaluate the structural properties of
the programs (static analysis), we measure the degree of
similarity by comparing the assessed program to the programs
belonging to the solution space provided by the teacher or
expert. A solution space is a set of paths representing the
different possible approaches for the same exercise. It can
contain the correct solutions as well as the incorrect ones. It is
made by an expert and has deemed pedagogically interesting
approaches. This method requires the passage through the
graphical representation of the compared programs. More
details are given in section B.
A. Dynamic Analysis with xUnit
Our aim being to remain as generic as possible, we chose
the xUnit framework. Indeed, it is a family of several similar
frameworks gathered in a family named xUnit. The JUnit
 2nd International Conference on Electrical and Information Technologies ICEIT’2016
framework was the first to be widely known but different
development platforms and programming languages followed
including nUnit (.Net), Dunit (Delphi), CppUnit (C ++), ... We
will explain in what follows some of the JUnit basic features
and architecture components shared with other frameworks of
the xUnit family.
JUnit is an open source framework for the development
and implementation of automated unit tests created by Kent
Beck and Erich Gamma [14], [15]. The main interest is to
ensure that the code still meets the needs even after possible
modifications. JUnit offers:
• A framework for the development of unit tests based on
assertions that test the expected results.
• Applications allowing the tests completion and the
results display.
The tests to automate are expressed in classes in the form
of test cases with their outcomes. JUnit runs these tests and
compares their outcomes with the expected results. That’s how
the class code is separated from the code that allows testing it.
Often to test a class, it is easy to create a main( ) method
containing the treatment of the tests. The downside is that this
superfluous code will be included in the class. Moreover, it
must be executed manually.
Writing test cases can have an immediate effect on
detecting bugs but mostly it has a long term effect that
facilitates the detection of side effects during modifications.
JUnit test cases are Java classes that contain one or more test
methods and that are grouped into test suites, as shown in
Fig.1.
Fig. 1. JUnit Architecture.
B. Static analysis
In order to statically analyse computer programs, it is
useful to choose a representation that is accurate but still
abstract with respect to the programming language. There are
different forms to represent and visualize a program:
1) Syntax Trees: an Abstract Syntax Tree or AST [16] is a
tree whose internal nodes are labeled by operators and whose
leaves are the operands. In other words, generally, a leave is a
variable or a constant. This representation is commonly used
in the search for similarity between different programs or
within the same program. After creating the AST, we look for
similar subtrees within the generated tree via some algorithms.
Eventually, we return the matched trees.
2) Graphs : graphs are another way for representing
programs. Through the graphs you can display the program
structure in order to analyse and compare it with other
programs [17]. There are several types of graphs:
a) The Control Flow Graph (CFG) is a directed graph
where each node represents a basic block i.e. a straight-line
piece of code without any jumps or jump targets; jump targets
start a block, and jumps end a block. Directed edges are used
to represent jumps in the control flow. It highlights loops,
conditional statements and branches. A path in this graph
represents a program implementation scenario. The program
illustrated in Fig. 2 is used to provide an example for all the
type of graphs. Fig. 3 displays the program corresponding
CFG.
void main() {
int x = 0;
int y = 1;
while (y < 10) {
y = y * 2;
x = x + 1;
}
printf(“%d”,x);
printf(“%d”,y);
}
Fig. 2. Program example.
Fig. 3. Control Flow Graph.
b) The Control Dependence Graph (CDG) shows which
instructions will be executed based on the value of an
expression in the program. The nodes of the graph are the
same as those of the control flow graph. An edge goes from
two nodes p and q, if the value of the p expression has an
impact on the execution of the q instruction. See the
corresponding CDG for our program in Fig. 4. In this
example, we may note that the presence of the loop on the
expression y <10 is due to the fact that it must be re-evaluated
if true, in other words until it is false.
c) The Data Dependence Graph (DDG) is a directed
graph representing data dependencies of several objects
towards each other. It contains the same set of nodes in the
previous graphs. The edges incident to a node represent data
values on which the node’s operations might depend. In fact,
 2nd International Conference on Electrical and Information Technologies ICEIT’2016
an edge exists between two nodes p and q in a program if p
defines a variable and q uses it. Fig. 5. shows the DDG for the
above mentionned program.
Fig. 4. Control Dependence Graph.
Fig. 5. Data Dependence Graph.
We can deduce that each and every graphical method for
program representation supports some unique features. Since
our intention is to assess students’ productions in introductory
programming courses, we choose to represent programs with
control flow graphs. Indeed, in the flow control approach, the
focus is on the sequencing of operations in a process. Control
flow graphs are used as models to describe the structure of
computer programs. They are used both for static analysis [18]
and as a model for program coverage. Therefore, it’s a suitable
representation for structural comparison. However, we will not
definitively exclude the other types of graphs because they can
be interesting for future modifications in our system.
V. CONCLUSION
In this paper, we propose a new assessment system for
students’ productions aiming introductory programming
courses. The system is based on merging information from
two different evaluation methods, dynamic and static analysis.
The first is carried out using the xUnit framework, which
provides features that do not only ease the dynamic analysis
process but also makes it flexible and reusable.
The second analysis evaluates how close a student program
is to the expert solutions. This process consists of two parts:
the transformation of student program into an intermediate
representation and the examination of similarity
measurements. The intermediate representation in our
approach is based on the Control Flow Graphs (CFG) and the
similarity measurements will be our concern in the next stage
of our study. We will focus on the most accurate similarity
measure technique for our domain of interest, in order to get a
reliable similarity percentage between students’ answer and
the instructor solutions.
As soon as the new system is developed, we intend to test
it with real users. We plan to design and implement an
experiment in real learning environments to assess the
usability and performance of the proposed system. This
experiment will also allow us to evaluate its weaknesses and
therefore improve it.
REFERENCES
[1] P. Tchounikine, "Précis de recherche en ingénierie de EIAH", Juin 2009.
(references)
[2] Higgins, S., Hall, E., Baumfield, V., Moseley, D. (2005). A meta-
analysis of the impact of the implementation of thinking skills
approaches on pupils. In: Research Evidence in Education Library.
London: EPPICentre, Social Science Research Unit, Institute of
Education, University of London.
[3] Allen, I.E., & Seaman, J. [2010]. Class Differences: Online Education
in the United States, 2010. Newburyport, MA: Babson Survey Research
Group and The Sloan Consortium Green.
[4] J. Hollingsworth. Automatic graders for programming classes.
Communications of the ACM, 3:528–529, October 1960.
[5] C.Douce, D.Livingstone, and J. Orwell. 2005. Automatic test-based
assessment of programming: A review. Journal on Educational
Resources in Computing (JERIC) 5, 3 (2005), 4.
[6] S. Benford, E. Burke, E. Foxley, N. Gutteridge, and A. M. Zin.
Experiences with the Ceilidh system. In Proceedings of the International
Conference in Computer Based Learning in Science, Vienna, 1993.
[7] Kenneth A. Reek. The TRY system -or- how to avoid testing student
programs. In Proceedings of the twentieth SIGCSE technical symposium
on Computer science education, SIGCSE ’89, pages 112–116, New
York, NY, USA, 1989. ACM.
[8] J. B. Hext and J. W. Winings. An automatic grading scheme for simple
programming exercises. Commun. ACM, 12(5):272–275, May 1969.
[9] Urs Von Matt. Kassandra: the automatic grading system. SIGCUE
Outlook, 22(1):26–40, January 1994.
[10] Ian F. Darwin, « Checking C Programs with Lint », O'Reilly Media,
October 1988.
[11] Wang Tiantian; Su Xiaohong; Ma Peijun; Wang Yuying; Wang
Kuanquan, "AutoLEP: An Automated Learning and Examination
System for Programming and its Application in Programming Course,"
in Education Technology and Computer Science, 2009. ETCS '09. First
International Workshop on , vol.1, no., pp.43-46, 7-8 March 2009.
[12] Ala-Mutka, K., T. Uimonen and H.-M. Järvinen (2004). Supporting
students in C++ programming courses with automatic program style
assessment. Journal of Information Technology Education, 3, 245–262.
[13] HTTP://WWW.JMDOUDOUX.FR/JAVA/DEJ/CHAP-FRAMEWORKS-TEST.HTM
[14] https://fr.wikipedia.org/wiki/JUnit
[15] Petar Tahchiev (Author), Felipe Leme (Author), Vincent
Massol (Author), Gary Gregory (Author), JUnit in Action, Second
Edition, 2011.
[16] Ronald Cohn Jesse Russell, Abstract syntax tree Paperback – 1 Jan
2012.
[17] Vincent Mathieu. Outils d’analyse statique. 2001.
[18] S. Rao Kosaraju. Analysis of structured programs. J. Comput. Syst. Sci.,
9(3) :232-255, 1974.