2015 International Conference on Data and Software Engineering

A Domain-Speecific Language for Auto

omatic
Genneration of Checkers

Ryan Iggnatius Hadiwijaya, M.M. Inggriani Liem

Schoool Of Electrical Engineering and Informatics
Bandung Institute of Technology
Jl. Ganesha 10 Bandung 40132, Indonesia
ryann.ign54@gmail.com, inge@informatika.org

Abstract— One of the important moduless of a black-box

automatic program grader is a “checker”. In programming
competition environment, a checker is a prograam written for the
purpose to check the output of the contestantt’s program for a
task that has many solutions. Usually, a ch hecker is written
manually as needed. In this paper, the idea of th
he output checker
in the programming competition environmen nt is extended to
input checker and source code checker as a partt of the automatic
grader in our programming learning environmeent. Input checker
validates the input coverage. The source code cchecker is used to
validate a set of properties from a source code against the given
coding specification. A Domain-Specific L Language (DSL)
grammar is designed and developed as a speecification for the
automatic generation of the output, input, and source code
checkers. The DSL grammar and the checker ggenerator tool set
are used to evaluate source codes in our programming class. By
writing the checkers specification in DSL, th he specification is
automatically documented and can be reu used for similar
properties.
Keywords—automatic program grading; Domain-Specific
Language; property checker
I. INTRODUCTION
Our programming classes at Schooll of Electrical
Engineering and Informatics, Bandung Institutte of Technology
(SEEI ITB) have been using an automatic gradding system since
four years ago [2]. The automatic grading syystem is adapted
from the programming competition system [22]. The automatic
grading process is faster and more reliable com mpared to manual
grading system. The students can do many moore exercises and
have feedback immediately. The system gradees the source code
submitted by students with a black-box meethod, where the
source code is executed with test cases. The system compares
the output of the student’s program with the expected output.
The workflow of the current automatic graading process is
shown in Figure 1. The problem setter iss a person who
prepared the task (programming problem) desscription, the task
solution, and a test suite for the task. A test suite consists of
input files, output files, score files, and an outpput checker.
The black-box method is sufficient for autoomatic grading in
the programming competition environment w where the quality
of the source code is not examined. Hoow a contestant
implements the code is not a part of the gradingg process.
Figure 1. Workflow of Current Auttomatic Grading Process
However, the black-box meth hod is not sufficient for
automatic grading of the tasks in the programming learning
environment. The evaluation of the source code is an important
aspect of grading. The students should implement a program
respecting the coding conventtion and detail coding
specification. The black-box method d is not able to check these
kinds of constraints in the source code.
A single programming exercise can have many versions of
correct solutions. As a classic exam mple, students are asked to
write a program in a given program mming language to read an
integer N, and then calculate the facctorial of N (N!), where N is
an integer greater than zero. The sollution of this exercise could
be either a main program with a loop, a function, or a
procedure to calculate N!. As an n exercise in recursion, a
lecturer may define the implementtation specifications, where
students are asked to implement it by writing a recursive
factorial function. By using the black-box grading system, the
student’s program is graded as a corrrect answer if the execution
of the source code can deliver N! for a given test case of N.
However, a black-box grading system judges a source code that
computes N! by a loop and a recurssive function as the same, if
the execution result is the same. Fo or identifying the recursive
function, we have to check the sourrce code and if the students
do not implement a recursive fun nction, then the solution is
graded as a wrong answer, or at leasst the grade is penalized.
Usually, these kinds of simple exercises
e are given in basic
programming classes where we hav ve hundreds of students (we
have 400 students in Introductory Programming
P class and 150
students in Object Oriented Prograamming class) with two to
four exercises per weeks [2]. It iss impossible to check each
source code one by one manually. Therefore, we aim to

978-1-4673-8430-8/15/$31.00 ©2015 IEEE 7

 2015 International Conference on Data and Software Engineering
implement an automatic source code checker as a complement
to a black-box grading.
In a programming competition, contestannts are asked to
solve complex tasks by writing a program m. The output is
examined by an output checker for a taskk that has many
possible correct answers. Currently, the ouutput checker is
written manually as needed. The output checcker source code
cannot be reused for another task. The test casses are also made
manually or generated by a specific test case generator. Most of
the tasks are specific; some of them are classiffied into problem
classes or categories, such as knapsack probblem and greedy
problem. The checker can be reused and re-ggenerated for the
tasks of the same class of problems. In [1], test cases are
designed and specified by using a DSL. The black-box
evaluation is improved by the Metamorphiic Testing (MT)
method [7]. MT is a technique to generate follow-up test cases
based on the existing test cases by makiing reference to
Metamorphic Relations (MR). Then, MT checcks the properties
expression in MR. This MT system is adaptted to a learning
environment. By using the system, the inpput checker can
increase the quality of the test cases and the ouutput checker can
enhance the black-box evaluation by Metaamorphic Testing
method [1].
The problem statement and our propossed solution are
presented in Section 2. Section 3 of this paper gives some
relevant works in the domain. In section 4, thee development of
DSL is presented as well as the work oof the proposed
generator. Section 5 illustrates some exam mples where the
proposed system is used in our context. We coonclude this paper
by discussion and conclusion.
II. PROBLEM STATEMENT AND PROPOSED
D SOLUTION
In current programming practices, only ooutput checker is
used to check the output. However, output checker is made by
ad-hoc. In the programming learning environnment, the input
checker is also needed to validate the input teest cases and the
source code checker is needed to check thee implementation
specification.
In this research, we defined a single checkker generator that
can be used by a problem setter to specify the input, output and
source code checkers, based on a single Domain-Specific
Language (DSL). DSL is a language that is ussed for a specific
domain. DSL contains syntax and semantic ffor modeling the
abstraction of the domain [6]. In our research,, the domains are
the checkers. Checkers specification is writtenn as a script using
the DSL grammar.
Then, the checker(s) source code is generatted automatically
by our system (see the workflow in Figure 22) from the DSL
specification script. The checkers are used as a part of the test
suite and improve the current checker in the tesst suite.
In a learning programming class, the inputt, the output, and
the source code are needed to be checked against a set of
specification. To have a self-documented checker, the
specification is written in a DSL.
Given a programming exercise (input, outpput, and solution),
and a set of properties defined by a problem setter, the
generated checkers validate the programmingg exercise against
the properties and give a score. The specifications of the
checkers (output checkers, input checkers,
c and source code
checkers) are expressed in a prop
perty based DSL, and then
generated automatically.
Figure 2. Workflow of Cheecker Generation
The input and the output depend on the given format. The
problem is that source code is dependent
d on the language
which is written. The research qu uestion is how to write a
checker independently from the prog
gramming language.
III. RELATED WORKS
Some automatic source cod de evaluators have been
implemented for examining staticaally (without executing) a
source code, such as a plagiarism detector
d [3], static analyzer
[4], and code smell detection [5]. The
T source code plagiarism
detector calculates the similarity bettween student’s source code
and a program solution. The system m defines a similarity rating
and considers some variations su uch as the replacement of
variable names, changing the loop ty ype, or alternating the cases
in an if/switch statement [3]. Thee automatic static analyzer
grades student’s program without executing
e it, to detect some
flaws and give a penalty for each h flaw [4]. The code smell
detector is used to detect bad smelll, that is used to transform
into a penalty factor to the source co
ode [5]. These three types of
source code evaluator cannott detect implementation
specification.
In this paper, we propose anoth her way to check a source
code with the property checking metthod. The correctness of the
execution has been judged by a blacck-box method. This way is
more effective for grading many sim
mple and short source codes
as we need in our environment. By property checking method,
we don’t need to check the wholle source code as in other
white-box methods that have been mentioned
m above.
One example of property based testing used is QuickCheck
[8]. A property based testing is a testing method where
boolean functions are used to test t the correctness of a
program. The program is correct if all of the boolean functions
return true for all possibilities off arguments. If a boolean
function return false, then QuickCheck delivers a
counterexample. In our research h, we used a predefined
properties for checking the source code, not to determine the
correctness of the program. The prroperty checking method is
the same; by using boolean functionns.
There are also many source co
ode analysis tools, such as
StyleCop and FxCop. StyleCop is an open source static code
analysis tool from Microsoft that cheecks C# code. The StyleCop
tool analyzes the style and consisttency rule violations in the
source code [11]. FxCop is a free staatic code analysis tool from
8
 2015 International Conference on Data and Software Engineering
Microsoft that analyzes managed code asssemblies. FxCop
concerns with the violations of the programm ming and design
rules set in the design guidelines. FxCop proviides a tool to help
developers to follow their company's codingg standards [12].
But, both of the StyleCop and FxCop check coding style and
coding standard of a program. They do not ccheck a program
based on the implementation specification.
Domain-Specific Language has been uused to generate
program for a specific domain because of its ssimplicity and its
well known vocabulary by domain experts. In [9], the DSL has
been used to define use cases. Test casees are generated
automatically based on the use cases that havve been specified
in the DSL.
A test case generator proposed by [1], alsoo uses a DSL for
writing the test case and checker specifi fications for the
programming competition task. The checcker checks the
property based on the Metamorphic Relation. The purpose of
the system is to make a test case and checkker creation more
systematic. In this paper, we adopt the DSL foor a more generic
checker for programming learning class. Thee purpose of the
system is to enhance the quality of black-boxx grading and to
check the implementation specification by deffining a property.
A Metamorphic Relation is used as a propeerty that will be
checked.
IV. A DSL FOR PROPERTIES CHECKER
In our previous paper [1], we havee designed and
implemented a DSL as a language to expreess test case and
input-output checker. The specification is ussed to generate a
test case generator and also input-output checkker. Input checker
is used to check the input of the generated teest cases. Output
checker is used to check the output propertiess and the relation
(metamorphic relation) between outputs. Testt case (input and
output) has a specified format. The input form
mat is used to read
and write input, and output format is used too read and write
output. The checker needs to read input annd output before
checking their properties. The test case geenerator uses the
formats for reproducing test cases.
In this research, the previous DSL is analyzzed and extended
so that it can support a generic checker, comprises input,
output, and source code checker. Source coode is a program
written in a programming language. It is not a data and has a
specific structure depends on the program mming language.
However, the source code can be considdered as a text
independently to a programming language. Thhe data (input and
output) can also be considered as a text. Thherefore, text is a
generic representation of the object to be checked. The
generalization of the object to be checked is shhown in Figure 3.
The checkers generated by our new system m check a text. If
the text is data, then the text is read by usiing the specified
format. Otherwise, if the text is a source codee, then the text is
read as a list of strings. The existing DSL in [11] already had the
capability to read and write data (input and ooutput) according
to data format. It is now reused and extendeed to support any
text, including source code. Therefore, we genneralize the DSL
to support source code checking.
When a source code is processed by prograamming language
syntax, a lexer transforms a source code innto tokens. Each
token is a keyword in the languaage. Then, the tokens are
arranged in a parse tree. To support the programming language
independency, we do not processs a source code by this
approach. Instead of having an AST T, we transform the source
code into a list of strings, where eaach string is a semantic unit
such as statement, function, or a chhunk of source code such as
comments. By this approach, we trreat all source codes in any
languages as strings. We check thee source code by providing
specific properties which contain n programming language
keywords that will be checked.
Figure 3. Generalization of the Object
O to be Checked
The DSL should accommodaate a definition of string
delimiter and program comments. String delimiter is used to
transform a source code into a list of strings for each type of
chunk such as statement, function, or procedure existing in a
source code. A source code can contain comments. Some
comments can be checked (for example, to check if the
students make a good statement of initial state, final state, and
process description). But, some pro ogramming exercises ignore
the comments. In this case, the comments
c are ignored and
eliminated from the list of the strinngs. Properties are checked
by a string matching algorithm, to o the list of strings of the
source code. The string matching can vary from very rigid
(exact match) to the existing of a pattern.
p Regex matching is
used to search a pattern in the source code. The underlying idea
of the proposed source code checker is independency from any
programming language based on string matching, without
considering the grammar of the prrogramming language. The
regex pattern is automatically generated from property
expression in the DSL.
The DSL is used to define the property
p to be checked and
also the score of the existence or non-existence
n of a property.
For example, a user can define two t properties P1 and P2,
followed by their associate scores to check outputs. Property
P1 has a score equal to 30 and property P2 has a score equal to
70. The generated output checker checks P1 and P2 in the
outputs. If P1 is not satisfied, then th
he score is reduced by 30. If
P2 is not satisfied, then the score is reduced
r by 70. Therefore, if
both P1 and P2 are not satisfied, theen the judge gives a zero (0)
score.
To design the DSL grammar for f this research, we have
analyzed the checker domain. The checker domain consists of
the checker code and the object to o be checked. The checker
code is a source code that is geneerated automatically by the
DSL. The object to be checked is generalized
g as a text that is
shown in Figure 3. The checker do omain analysis of the object
to be checked is represented in Figu
ure 4. The class diagram in
9
 2015 International Conference on Data and Software Engineering
Figure 4 reflects the top level DSL grammar. The complete
grammar is available in our online repository1.
Figure 4. The Checker Domain Analysis for the Object to be Checked
The input and the output have variables. DataFormat is
used to extract the variable values from the input and the output
files. DataFormat has one or more FormatExpressions which
expresses each line of the input/output. The input and the
output can have many Properties. The Property has one or
more BooleanExpressions which define a relation between
variable values in the input/output. The Property is satisfied if
at least one of the BooleanExpession returns true. The input
and the output also have a Checker. A Checker consists of
one or more PropertyUsages. A PropertyUsage is a
declaration to use one or more Properties to check files
specified in the Filenames. The Score is given to the
input/output if all of the Properties specified in the
PropertyUsage are satisfied.
On the other hand, a source code consists of a list of strings.
SourceFormat is used to split the source code into the list of
strings. The source code can have many Properties. A
Property for the source code has one or more
RegexExpressions that is used to check the existence or non-
existence of the regex pattern in the list of strings. The
Property is satisfied if at least one of the RegexExpession
returns true. The source code also has a Checker. The
Checker in the source code is the same as the Checker in the
input/output.
The methodology for DSL and generator development is
similar to the methodology used in [1]. Instead of generating
test cases, the new generator in this research generates checkers
(input checkers, output checkers, and source code checkers).
The checkers can be run by a driver in a standalone or in the
existing automatic grading system in our programming learning
environment. The checkers are generated in Java language, to
gain the advantage of the Java platform that is “code (generate)
once, run everywhere” [10] and we can also gain the advantage
of the built-in string / regex matching function in Java.
The DSL script contains four declarations, that are variable
1
https://github.com/ryanignatius/GenericCheckerDSL
2
http://codeforces.com/problemset/problem/472/A
declaration, format declaration, property declaration, and
checker declaration. We have provided the generator’s code
that is used to generate code from the DSL Script. The
generation process is supported in the XText framework [13] by
defining a IJvmModelInferrer. The XText framework is also
used in our previous research [1]. The IJvmModelInferrer is
used to generate the checkers from the DSL Script. The
IJvmModelInferrer contains a specific code to generate each
element in the DSL. Each element of the DSL is generated as
an attribute or a method in the generated checker. The
generated checker has a class that is specified in the DSL
Script.
The Variable Declaration contains a list of variables
and its type. For each variable in the DSL Script, our system
generates a private variable, its public getter and setter, and a
function to read and write the variable depends on the variable
type.
The Format Declaration can be used to describe data
(input and output) format or the source code format. Our
system generates a public function for reading or writing the
data or the source code based on the DSL script.
The Property Declaration contains a list of predicates
and its identification number. For each property, our system
generates a public function to return a boolean value. The
function checks the property represented by one or more
relations as specified in the DSL. If at least one of the relations
in the property is satisfied, then the function returns true,
otherwise false.
The Checker Declaration contains pairs of
<properties, files> that are the type of the files (input, output,
source code) where the properties are checked. For each
checker, our system generates a public function to return an
integer. The function checks the properties in the list of files
given in the DSL. Property returns true if the source code meet
the specification, otherwise false. If a property returns false,
then the function returns zero (0). If all properties return true,
then the function returns an integer representing the score,
obtained from the DSL script. There are two types of the
checking type, by using “all” keyword and “each” keyword.
“All” keyword is used to check a set of files. The score is given
to that set of files. “Each” keyword is used to give a score to
each file individually.
V. CASE STUDIES
The DSL in this research has been implemented and tested,
which is then to be used in case studies. Each case study
consists of some examples of utilization. Examples presented
briefly on this paper are taken from a programming assignment
class. The complete implementation of the DSL, the generator
code, and the examples are accessible from our online
repository1.
We give an example to write a checker specification in the
DSL by writing a very simple checker to check a programming
task of codeforces2. In this task, an output checker is needed to
check if two integers x and y satisfy a property x + y = n (n is
given as input), 1 < x,y < n, and both of x and y must be a
composite number (greater than 1 and not a prime number).
10
 2015 International Conference on Data and Software Engineering
The value of n needs to be checked if it satisfies the property
stated in the task: 12 < n < 106. The DSL Script to define
checkers for this task is shown in Code 1.
class Example {
int n,x,y
format : input { 1(n) }
format : output { 1(x,y) }
P 1 { def int z = x+y; z == n }
P 2 { 1 < x }
P 3 { x <= n }
P 4 { not prime(x) }
P 5 { 1 < y }
P 6 { y <= n }
P 7 { not prime(y) }
P 8 { 12 <= n }
P 9 { n <= 1000000 }
check : output { P [1,2,3,4,5,6,7] check each [".out"]
score 100}
check : input { P [8,9] check each [".in"] score 100}
Code 1. Example of DSL Script for a Task in Codeforces
From the DSL script written in Code 1, one input checker
and one output checker are generated. The output checker is
used to check P1 until P7 for each output file (*.out). The input
checker is used to check P8 and P9 for each input file (*.in).
The next example is presented to demonstrate that the
system can also generate a source code checker. Code 2 shows
a DSL script for checking an implementation specification of
factorial. The keyword regex is used for generating the
associated regex expression. If the prototype of the recursive
function is known, then it can be checked by a regex matching
function. The function is written in C++ and the prototype of
the factorial recursive function is: int fac (int n).
class FactorialCheck {
format : source {
function block : “{“ “}”
single comment : “//”
multi comment : "/*" "*/" }
P 1 { exist regex "(?s)int fac(int n)..fac(..n..)" }
check : source {P [1] check all [".cpp"] score 100} }
Code 2. DSL Script for Checking the Existence of Recursive Function in C++
The DSL script in Code 2 is written by the problem setter.
It is used to generate a source code checker that checks P1 for
all source code files (*.cpp) submitted by the students. The
example of the source code that computes factorial using the
correct prototype is shown in Code 3.
#include <iostream>
int fac (int n){
if (n == 0) return 1;
return n*fac(n-1); }
int main(){
int n;
std::cin >> n;
std::cout << fac(n) << std::endl; }
Code 3. Example of a Correct Source Code that Computes Factorial
Furthermore, we also give three examples of properties that
can be checked from the source code in Code 3. The
description of the properties can be seen in the Table 1. The
first property checks whether the source code contains
recursive function with prototype int fac(int n), so the
source code get full score (100). The second property checks
whether the source code contains recursive function with
prototype int factorial(int n), so the source code
does not get score because the name of the function is different.
The third property checks whether the source code contains
iterative loop in the main function, so the source code does not
get score because the source code does not contain any loop.
Table 1. Properties Description
No Properties Description Result Score
1 Checks recursive function with prototype Correct 100
int fac(int n).
2 Checks recursive function with prototype Incorrect 0
int factorial(int n).
3 Checks iterative loop in the main function. Incorrect 0
To check a source code in another language (for example,
in Pascal or in Java), the problem setter can simply change the
checker property. The checker property for checking the
existence of a recursive function in Pascal is shown in the
Code 4, and the checker property for checking the existence of
a recursive function in Java is shown in the Code 5.
P 1 { exist regex
"(?s)function fac(n:integer):integer..fac(..n..)"}
Code 4. Property for Checking the Existence of Recursive Function in Pascal
P 1 { exist regex "(?s)int fac(int n)..fac(..n..)"}
Code 5. Property for Checking the Existence of Recursive Function in Java
Another example of the DSL script for source code checker
is to check the usage of temporary variable in a function
written in C++ language. The prototype of the function is int
incr (int x), which returns the value of x+1. The
property expression that is used for checking a temporary
variable is shown in Code 6.
P 1 {forbid regex "(?s)int incr(int x)..="}
Code 6. Property for Checking Temporary Variable in C++
The checker property can also be used to check a static
array declaration in C++, as shown in the Code 7. The type of
the array needs to be specified. In this example, the checker
property checks a static array declaration with int type.
P 1 { forbid regex "int..[..]" }
Code 7. Property for Checking a Static Array Declaration in C++
Another example is to check an implementation of a zero
based recursive function in the LISP language. The header of
the function is NbElmt(L). The function uses the function
IsEmpty(L) for checking whether the list is empty or not as
the base case. The checker property for this checker is shown
in Code 8. In this simple function, the checker property can
also be used to check the implementation of the function by
providing checker properties for all possible correct answers.
For this example, there are four correct reccurrence
expressions, that are 1+NbElmt(Tail(L)), NbElmt(Tail(L))+1,
1+NbElmt(Head(L)), and NbElmt(Head(L))+1. We also
provide a regex generator from the function prototype. The
function prototype will be transformed into a regex pattern that
is used to check the properties.
P 1 { exist
exist
regex
regex
"(?s)NbElmt(L)..1+NbElmt(Tail(L))";
"(?s)NbElmt(L)..NbElmt(Tail(L))+1";
exist regex "(?s)NbElmt(L)..1+NbElmt(Head(L))";
exist
P 2 { exist
regex
regex
"(?s)NbElmt(L)..NbElmt(Head(L))+1"}
"(?s)NbElmt(L)..if..IsEmpty(L)..0"}
Code 8. Properties for Checking a LISP Source Code
In our introductory programming class, we have about one
hundred of small exercises as in Code 8. The checker is very
useful and efficient for the automatic grading of small
exercises.
11
 2015 International Conference on Data and Software Engineering
VI. DISCUSSION
In the previous sessions, we presented a DSL that covered
the specification of input, output, as well as a source code
checker. We also developed a tool set that used the given DSL
script to generate the checker(s) automatically. The results have
been used to grade programming assignment automatically,
and it can be run in a stand alone environment or integrated
with an existing automatic grading system [2].
A checker is defined by a script written in our DSL
grammar, and then generated automatically. By this way, the
checkers are based on specification and are self-documented.
Three types of checkers are generalized by property. We
adopted the idea of a property expressed for MT as a way to
express any kind of property in input, output, and source code.
Input and output are data with their properties are formatted
and the relation between the data. The source code checker has
different characteristics since a source code depends on the
language and has a structure. The properties of the source code
are expressed as a regular expression.
The generated checkers are used systematically for an
automatic grading process for the student’s submission in
programming class. Input checker is important for verifying the
completeness and coverage of the test case. An output checker
is needed when a programming task has many possible correct
answers for an input and also when the oracle problem will be
tested by MT method [1]. A source code checker is used for
checking the implementation specification of the student’s
source code.
By using the DSL, problem setter can focus on
programming task definition. The drawback is that he/she
needs to understand the DSL first. Once a problem setter can
define a checker, he/she can reuse it in another task. We have
defined many checkers and we can produce programming task
more rapidly. The reliability of test cases is also improved by
defining the input checkers. In the DSL we also provide a set of
predefined functions that are frequently used in programming
exercise such as prime(x) to check whether x is a prime number
or not.
The proposed source code checker is independent of the
programming language where the source code of the students is
written. By this independence, we do not need to write a parser
for each language. But, this gives a drawback because the
students need to implement a function or procedure according
to a rigid specification, or uses a given variable name.
The programming learning environment is different from
the programming competition environment. In the
programming competition environment, contestants are asked
to solve complex task and has the freedom to implement the
solution. However, in the programming learning environment,
the students need to respect some coding rules and
implementation specification. In this case, our source code
checker is an important tool for automatic grading of the
programming assignment in the programming learning
environment.
The regex matching is a part of Java library and its
capability to check a pattern has been proven. However, the
problem setter needs to be familiar with regex. An error in
writing the regex produces a false checker. The generated code
needs to be validated before its usage. A regex generator is
built for the problem setter so that it is easier to write
specification implementation.
VII. CONCLUSION
In this research, we have made a single DSL for generating
checkers (output checker, input checker, and source code
checker) with our provided tool set. By using our tool set, the
checker creation process is faster and easier.
Our checkers use a property based checking method to
check the properties in the text, whether the text is input data,
output data, or source code. The expression of the properties is
inspired by the property defined in the Metamorphic Relation,
which is a relation between the data for the input and the
output. For the source code, the property is a string pattern that
can be checked by using regex matching function.
Source code checker does a partial check on the source
code to search the properties. This partial check cannot
guarantee the correctness of the source code (program). So, this
partial check has to be performed after the source code is
graded by the Autograder to determine its correctness. For a
simple program, source code checker can also be used to check
the implementation detail by providing a regex pattern for all
possible correct answers. The regex pattern can be generated
from the function prototype.
REFERENCES
[1] R.I. Hadiwijaya and M.M.I. Liem, “Metamorphic Testing and DSL for
Test Cases & Checker Generators”. Olympiads in Informatics, Vol. 9,
Vilnius University, 2015.
[2] K. Danutama and I. Liem, “Scalable Autograder and LMS Integration”.
ICEEI 2013, 387-395.
[3] C. Kustanto and I. Liem, “Automatics Source Code Plagiarism
Detector”. ACIS/SNPD 2009, Daigu, Korea, 481-486.
[4] S.V. Yulianto, “Automatic grader for programming assignment using
source code analyzer”. Data and Software Engineering (ICODSE), 2014
International Conference, 2014.
[5] H.P. Putro, “Deteksi Code Smell pada Kode Program Dalam
Representasi AST dengan Pendekatan By Rules”. Seminar Nasional
Aplikasi dan Teknologi Informasi, UII Yogyakarta, 2010.
[6] D. Ghosh, “DSLs in Action”. Manning Publication, 2011.
[7] Chen, T.Y., Cheung, S.C., and Yiu, S.M. Metamorphic testing: a new
approach for generating next test cases, Technical Report HKUST-
CS98-01, Department of Computer Science, Hong Kong University of
Science and Technology, Hong Kong, 1998.
[8] K. Claessen, J. Hughes, “QuickCheck: A Lightweight Tool for Random
Testing of Haskell Programs”. In Proc. of International Conference on
Functional Programming (ICFP), ACM SIGPLAN, 2000.
[9] K. Im., T. Im, J. D. McGregor. Automating test case definition using a
domain specific language. In: ACM-SE 46 Proceedings of the 46th
Annual Southeast Regional Conference on XX. 180–185, 2008.
[10] Oracle, ‘The Java™ Tutorials’, 2015. [Online]. Available:
https://docs.oracle.com/javase/tutorial [Accessed: 27- Jul- 2015].
[11] CodePlex, 'StyleCop', 2015. [Online]. Available:
https://stylecop.codeplex.com/. [Accessed: 27- Jul- 2015].
[12] CodePlex, 'FxCop Installer', 2015. [Online]. Available:
https://fxcopinstaller.codeplex.com/. [Accessed: 27- Jul- 2015].
[13] S. Efftinge and M. Spoenemann, 'Xtext', Eclipse.org, 2015. [Online].
Available: https://eclipse.org/Xtext. [Accessed: 27- Jul- 2015].
12