2015 41st Euromicro Conference on Software Engineering and Advanced Applications
Applying property-based testing in teaching
safety-critical system programming
Lars-Åke Fredlund, Ángel Herranz and Julio Mariño
Babel Group
Universidad Politécnica de Madrid, Spain
Email: {lfredlund,jmarino,aherranz}@fi.upm.es
Abstract—At the Universidad Politcnica de Madrid students
attending a course on concurrency are taught a high-level
formalism which permits concise specification of shared resources.
This formalism is used to express safety-critical access policies
for typical control problems such as robot plants. Students are
moreover provided with programming recipes for implementing
such shared resource specifications in programming languages
(typically Java). The teachers of the course use various tools
to ensure that the implementations developed by students for a
shared resource are of an acceptable quality. Such tools include
normal unit tests, but also the systematic application of property-
based testing to judge the quality of the exercises. In this article
we provide an overview of the tools, techniques and methods used
in one particular exercise of the course: the implementation of a
control system for an automated warehouse.
Keywords—Testing; Java; Concurrency; Safety;
I. I NTRODUCTION
Programming concurrent applications is today still a chal-
lenging task. A programmer has to understand the concurrency
guarantees provided by a programming language (or operating
system) with regards to thread (or process) scheduling, and
with regards to shared memory writes and accesses. The
programming language provides basic tools to combat this
complexity, e.g., for Java, synchronized methods, synchronized
regions, and the possibility to declare variables “volatile”.
However, as is evident from the vast number of questions
found on the Internet1 , these, and other concurrency primitives
and libraries for Java, are not well understood. Moreover,
concurrency is becoming more important: almost all proces-
sors today come equipped with multiple processing cores
which share memory. Given the importance of concurrent
programming, and the inherent difficulty, it is no surprise that
a course on concurrency is taught to undergraduate students
at the Universidad Politécnica de Madrid (Madrid Technical
University). The course, which is taught in the fourth term
in the computer science bachelor degree program at the
University, uses the Java programming language as a basis.
The choice of Java is not because the language provides an
especially good foundation for concurrent programming, but
rather because students are already familiar with the language
from earlier programming courses. The course has a first part
where the classical language-based concurrency tools are intro-
duced (monitors, semaphores, etc) and their implementations
This work has been partially funded by the European Commission FP7
project ICT-2011-317820 PROWESS, the Spanish MINECO project TIN2012-
39391-C04-02 STRONGSOFT, and the ARTEMIS Joint Undertaking under
grant agreement no 295373 (project nSafeCer).
1 e.g., as seen on the popular stackoverflow.com site
978-1-4673-7585-6/15 $31.00 © 2015 IEEE
DOI 10.1109/SEAA.2015.53
as Java libraries (java.util.concurrent, etc); graded
small programming exercises are used to encourage student
learning and participation, and to judge their skill set. However,
although such relatively low-level libraries and concurrency
primitives are highly useful for programming high-performing
concurrent applications, they are, as experience show, not easy
to master. A second part of the course introduces both a higher-
level concurrent library JCSP2 [1] which provides CSP-like
concurrency primitives for Java, and, the topic of this paper,
the shared resource formalism [2].
The core idea with shared resources is that the design must
distinguish active entities (threads, processes) from passive
ones (resources, shared memory locations). The latter represent
all kind of interaction among the former, and are formally
modelled using an abstraction (shared resource specifications)
that contains a clear interface – which can be invoked from
processes – and a transactional transition semantics.
The code obtained from the active entities is considered
light in the sense that it is assumed to be free from concurrency-
specific constructs and is, thus, easier to verify, more portable,
and does not require specially trained programmers to develop
or test. On the other hand, the code from the shared resources is
considered heavy code, as it is here where all the concurrency
specific code is placed. It is convenient then, that this code
is carefully derived from validated designs so that it can be
regenerated if requirements change, rather than modified by
hand. In practice, this translation of shared resources into code
in an actual programming language is semi-automatic, by means
of certain idioms and code patterns[3], [4]. Certainly, the use
of these patterns enforces some discipline in the use of error-
prone concurrency primitives, thus alleviating some of the
aforementioned language-related issues.
The focus of this paper is on describing the experiences of
giving the undergraduate students, in the course on concurrent
programming, the task of correctly implementing a particular
shared resource specification. To focus attention away from
lower-level concurrency issues, and instead centering it on
issues to do with safety critical systems, and embedded systems,
the specification describes an automatic (shipping) warehouse
in which autonomous robots operate. We describe in detail
how the exercise was set up, the tools and techniques given to
students to help them in their task, and finally evaluate how
successful the students were in solving this (in our opinion quite
representative) concurrent programming task. The contributions
of this are twofold, first, we provide a detailed recipe for how
2 http://www.cs.kent.ac.uk/projects/ofa/jcsp/
309
similar courses can be run in other academic institutions, and
secondly, we use a probably quite typical set of students to
evaluate the difficulty of concurrent programming.
In the following Sect. II we describe the exercise topic: the
warehouse example. Next, in Sect. III we describe the shared
resource specification formalism briefly, and then, in Sect. IV
describe a technique for implementing such shared resources
using a combination of the Java programming language and a
particular low-level concurrency library. Next, Sect. V describes
in detail how the exercise was run, and the outcome. Finally,
Sect. VI summarizes the findings, and details issues for future
work.
II. T HE WAREHOUSE E XERCISE
The running example used in the article is the control system for
a warehouse complex serviced by a set of autonomous robots.
An example warehouse complex, with robots, is depicted in
Fig. 1.
A robot must first enter warehouse 0, then it may load an
item, and next it exits warehouse 0 and enters the corridor
between warehouse 0 and warehouse 1. Then, it enters
warehouse 1, etc, until it finally exits the warehouse complex
by exiting the last warehouse (warehouse 2 in the figure). Each
robot has a weight, and the total weight of a robot and its cargo
increases monotonously as it moves around in the warehouse
complex. A warehouse can admit any number of robots, but to
ensure safe operations the total weight of robots and their cargo
cannot exceed the constant MAX WEIGHT IN WAREHOUSE
when a new robot enters the warehouse. It is permitted that the
total weight in a warehouse is temporarily above the limit, due
to loading operations, but then no more robots can be admitted
to the warehouse (until a robot leaves). A corridor has place
for a single robot.
In Fig. 1, the constant MAX WEIGHT IN WAREHOUSE is
set to 1000kg, and thus, for example, we can see that since
the total weight in warehouse 0 is 500 + 200 + 200 = 900 the
robots with weights 200 and 300 that want to enter should be
blocked, while the robot with weight 100 can be permitted to
enter. Moreover, as the corridor between warehouse 1 and 2
is occupied, the robots inside warehouse 1 should be blocked
from exiting it, until the robot occupying the corridor enters
warehouse 2.
A shared resource specifies which operations are safe,
however, we often want to include a progress requirement
too, i.e., specifying which calls must be executed by the shared
resource, given a particular resource state. In the exercise
described in this article, there was a general such progress
requirement which interpreted for the robot plant reads:
In any (resource) state, if there is at least one robot
that can enter or exit a warehouse, then a robot must
eventually be permitted to enter or exit a warehouse.
Notice that the above condition does not specify any priority
(e.g., based on weights of robots) on which robot to service,
should several robots be ready to act.
310
III. R ESOURCES
One commonly used mechanism for controlling interactions
between concurrent processes is to impose some form of central
control, to serialize potentially conflicting requests.
The shared resources introduced in [2] is one such central-
ized mechanism, and we will explain its syntax and semantics
using the robot warehouse example. Figure 2 contains the
specification of the control part of the robot warehouse example.
The resource specification details two operations that can
be used to coordinate movements between warehouses:
• enterWarehouse(n,w) – A request for permission for
a robot to enter warehouse n carrying weight w.
• exitWarehouse(n,w) – A request for permission for a
robot to exit a warehouse n towards a corridor carrying
weight w.
The state of the resource has two fields: weight, a mapping
from a warehouse to weight (a natural number), and occupied,
a mapping from a warehouse to a boolean. Intuitively, weight
should correspond to the accumulated weight in the warehouse,
and occupied[n] is true if there is a robot present in the corridor
n leading from the warehouse.
Initially, as specified in the INITIAL clause, the weight in
all warehouses is zero, and no robot is present in any corridor.
The resource has an invariant over the state, as specified by
the INVARIANT clause, i.e., that the weight in a warehouse
should always be less than or equal to the maximum weight
MAX WEIGHT WAREHOUSE.
A robot that wants to enter warehouse n with weight w
should first call enterWarehouse(n,w) to ask the resource
(controller) for permission to do so. It is the task of the
(implemented) resource to ensure that the call does not return
(i.e., that it blocks) until it is safe for the robot to enter
the warehouse. The concurrency precondition CPRE specifies
when access is safe, i.e., when the accumulated weight of the
robots already in the warehouse plus the weight of the new
robot is less than or equal to the allowed maximum weight.
The POST condition specifies the change on the resource
state provoked by the completion of a call. It is possible to
provide preconditions (PRE) for operations too, which specify
requirements on the arguments to an operation that every call
must satisfy.
Similarly, a robot should always call the operation exit-
Warehouse(n,p) to ask for permission to leave a warehouse.
The CPRE condition of the resource specification ensures that
the call does not return until the corridor leading away from
the warehouse is free from robots. A restriction on the caller to
these operations is that the weight w provided as argument to the
operation exitWarehouse(n,w) when asking for permission for
leaving a warehouse, must be identical to the weight provided
when asking for permission to enter the warehouse, i.e., en-
terWarehouse(n,w). That is, the exit weight should not reflect
any cargo loaded in the warehouse, instead, the weight increase
should be factored into the next call to enterWarehouse, e.g.,
enterWarehouse(n+1,w+cargoWeight).
Note that the control system does not need to identify
individual robots, and indeed cannot, as there is no robot
 500
cannot exit
300
200
cannot exit
100 200
cannot
enter
200
200
0 1
Fig. 1. Warehouses and robot movements.
identifier passed along in the two operations. Moreover note
that the example abstracts away from the use of e.g. scales to
weigh robots and cargo, i.e., it is assumed that calls to these
two operations always identify the correct weights of robots.
IV. A S HARED R ESOURCE I MPLEMENTED IN JAVA
A correct implementation of a shared resource ensures
that its operations are executed only when the concurrency
precondition (CPRE) so permits, and in isolation. However,
there may also be additional requirements on the order in
which different calls are served which are not expressed by the
resource specification. In the warehouse exercise, for instance,
students were expected to implement the progress condition
that if in a state if it is safe for a non-empty set of robots to
enter or exit warehouses, then some robot must eventually do
so.
A resource specification can be implemented in different
languages, using different concurrency language primitives.
We can implement a resource in Java, for instance, using
e.g. the Locks and Condition classes provided by the
java.util.concurrent package. As an example, Fig. 3
provides a (sketched) Java class that can serve as a starting
point for a complete implementation. Note that the class is
rather incomplete. It does for instance not address the special
role of the last and first warehouses, i.e., that there is no corridor
before the first warehouse, and the absence of a corridor after
the last warehouse.
The exitWarehouse(n,w) method begins by acquiring
a lock, ensuring that no other call executes simultaneously. Then,
the concurrency precondition (CPRE) is continuously evaluated.
311
100 300
300
300 300
400
cannot
enter
100
2
MAX WEIGHT IN WAREHOUSE == 1000
If CPRE does not hold, because the corridor is not empty,
the thread executing the method will wait on the condition
freedCorridor[n+1] until another thread signals it (in
enterWarehouse(n,w)).
Once the CPRE is established, the POST condition is
established by modifying the state of the resource (not shown
in the code excerpt). Then, finally, the method signals any other
thread, corresponding to a robot waiting to enter warehouse n
which the robot executing exitWarehouse(n,w) just left.
V. E XERCISE : F ROM S PECIFICATION TO A G RADED
A SSIGNMENT
As the main (graded) exercise of the course, students were
provided with the specification of the warehouse example
(in Fig. 2), and instructed to implement the resource using
Java. Moreover, the students were required to use a particular
concurrency construct [5], which is an improvement on the
lock and condition solution seen in Fig. 3, in that it is not
needed to test the concurrency precondition using a while loop.
In earlier lectures, students had been taught in detail about
all these topics. They had seen, for instance, several other
shared resource specifications, and had attended lectures on
how to implement such specifications in Java. Students were
grouped together3 in teams of two students to solve the exercise.
As the course does not teach software engineering skills in
general, but rather basic skills in concurrent programming and
design, both students in a pair were supposed to partake equally
in all activities required to solve the exercise. However, no
3 the selection of an exercise partner could be made freely
 C-DAT WarehouseAccessControl
OPERATIONS
ACTION enterWarehouse: Warehouse[i] × Weight
ACTION exitWarehouse: Warehouse[i] × Weight

BEHAVIOUR
DOMAIN:
TYPE: WarehouseAccessControl = (weight: Warehouse → Weight × occupied: Warehouse → B)
Warehouse = 0 .. N WAREHOUSES - 1
Weight = 0 .. MAX WEIGHT WAREHOUSE

INITIAL: ∀ n ∈ Warehouse • self.weight(n) = 0 ∧ ¬self.occupied(n)

INVARIANT: ∀ n ∈ Warehouse • self.weight(n) ≤ MAX WEIGHT WAREHOUSE

CPRE: p + self.weight(n) ≤ MAX WEIGHT WAREHOUSE

enterWarehouse(n,p)
POST: self.weight = selfpre .weight ⊕ {n → selfpre .weight(n) + p}
∧ (n > 0 ⇒ self.occupied = selfpre .occupied ⊕ {n → False})
∧ (n = 0 ⇒ self.occupied = selfpre .occupied})

CPRE: n = N WAREHOUSES − 1 ∨ ¬self.occupied(n + 1)

exitWarehouse(n,p)
POST: self.weight = selfpre .weight ⊕ {n → selfpre .weight(n) − p}
∧ (n < N WAREHOUSES − 1 ⇒ self.occupied = selfpre .occupied ⊕ {n + 1 → True})
∧ (n = N WAREHOUSES − 1 ⇒ self.occupied = selfpre .occupied)

Fig. 2. Specification of the robot controller.

attempt was made to determine whether indeed such an equal
participation took place. Apart from attending lectures, students
had individual access to teachers to resolve questions during the
entire exercise period. In total around 220 students attempted
to solve the exercise; there were three teachers available to
assist the students during the exercise.
A. Exercise Time Plan
The exercise was setup in two phases: the first phase ran
during almost two weeks, which finished with students having
the possibility of handing in a preliminary solution. Although
the handing of a solution in this phase was not strictly obligatory,
almost all student groups did this. After a brief period of
evaluation by teachers (2 days), students were presented with
individualised feedback (to be described later), and phase two of
the exercise started. During phase two, which lasted roughly a
week, students improved their solutions and eventually handed
in their final solutions to the programming task. Then, the
teachers of the course assessed the final solutions, partly with
the help of automated testing tools, and assigned final exercise
grades.
B. Exercise Tools
Student had two basic tools to aid them in comprehending
the exercise task, and in assessing the quality of their solutions.
First, a warehouse simulator was provided. This simulator
was rather simplistic. It was a text based simulator, which
created a set of robots, letting them load random items (with
random weights) in warehouses, and calling the control system
programmed by the students to illustrate the calls which their
control system could expect, and the overall dynamic of the
warehouse example. The simulator had ample print-statements
to provide ample feedback to students.
The second tool was the system for handing in solutions
itself. Solution were not handed in using electronic mail,
but using an automated web-based program. This web-based
program came with a small set of test cases (15 during phase
one), programmed by hand by teachers, which attempted to
detect violation of the main system invariant and the overall
progress condition, i.e., that the maximum weight in any
warehouse can be at most 1000 weight units, and that if it
is safe for some robot to enter or exit a warehouse, some
such safe action must be taken. Such test cases essentially
created a few robots with certain weight, let these robots make
calls to the student programmed control system, and judged
whether the control system made the correct decision in letting
a robots proceed (or not). In case a student solution failed one
or more of these simple test cases, the web-based program
simply refused to accept their implementation, never forwarding
it to the teachers responsible for grading.
To get feedback, students could attempt to hand in a solution
repeatedly, without penalty, just to get information from running
the test cases. In case a solution was successful, students could
still submit improved versions (until phase one had terminated);
the latest solution submitted was used for (eventual) grading.
Moreover, the test cases were available, in source code form,
to students.
A typical such test case is shown in Fig. 4, using English
language for clarity (test cases were programmed in Java). The
test case depicted involves a set of robots that ask the control

312
public c l a s s WarehouseResource {
/ / Resource s t a t e
private int weight [ ] ;
p r i v a t e boolean occupied [ ] ;

/ / Handling concurrency
p r i v a t e Lock l o c k ;
private Condition freedWarehouse [ ] ;
private Condition freedCorridor [ ] ;

public WarehouseResource ( ) {
/ / i n i t i a l i z e s t a t e and c r e a t e m o n i t o r s and c o n d i t i o n s
}

p u b l i c v o i d e n t e r W a r e h o u s e ( i n t n , i n t w) {
lock . lock ( ) ;

/ / Check CPR −− c o d e d i n J a v a −− u n t i l i t becomes t r u e

w h i l e ( ! CPRE ( . . . ) ) f r e e d W a r e h o u s e [ n ] . a w a i t ( ) ;

/ / CPRE h o l d s h e r e , u p d a t e r e s o u r c e s t a t e ( POST )
// ...

/ / S i g n a l w a i t e r s t h a t t h e r o b o t has l e f t t h e c o r r i d o r
freedCorridor [n ]. signal ( ) ;

lock . unlock ( ) ;
}

p u b l i c v o i d e x i t W a r e h o u s e ( i n t n , i n t w) {
lock . lock ( ) ;

/ / Check CPR −− c o d e d i n J a v a −− u n t i l i t becomes t r u e

w h i l e ( ! CPRE ( . . . ) ) f r e e d C o r r i d o r [ n + 1 ] . a w a i t ( ) ;

/ / CPRE h o l d s h e r e , u p d a t e r e s o u r c e s t a t e ( POST )
// ...

/ / S i g n a l w a i t e r s t h a t t h e r o b o t has l e f t t h e warehouse
freedWarehouse [ n ] . s i g n a l ( ) ;

lock . unlock ( ) ;
}
}

Fig. 3. An Implementation Sketch of the Warehouse Resource

Robot 0 asks for permission to enter warehouse 0 with weight 800

(should not block)
Robot 1 asks for permission to enter warehouse 0 with weight 100
(should not block)
Robot 2 asks for permission to enter warehouse 0 with weight 400
(should block)
Robot 3 asks for permission to enter warehouse 0 with weight 150
(should block)
Robot 1 asks for permission to exit warehouse 0 with weight 100
(should not block, should also unblock Robot 2)

Fig. 4. A typical test case

313
test20(TestRobots)
java.lang.AssertionError: the call enterWarehouse(0,400) of robot 2
raised the exception java.lang.NullPointerException
Stacktrace:
java.lang.NullPointerException
at ControlAccesoNavesMonitor.enterWarehouse(ControlAccesoNavesMonitor.java:61)
at Call.call(Call.java:117)
at Call.toTry(Call.java:112)
at es.upm.babel.cclib.Tryer.run(Tryer.java:50)
Call trace:
enterWarehouse(0,800) of robot 0 -- did not block
enterWarehouse(0,100) of robot 1 -- did not block
enterWarehouse(0,400) of robot 2
Fig. 5. A typical test case failure report – an exception (text translated from spanish)
system for permission to enter and exit warehouses; the text in
parenthesis show what the expected decisions by the control
system are. Feedback to students when their solution failed
a test case was in natural language, an example is shown in
Fig. 5.
Moreover, students were encouraged, but not forced, to
develop their own unit tests for their solutions. In practice, very
few students seems to have done this.
C. Feedback on Hand-ins after phase one
At the end of phase one, almost all student groups had
been able to successfully hand in their solutions using the
automatic web-based system, i.e., the test cases did not reveal
any problems. Still, it was known that the test cases were
not nearly exhaustive enough to determine whether a solution
was correct, and moreover, we wanted to examine the code to
assess e.g. whether a solution was really using the required Java
library to solve the assignment. The solutions were assessed
according to the following criteria:
• CPRE-while: not testing the concurrency precondition
in a while loop. This condition has to do with a stylistic
concern which had been stressed repeatedly in lectures,
which aims to reduce the number of concurrency errors
(see [5] for details)
• NO-static: static (class variables) variables must not
be used
• NO-comments: no, or few, comments
• Multiple-monitors: the code incorrectly uses multiple
monitors
• Has-bugs: functional bugs were detected.
Students received a “report sheet” listing any problems detected
in their solutions according to the above criteria. The sheet
noted that a few additional style requirements had not been
checked against their code, but would be checked during phase
two.
Violations of the criteria above, except for Has-bugs, was
detected using automatic source code analysis combined with
manual inspection. For instance, to detect usage of multiple
monitors, the UNIX grep utility was used to count the number
of occurrences of the Java Monitor class in the source code, and
314
manually inspection then determined whether implementations
marked as suspicious indeed used more than one monitor. To
determine whether a solution had bugs (i.e., as reported under
the setting Has-bugs), we extended the number of test cases to
around 30, taking care to design test cases that more carefully
tested solutions.
The results from the assessment were that around 15 solu-
tions had programmed the concurrency precondition incorrectly
(CPRE-while), that 11 solutions used static variables, that 7
solutions had too few comments, that around 40 solutions still
had bugs, and five solutions were using multiple monitors.
The more critical bugs were clearly the ones which had
programmed the concurrency precondition incorrectly, or were
using multiple monitors. Although we had stressed these (part
style) concerns repeatedly during the course, and we had stated
clearly that solutions that did not follow these guidelines were
not acceptable, still a fair number of solutions did not meet
the requirements. That solutions still contained bugs was not
wholly unexpected, given the incomplete test suite provided.
D. phase two
For phase two, the test cases used in the automatic web-
based handing-in system was extended; we used the same
test cases that had been used to assess the quality of the
implementation (Has-bugs) at the end of phase one. We could
have extended the number of test cases further, but the aim of
providing them was not force students to write (functionally)
correct code, but rather to help them to understand likely
problems with their solutions. We consider it unlikely that
student working in industry after graduation on a programming
task will have access to high quality test suites, and so we
did not want to subject students to them during this exercise
either. Rather, students were told that the test suite provided had
defects, and that they should not simply rely on its diagnostics,
but that they were ultimately responsible for deciding when
their solutions were of an acceptable quality.
E. Grading
To assess the final quality of the solutions after phase two,
the procedure from the assessment of phase one was repeated,
and was extended by automated testing of solutions.
 The code style comments had now been fixed4 . The style
criteria for how to program a shared resources were checked
a bit more carefully, and although warned in advance, there
were a number of student groups that did not follow the new
requirements.
The automatic testing was done by developing a model of
the problem. To do this we used the Quviq QuickCheck [6]
tool. Essentially, we described the correct functioning of an
implementation of the robot warehouse using a state machine,
which was used both to automatically generate random test
cases, and to judge whether such automatically generated test
cases were executed correctly by the student implementations.
The details for how this was accomplished are described in [7].
The QuickCheck based model used for automatic testing was
not made available to students.
Although the task of correctly programming the shared
resource warehouse example may not appear overly difficult,
the results of our testing using QuickCheck were, at least to us,
surprising. Of the 103 solutions tested, which had passed the
manually crafted test suite, we found errors in 55 of them, i.e.,
54.40% of the solutions handed contained at least one error,
indicating that the particular test suite was not particularly
good at finding errors, and that the QuickCheck-based testing
approach was much more successful.
We separated the testing of the implementations into three
experiments. First, we tried to identify implementation which
failed a basic safety criterion (test1), i.e., an implementation
admits a robot in a warehouse even when the total resulting
weight exceeds the limit, or admits a robot in a corridor
even when the corridor is already occupied by another robot.
The second property (test2) additionally tests whether there
are calls (to enter or exit a warehouse) which the model
permits, but which the implementation blocks. That is, the
implementation does not satisfy a liveness condition. Finally
test3, in contrast to test1 and test2, tries to execute multiple calls
concurrently, to detect possible incorrect uses of the basic Java
concurrency mechanisms. The correctness property checked
for test3 includes both safety (test1) and liveness (test2).
The result of the testing using our testing framework are
summarized in Figure 6. Note that test3 was run only if no errors
were detected using test1 or test2. To properly interpret the
results, note that the properties tested in the three experiments
are not mutually exclusive. That is, test2 is a more strict test
that test1 (i.e., testing both safety and liveness), whereas test3 is
more strict than both test2 and test1, as it tests both safety and
liveness under the added complication of (possibly) concurrent
calls. As the testing is randomized, there is a chance that
although an implementation error is, say, first detected during
experiment test2, it may be still be a safety error which due to
the nature of random test generation, was by chance not detected
during test1. In the figure “PASSED” are those implementations
(48) in which testing failed to detect an error, and those under
the heading “FAILED” (55) had at least one bug. The figure
moreover indicates which test spotted an error. Thus, as an
example, among the failed implementations, row 2 shows that
there were 13 implementations that failed experiment test2 (and
did not fail test1), while the last row tells us that two additional
4 Except
for a few cases were students had not submitted any solution at
the end of phase one, and had thus not received any individualised feedback.
315
PASSED FAILED
55
test1 test2 test3 #
48  - 2
-  13
  38
- -  2
Fig. 6. Test results for student’s Warehouse implementations, classified by
experiments.
implementations were found erroneous using test3 which were
not detected by either test1 or test2.
VI. C ONCLUSION AND F UTURE WORK
Clearly programming concurrent systems is a quite challeng-
ing task for the average undergraduate student. Even though
we had devoted quite substantial resources to aiding students
in developing correct solution, when the exercise reported in
this article had finished, we could still find bugs in over 50%
of the solutions handed in, although students were motivated
(by grading concerns) to hand in good solutions.
It must be stressed that their difficulty lay not mainly in
interpreting the high-level concurrency “design pattern” shared
resource specification correctly, rather, based on observations
during individual tutoring of students and code inspection, the
students had significant difficulties in correctly using the basic
Java concurrency tools for implementing a shared resource.
Clearly, we should (ideally) invest even more (teaching)
resources into teaching students concurrent programming;
this is not surprising given the inherent difficulty of the
task. Unfortunately, at least in our university, the course on
concurrent programming receives less resources than most other
programming courses. Moreover, this situation is likely not
about to change. So, given that concurrent programming is
truly quite hard, and there are not enough teaching resources
available to enable student to become truly proficient in its use,
what should we do?
One way forward, in our opinion, would be to de-emphasize
the use of low-level concurrency mechanisms for solving basic
concurrent tasks, and focusing more attention on higher-level
concurrency constructs. For instance, we would like to continue
having students using shared resources (but attempting to de-
velop tools for automatically deriving correct implementations
of such resources), or having students using the JCSP library
for Java [1], or using an actor-based language such as e.g.
Erlang [8].
Clearly there still are concurrent application where the
performance benefits from using low-level libraries outweigh
their drawbacks, but for many other uses the price in terms
of low programmer productivity, and potential for introducing
difficult to find race condition bugs, makes their application
prohibitively expensive.
R EFERENCES
[1] P. H. Welch, N. Brown, J. Moores, K. Chalmers, and B. H. C.
Sputh, “Integrating and extending JCSP,” in The 30th Communicating
Process Architectures Conference, CPA 2007, organised under the
auspices of WoTUG and the University of Surrey, Guildford, Surrey,
 UK, 8-11 July 2007, ser. Concurrent Systems Engineering Series,
A. A. McEwan, S. A. Schneider, W. Ifill, and P. H. Welch,
Eds., vol. 65. IOS Press, 2007, pp. 349–370. [Online]. Available:
http://www.booksonline.iospress.nl/Content/View.aspx?piid=5982
[2] A. Herranz, J. Mariño, M. Carro, and J. J. Moreno-Navarro,
“Modeling concurrent systems with shared resources,” in Formal
Methods for Industrial Critical Systems, 14th International Workshop,
FMICS 2009, Eindhoven, The Netherlands, November 2-3, 2009.
Proceedings, ser. Lecture Notes in Computer Science, vol. 5825, 2009,
pp. 102–116. [Online]. Available: http://www.springerlink.com/content/
b83m037648436667/
[3] J. Mario, R. N. N. Alborodo, and Ángel Herranz, “Model-based
thread-safe Java code generation from JML specifications,” 2015, in
preparation. [Online]. Available: http://babel.upm.es/∼rnnalborodo/sr
web/jml clause paper.pdf
[4] M. Carro, J. Mario, . Herranz, and J. Moreno-Navarro, “Teaching how
to derive correct concurrent programs from state-based specifications
and code patterns,” in Teaching Formal Methods, ser. Lecture Notes
in Computer Science, C. Dean and R. Boute, Eds. Springer
Berlin Heidelberg, 2004, vol. 3294, pp. 85–106. [Online]. Available:
http://dx.doi.org/10.1007/978-3-540-30472-2 6
316
[5] A. Herranz and J. Mariño, “A verified implementation of priority monitors
in Java,” in Proceedings 2nd. International Conference on Formal
Verification of Object-Oriented Software (FoVeOOS’11), Revised Lectures,
ser. Lecture Notes in Computer Science, B. Beckert, F. Damiani, and
D. Gurov, Eds., vol. 7421. Springer, 2012, pp. 160–177.
[6] T. Arts, J. Hughes, J. Johansson, and U. T. Wiger, “Testing telecoms
software with quviq quickcheck,” in Proceedings of the 2006 ACM
SIGPLAN Workshop on Erlang, Portland, Oregon, USA, 2006, pp. 2–10.
[7] L. Fredlund, Á. Herranz-Nieva, and J. Mariño, “A testing-based
approach to ensure the safety of shared resource concurrent systems,” in
Software Engineering and Formal Methods - SEFM 2014 Collocated
Workshops: HOFM, SAFOME, OpenCert, MoKMaSD, WS-FMDS,
Grenoble, France, September 1-2, 2014, Revised Selected Papers,
ser. Lecture Notes in Computer Science, C. Canal and A. Idani,
Eds., vol. 8938. Springer, 2014, pp. 116–130. [Online]. Available:
http://dx.doi.org/10.1007/978-3-319-15201-1 8
[8] F. Cesarini and S. Thompson, Erlang Programming – A Concurrent
Approach to Software Development. O’Reilly Media, 2009.