Professional Documents
Culture Documents
Agile Methods As Problem-Based Learning Designs
Agile Methods As Problem-Based Learning Designs
assessment
Elena García Barriocanal Miguel-Angel Sicilia Salvador Sánchez-Alonso
Dept. Computer Science Dept. Computer Science Dept. Computer Science
University of Alcalá University of Alcalá University of Alcalá
Spain Spain Spain
elena.garciab@uah.es msicilia@uah.es salvador.sanchez@uah.es
Juan-José Cuadrado
Dept. Computer Science
University of Alcalá
Spain
jjcg@uah.es
lead to resolution of the problem. This procedure enhances Section 4 and finally conclusions and pitfalls are sketched in
content knowledge while simultaneously fosters the development Section 5.
of communication, problem-solving, critical thinking,
collaboration, and self-directed learning skills. PBL may position 2 BACKGROUND
students in a simulated real world working and professional
context which involves policy, process, and ethical problems that 2.1 Problem-based learning and learning
will need to be understood and resolved to some outcome. analytics
The PBL learning process should be approached following The increased usage of learning technologies such as learning
some methods: (i) Presentation of an "ill-structured" (open-ended) management systems, web-based learning environments, Web 2.0
problem, (ii) generation of a "knowledge inventory" (a list of "what tools, social media etc, has led to the generation of a large variety
the student know about the problem" and "what they need to of different and multimodal educational data [5]. Thus, an
know"), (iii) generation of possible solution, and (iv) formulation important question is how we can gather and make sense of all
of learning issues for self-directed and coached learning. these data to improve learners’ performance, teachers’ role and
Arguably, Software Engineering (SE), when focused on a project course’s design. In this context, learning analytics (LA) merge
perspective, resembles closely these methods. It departs from closely connected to multiple and different research fields and
some user or stakeholder requirements that are often ill-defined, interests related to analysis, such as business intelligence,
and the team progresses by analysis and design identifying the statistics, web analytics, academic analytics, data mining, social
parts that the team can develop and which of them requires network analysis as well as research interest in the field of
further investigation (e.g. ideating and evaluating architectural or learning sciences such as pedagogies, technology enhanced
design approaches, selecting technological components) to learning, cognitive sciences etc. LA is finally strongly related to
accomplish the project requirements. This is particularly true with the learning technologies that have become an integrated part of
agile methods as Scrum [3], in which the team has an ample the learning experience through the technology enhanced
degree of discretion in choosing the content of the iterations, learning paradigm. Such technologies generate large amounts of
including how to approach the work, the number of tasks to fulfill educational data and range from cognitive tools to more
and the responsible of them. sophisticated and complex environments like learning
management systems, virtual learning environments and the
All the above suggests that a PBL approach in the context of a recent massive open online courses. LA definitions share an
realistic setting would make the best fit of actual SE practice with emphasis on collecting and analysing learning data and
classroom activity. While first, entry level courses on SE need to translating them into meaningful actions for improving the
introduce many concepts and techniques, second courses start learning experience for all involved stakeholders [6].
from an overall knowledge of the main topics, and thus represent
a good opportunity to focus on practice by using PBL. Further, this Currently, there seems to be a lack of available knowledge on
allows for “authentic” assessment methods that incorporate how to utilize LA when the courses are structured using the PBL
metrics and indications of productivity that are close to those of a strategy. Projects like PBL 3.01 are actually focused on integrating
professional setting [4]. these approaches to capitalize on their respective strengths and
overcome their drawbacks. In addition, the project aims to enrich
In this paper, we report the instructional design and the the new paradigm including the semantic annotation of resources
evaluation of a PBL experience on an advanced software to enable personalized learning [7].
engineering course using Scrum along with the integration of
learning analytics (LA) in the whole process. Previous work has Tempelaar et al. [8] carried out a research study for learning
applied ideas from PBL for teaching agile methods themselves or and teaching of mathematics and statistics in a blended learning
to provide a realistic setting for work and assessment, but here we environment. The methodology used includes a face-to-face part
report the instructional design systematically and focus on of the learning process which followed the PBL strategy and an
learning actual modern software engineering concepts, including online part which was optional. Within PBL, students were
the use of learning analytics to improve and track students’ required to form small groups, where each group had a specific
progress. content expert and mentor. The data collected during the learning
process included performance scores, groups formed, Frequency
The rest of this paper is structured as follows. In Section 2 of using the practice tests, time spent on practice tests, number of
background is described, showing a brief review of current attempts at solving a problem, etc. The analysis of the gathered
integration learning analytics in PBL approaches and the data provides insights to students that help them monitor and
experiences in software engineering teaching with PBL methods. reflect on their progress, their level of comprehension and get
Section 3 describes the instructional design of an advanced feedback from the teachers on their performance. The case study
software engineering course carried out using PBL on an agile concluded that the usage of the online environments as a
methodology driven development. The course integrates LA in the
whole learning process. Results and discussion are shown in
1 http://pbl3-project.eu/
340
Agile methods as problem-based learning designs: setting and
TEEM 2018, October 2018, Salamanca, Spain
assessment
complimentary tool to PBL proved to help students, as they helped analysis, problem solving and project report. These main phases
in the support of self-direction, reflection and decision making. can be also described in terms of learning activities that take place
during these phases (Fig.1).
Other experience on integrating PBL approach with learning
analytics is the one reported by Göhnert et al [9] in the context of Group Forming
KOLEGEA project [10]. This project provides trainings for doctors Problem
practicing in family medicine using the PBL strategy. Medicine is Formulation
one of the educational domains where PBL is most commonly Task Formulation
used, and in this case, doctors were required to generate cases and
Problem Analysis
solve these cases in a PBL context. The platform used in Data Gathering
workbench enables gathering of data from the project platform to Problem Solving
Implementation
be processed and analysed, but no specific information is provided
in the study regarding the effectiveness of the solution and actual Evaluation
3 INSTRUCTIONAL DESIGN
3.1 Application of PBL phases
Aalborg PBL model [13] is the framework to guide students’ and
tutors’ activity during the reported course. This model stablishes Students are encouraged to form groups up to 5 persons during
several phases for the problem resolution to clearly put in practice the early beginning of the course. At the same time, a unique ill-
knowledge provided for other courses. According to it, the process defined problem formulation is given to students. The problem
of project work can be divided in three main phases: problem statement is formulated introducing vaguely non-functional
341
TEEM 2018, October 2018, Salamanca, Spain E. García Barriocanal et al.
requirements which lead to the implementation of solutions that consideration a large amount of parameters. Students’ assessment
require the concepts stated as learning results of the course. should also be connected with their participation, the spirit of
Before a sprint, students go to a meeting to plan the work in the collaboration, and the quality of engagement and creativity in the
following 3 weeks and after the sprint a review meeting with the group assignments [13]. These parameters are difficult for an
tutor takes place. During this meeting, an evaluation of the work educator to assess without using sophisticated analytics
progress is done (see Section 3.3). methodologies and tools. Such tools can also exploit data recorded
in an LMS and allow the educators to create a learner’s model of
This instructional design fits the Aalborg PBL model, for first
successful behaviour, so that they can design their future courses
weeks (from 1 to 6) cover from group forming to problem analysis,
based on this model and increase the probability of enhanced
splitting up user stories in task and subtask, making an estimation
learners’ performances [14]. In this sense, and in the specific
of the effort for each task and subtask, sketching dependencies
described course, JIRA and GitHub can be considered the source
between tasks, prioritizing each task, labelling them (high,
of most of the data, however they are not LMSs.
moderate, low) and, at last, sorting tasks in the backlog. Each task
is assigned to a student. The last part of the course (weeks 7 to 15) In this instructional design, students obtain an individual grade
covers problem solving and reporting. All these activities taking into consideration 3 aspects:
introduce students in a realistic software project development.
1.Peer assessment: Students send to the tutor an
evaluation of the rest of the team along with a
3.2 Tools used
description of the work done by each member of the
Students work is registered using JIRA Software2 platform and group.
GitHub3. JIRA Software is a platform to manage task and activities 2. Group grade: This grade is obtained according to the
in a project. This tool supports SCRUM, among other, and it evaluation of three parameters: (a) Productivity, i.e., the
allows to register developers, task responsible, task number of requirements fulfilled, (b) Quality of the
decomposition and effort estimation, as well as sprint definition, design from a software engineering viewpoint and (c)
execution and finalization. the use of software tools, both from a software
Once groups are formed, teacher should assign a project to engineering and innovation viewpoint.
each group. Students (the one who adopts the SCRUM master role) 3. Individual grade: This part of the mark is obtained both
must make, by consensus, a task decomposition, prioritization and from the qualitative assessment of the software
estimation, they must assign each task to a student in charge for provided for each student, taking GitHub repository
it and they must also define the backlog. At the beginning of a items as evidence, and from the student activity in the
sprint, all the tasks to be approached are labelled as to-do. Day by group, which can be observed in JIRA.
day, the student responsible for a task should indicate if the task Along with these aspects, in sprint review meetings the tutor
is in-progress or finally done. At the same time, each group has a should monitor the overall activity during the course,
repository for its project code in GitHub. All the code and artifacts reconducting projects that do not progress and detecting those
they generate should be uploaded to the project repository. The students that fall behind with the rest group, in order to avoid they
student in charge for a task should be the person who uploads the finally cut off.
code.
4 RESULTS AND DISCUSSION
JIRA Software also provides a catalog of graphical reporting
tools to check and predict developers’ and teams’ efficiency. From 4.1 Using JIRA as a learning analytics tool
the learning analytics viewpoint, JIRA has been used as a general-
purpose learning analytic tool that includes solutions to provide In this section a description of the Jira tools that can be used for
information and visualizations around different types of data for learning analytic purposes is shown. As previously mentioned,
multiple stakeholders, i.e. students (developers in JIRA), teachers Jira Software is a platform typically used for software
(administrators in JIRA), etc., as described in Section 4. These developements that can be used to conduct PBL approached
visualizations can be in the form of graphs, tables, charts etc., and Software Engineering courses. Nonetheless, tutors can take
aim to assist stakeholders in getting a more holistic overview of advantage of some of the features of the platform to make
course activity realized in a specific platform. In this context, it learning analytics and to act consecuently. These features and
should be noted that GitHub also provides graphics to visualize tools have been grouped by the PBL phase in which they are more
the contribution of developers to a repository. useful.
4.1.1. LA tools for task formulation and problem analysis phases
3.3 Assessment approach Project task breakdown per group can be displayed in JIRA. After
In a PBL context, the students’ performance evaluation in a problem formulation, each group defines task and subtask
complex computer supported learning environment is a tiring and breakdown, prioritises tasks, estimates each task effort and
time-consuming process for educators who must take into assigns each task to a member of the group. In the project backlog
2 https://www.atlassian.com/software/jira/agile 3 https://github.com
342
Agile methods as problem-based learning designs: setting and
TEEM 2018, October 2018, Salamanca, Spain
assessment
(Fig. 2), tutors can check the project tasks in the order that
students plan to approach them (the higher up the sooner to
approach). A summary of the task is shown: type of task, brief
description, responsible student and effort estimation.
343
TEEM 2018, October 2018, Salamanca, Spain E. García Barriocanal et al.
An especially useful chart is the “pie chart report” (Fig. 7). It the backlog) or a poor quality in the implemented code (many
shows a pie chart of issues per project with different filters: issue bugs are continuosly arising).
type, assignee, priority, etc When interpreting this chart, the tutor
should consider that tasks have different work load, so this chart
should be completed with others (see Section 4.2) which consider
task effort estimation.
In order to be aware of the work progress of a group and to 4.1 Quantitative result on student assessment
maximize the project success, the “single level group by report” data
(Fig. 8) is a useful tool. This diagram shows issues grouped by a Although JIRA Software provides a large set of tools to monitor
particular field for a filter: assignee, priority, etc, displaying the activity that can be used for learning analytics purposes, the use
amount or work done or to be completed. of other data such as the effort estimation for each task or the
number of commits in GitHub can be useful to be studied in
The “time since issues report” (Fig. 9) is a bar chart showing
relation with the student grades, among others. In this sense,
the number of issues for which your chosen date field (e.g.
students’ activity data from JIRA and GitHub have been obtained
'Created', 'Updated', 'Due', 'Resolved', etc) was set on a given date.
using the Rest API of both tools. Data have been processed using
An increasing trend when 'Created' filter is used can suggest a
Jupyter Anaconda notebooks, as shown in the architecture drawn
poor work in analysis phase (not enough tasks were included in
up in Fig 104.
4 The relational model of data scrapped from the platforms along with dataset and
344
Agile methods as problem-based learning designs: setting and
TEEM 2018, October 2018, Salamanca, Spain
assessment
345
TEEM 2018, October 2018, Salamanca, Spain E. García Barriocanal et al.
5 CONCLUSIONS AND OUTLOOK overall approach and it would allow to relieve pitfalls and to
obtain significant conclusions due to the use of a larger dataset.
In this paper, a case study on advanced software engineering
teaching using a PBL approach has been reported, along with how
learning analytics can be introduced in the process, both in the ACKNOWLEDGMENTS
groups and individual progress and in the final result. These This work is supported and co-funded by the Erasmus+
adoptions help tutors in two ways: (i) to early act when the programme of the European Union under the project number
individual or group ongoing activities do not lead to fulfil the 562236-EPP-1-2015-1-EL-EPPKA3-PI-FORWARD, PBL3.0. The
project and (ii) to do a deeper study of the final grades and their European Commission support for the production of this
relations with other factors in the process, in order to improve publication does not constitute an endorsement of the contents
future courses. which reflects the views only of the authors, and the Commission
cannot be held responsible for any use which may be made of the
Since an agile method, concretely Scrum, has been used to information contained therein.
carry out the projects, the use of Jira Software platform has
allowed to manage and register students’ tasks and activities REFERENCES
during the project work. Besides, they were encouraged to upload [1] Neville, A. J. (2009). Problem-based learning and medical education forty years
the project code to GitHub repository. Both tools provide visual on. Medical Principles and Practice, 18(1), 1-9.
reports to show different aspects of students’ progress and [2] Hmelo-Silver, C. E. (2004). Problem-based learning: What and how do students
learn? Educational Psychology Review, 16(3), 235-266
activities, which can be use by tutors to guide students during the [3] Schwaber, K., Hundhausen, R., Starr, D. (2015) Agile project management with
project work. In addition, Jupyter notebooks have been provided SCRUM, 2nd ed, Microsoft Press.
[4] Santos, S. C. D., & Soares, F. S. (2013). Authentic assessment in software
to make an analysis of those factor that have an impact in the engineering education based on PBL principles: a case study in the telecom
different components of the final grade (individual assessment, market. In Proceedings of the 2013 International Conference on Software
Engineering (pp. 1055-1062). IEEE Press.
group assessment or peer assessment, all of them made up of [5] Ferguson, R. (2012). Learning analytics: drivers, developments and challenges.
several items). International Journal of Technology Enhanced Learning, 4, 304-317.
[6] Chatti, M.A., Dyckhoff, A.L., Schroeder, U., & Thüs, H. (2012). A reference model
However, pitfalls have been detected during the case study. In for learning analytics. International Journal of Technology Enhanced Learning,
4 (5-6) 5, 318-331.
order to obtain accurate data, it is certainly important that each [7] Zotou, M. et al (2016) PBL3.0: Integrating Learning Analytics and Semantics in
student properly interacts with the tools, avoiding delegation of Problem-Based Learning. 11th European Conference on Technology Enhanced
tasks in other group members. It has been detected that the Learning: Pedagogical Grounded Learning Analytics Design Workshop.
[8] Tempelaar, D. T. et l (2013). Formative assessment and learning analytics. Third
student who play the role of Scrum master tends to do the International Conference on Learning Analytics and Knowledge, 205-209. ACM
assumption of the responsibility of uploading most of the code to press.
[9] Göhnert, T., Ziebarth, S., Verheyen, P., Ulrich Hoppe, H. (2014) Integration of a
the project repository, biasing the results of the analysis. Flexible Analytics Workbench with a Learning Platform for Medical Specialty
Training. LAK wokshops.
It should be also taken into consideration that students are [10] Ziebarth, S et al (2013) Design of a collaborative learning platform for medical
learning about Scrum and the use of supporting tools (Jira doctors specializing in family medicine. The Computer Supported Collaborative
Learning Conference 2013, vol 2, 205-208.
Software and GitHub) during the project ongoing, as a part of the [11] Kizaki, S., Tahara, Y., & Ohsuga, A. (2014). Software development PBL focusing
PBL approach. This can produce not realistic activity and progress on communication using Scrum. In Advanced Applied Informatics (IIAIAAI),
data after the first sprint. These data can be considerably different 2014 IIAI 3rd International Conference on (pp. 662-669). IEEE press
[12] El-Khalili, N. H. (2013). Teaching Agile software engineering using problem-
from those obtained at the end stages of the project. Reasons for based learning. International Journal of Information and Communication
this difference may be attributed to bad effort estimations, Technology Education (IJICTE), 9(3), 1-12.
[13] Barge, S. (2010). Principles of problem and project learning, the Aalborg PBL
overvaluation of coding capacity, Scrum/Jira misunderstandings, model. Aalborg: Aalborg University.
etc., all of them result of a lack of knowledge at the beginning of doi:http://www.aau.dk/digitalAssets/62/62747_pbl_aalborg_modellen.pdf
[13] Dyckhoff, A. L., Zielke, D., Bültmann, M., Chatti, M. A., & Schroeder, U. (2012).
the project work. Design and Implementation of a Learning Analytics Toolkit for Teachers.
Educational Technology & Society, 15 (3), 58–76.
For the aforementioned, rerunning the course using the same [14] Dietz-Uhler, B., & Hurn, J. E. (2013). Using learning analytics to predict (and
instructional design could be a good proof of concept for the improve) student success: A faculty perspective. Journal of Interactive Online
Learning. 12(1). 17-26.
346