2015 3rd International Conference on Information and Communication Technology (ICoICT)

Big Data Analytics on Large-Scale Socio-technical

Software Engineering Archives
Shahabedin Bayati, David Parsons, Teo Susnjak, Marzieh Heidary
School of Engineering and Advanced Technology
Massey University,
Auckland, New Zealand
S.bayati@massey.ac.nz

Abstract—Given the fast growing nature of software engineering
data in online software repositories and open source communities,
it would be helpful to analyse these assets to discover valuable
information about the software engineering development process
and other related data. Big Data Analytics (BDA) techniques and
frameworks can be applied on these data resources to achieve a
high-performance and relevant data collection and analysis.
Software engineering is a socio-technical process which needs
development team collaboration and technical knowledge to
develop a high-quality application. GitHub, as an online social
coding foundation, contains valuable information about the
software engineers’ communications and project life cycles. In
this paper, unsupervised data mining techniques are applied on
the data collected by general Big Data approaches to analyse
GitHub projects, source codes and interactions. Source codes and
projects are clustered using features and metrics derived from
historical data in repositories, object oriented programming
metrics and the influences of developers on source codes.
Index Terms— Big Data; GitHub Mining; Clustering; Mining
Software Repositories (MSR); Empirical Software Engineering
I. INTRODUCTION
Software engineering data is available in open source
software repositories through the Web. By increasing the
number of online repositories and distributing the development
teams across the world this accumulated data became a great
resource for data analysis and mining about software
engineering. Online software archives involves social
communications repositories, version control, source code,
commit logs, bug tracking and other development artifacts.
Web 2.0 and social network features also align with these
repositories and supports social collaboration among team
members and software engineers in large distributed
environments. As the size of archives has rapidly increased and
the complexity of stored data in these repositories gets more
complex, the benefit of applying distributed computing
techniques and Big Data approaches has become more obvious.
During the last decade data scientists in software
engineering archives presented a new field in knowledge
discovery and management known as Mining Software
Repositories (MSR), which is focused on discovering valuable
and actionable information and knowledge about software
engineering development processes. Although MSR is quite a
new field in data science and software engineering, valuable
research and applications have already been presented and this
community gets bigger and bigger. In MSR research, data
mining and machine learning algorithms like classification,
association rules, text mining, sentiment analysis, social
network analysis and clustering are widely used to find
information about software ecosystems or development through
software repositories [1].
Various online open source repositories are now available
through the Web. GitHub, SourceFoge, Code Google and
BitBucket are some of these resources, which may contain
various types of repositories. In this research we focus on
GitHub as a social coding environment which enables software
engineers to do their coding process through version control
and also communicate via social networking tools within
GitHub. GitHub has variety of events, repositories and APIs to
help software engineers in the development lifecycle. This
stored socio-technical data is a great resource for data mining
purposes. GitHub uses Git and subversion as version control
technologies. It has gained more than 7 million users since its
establishment. At the time of writing it is one the 130 most
popular Websites as ranked by Alexa and it has about 17
million project repositories.
Huge amounts of fast growing data in GitHub repositories
are stored in a variety of formats and complex structures which
illustrated that GitHub is a Big Data resource and has the main
characteristics of Big Data. Big Data involves 3-Vs (Volume,
Variety and Velocity) as the most important features of a data
set that may be classified as a Big Data resource [2]. Volume
means that data should be big in size, usually more than
terabyte (TB). Velocity means the generation rate of data
should also be fast, and it should be processed fast. Variety
means that a Big Data resource should have a variety of data
structures and formats, both unstructured and structured.
GitHub as the main software engineering resource meets these
criteria and is suitable for Big Data Analytics in the area of
software development lifecycle (SDLC).
In this paper a new framework is proposed to collect
GitHub data through a distributed REST API and using
Map/Reduce and NoSQL DB as a general approach in Big
Data for data collection and pre-processing. In the data

978-1-4799-7752-9/15/$31.00 ©2015 IEEE 65

 2015 3rd International Conference on Information and Communication Technology (ICoICT)
processing phase, unsupervised machine learning techniques
are used for clustering GitHub project source code based on the
metrics and features calculated in the pre-processing phase. The
result of this clustering is analysed and will be presented to
software engineers, project managers and researchers. This
paper is structured as follows. In the next section a brief
description of the basic concepts used for this research is
provided. In section three, related works is reviewed. Then in
two subsequent sections the main framework and research
methodology are illustrated. Then the preliminary results of the
proposed framework are demonstrated. Finally in the last
section the conclusions are summarized.
II. BASIC RESEARCH CONCEPTS
To provide the background to this research, this section
presents a brief overview of basic research concepts related to
this paper. Mining software repositories and Map/Reduce are
introduced in this section
A. Mining Software Repositories (MSR)
Software repositories contain various data about software
projects histories which can be mined to support software
engineering processes. MSR is the specific field in data science
which focused on analyzing the rich data in software
repositories to discover interesting and actionable information
about the projects, software ecosystems and systems [3]. This
extracted information can be used by software practitioners and
researchers in developing their understandings about software
engineering, decision makings, system maintenance and
empirical validation of their ideas.
Recommending software developer for a reported bug,
predicting defects and efforts in software projects, discovering
API usage patterns in applications, analyzing the developers'
communication topics, extracting functions and class change
patterns [4] and detecting code clones are some of the topics
introduced in MSR [5]. Software intelligence, software
analytics, software development analytics and empirical
software engineering are other terms for mining software
repositories (MSR) which are discussed in various related
research with similar definitions. Three types of repositories
have been used in MSR research which are listed with
repository examples in Table 1 [6].
TABLE 1. DIFFERENT TYPES OF SOFTWARE REPOSITORIES
Repository Type Repository Instances
Cooperation Source Code Repositories (CVS),
Artifacts Repositories
Coordination Issue Tracking, Reputation Systems,
Project Management
Communication Forums, Q&A Websites, Wikis,
Commit Log, User Reports, Code
Comments, Mailing List and Social Media
978-1-4799-7752-9/15/$31.00 ©2015 IEEE
B. Map/Reduce Process
Map/Reduce is a distributed processing technique which is
presented by Google to address large-scale datasets [7].
Map/Reduce is based on distributed divide and conquer
algorithm which has its origins in Lisp and other functional
programming languages such as Scala. It has two main phases
"Map" and "Reduce", and also "Shuffling" phase in between. In
most cases it works based on a <Key, Value> data format. In
the "Map" phase the key values are distributed for processing
in multiple processors and generate intermediate key value
pairs. In the "Reduce" phase all the key values pairs with the
same keys are aggregated by the reducer and after some
computation a single key value will be generated by the reducer.
Together with other reducer results the final result will be
presented. Figure 1 summarizes the Map/Reduce Process. A
very common example of Map/Reduce is word count in large
textual documents that can be used for indexing. In this paper a
new sample based on software engineering repositories will be
discussed which is closer to the topic of research. For using
Map/Reduce on any data we have some restrictions like the
data should be presented in key-values and there must not be
any dependencies among pairs.
Fig 1. Map/Reduce
A sample model for software evolution analysis which is
used on software repositories based on changes in lines of code
(LoC) per source file in each commit version, is presented here
by using the Map/Reduce method [7]. Assume that pairs with
<Project name+ version, Source code file> like <Prj_A 1.0,
a.java> and <Prj_A 1.0, b.java> are the input data to the
mapping phase. Then the LoC for that file is computed and the
new pair <Project name+ version, LoC count> is presented to
the shuffle mechanism like <Prj_A 1.0, 150> and <Prj_A 1.0,
250>. In this phase the aggregated keys are sent to the same
reducer to sum all the values per keys to calculate the <Project
Name+ version, Total LoC> as the result like <Prj_A 1.0, 400>.
By analyzing the results, the process of software evolution is
more obvious based on LoC metrics.
66
 2015 3rd International Conference on Information and Communication Technology (ICoICT)
III. RELATED WORKS
In this section a brief review of previous work related to
this project is presented, which covers research in Big Data in
MSR and GitHub Mining.
A. Big Data Researches in MSR
Most of the research in the field of MSR is focused on
development tasks. Other software development life cycle
phases are not analyzed as well as the development phase [5].
In other cases, most of the research only focuses on one project
and one type of repository, and limited data mining techniques
are applied to discover the valuable information in a special
context [8]. By presenting general Big Data Analytics
approaches and tools in the field of MSR a new approach for
Internet-Scale software repository analysis is presented.
Analyzing developers' interactions, documents, source codes
and artifacts through on-line repositories like GitHub is now
possible with BDA techniques. However, some customizations
in traditional MSR mining algorithms are inevitable. Already
some specific and high-cost approaches for large-scale data
mining on software repositories have been applied but they are
not applicable for other researches. By using Map/Reduce
methods and other general frameworks and tools from the Big
Data domain this work can be done more easily and cost
effectively [9].
B. GitHub Mining
As mentioned in previous sections GitHub is a great
resource for Big Data Analytics in the software engineering
area. The role of developers' profiles in some software
engineering related social media like GitHub and Stack
Overflow is investigated in [10]. This shows how a developer's
profile can affect their reputation and skills assessment by other
developers, and how recruiters look for developers' profiles. In
other MSR research [11] the way developers do the refactoring
process is discussed by analyzing the data gathered from
GitHub and Stack Overflow posts and answers. In GitHub they
analyzed comments on commits messages, issues and pull-
requests. They found that refactoring is related mostly to C#
and Java based projects and only a small portion of comments
are related to refactoring (less than 1%). In research by Thung
et al., [12] developer and project relationships in GitHub were
analyzed by using social network analysis approaches and data
mining techniques. They created two separate graphs which
show developer to developer and project to project
relationships. Their aim was to find which developers are more
active and influential and also what is the relationship between
developers and projects.
GitHub's role in improving group awareness in software
engineering is discussed by Lanubile, Calefato and Ebert. This
awareness can help projects in distributed environment to be
more successful and avoid failure. GitHub can support
awareness as a distributed social coding environment [13].
Although GitHub is a great resource for data mining, some
978-1-4799-7752-9/15/$31.00 ©2015 IEEE
problems are faced by data scientists who work on GitHub.
Some of the projects on GitHub are not real software
development projects, or are not active and many of the
projects do not use all the events and features of GitHub. Data
scientists should be aware of these issues when analyzing
GitHub [14]. A further data source for data analytics on GitHub
has been collected in a relational model through the torrents
protocols [15]. In other research GitHub users' social activities
were observed. The authors analyzed people with a high
number of followers to find the role of transparency in GitHub
environment [16].
IV. FRAMEWORK
To present our approach for Big Data Analytics, a
framework is proposed which describes the phases of a data
mining process through GitHub as the goal of this research.
Figure 2 schematically explains these processes. In the first
step a distributed application is needed to extract the required
data from GitHub by using its pre-defined REST APIs. The
GitHub API returns its data in JSON format through service
calls and it has some restrictions on usage which forces us to
use a distributed application which is presented in figure 2 as
GitHub API Extractors. The retrieved data will be stored in a
No-SQL document storage DB and a pre-processing phase will
be run on this data through using the Map/Reduce technique. In
NoSQL DB the extracted history of projects, developers and
source code are stored. The calculated metrics and features of
object oriented projects will be gathered in a Relational DBMS
and the required data will be extracted through SQL queries in
the form of CSV for applying any clustering algorithm through
different tools. As the size of stored data in the NoSQL DB is
large, Map/Reduce is applied for metric calculation process.
The main contributions of this framework to MSR research are
1- using general and cost-efficient Big Data approaches such as
NoSQL, distributed processing and Map/Reduce in MSR. 2-
Presenting a general framework for data analysis on large-scale
open access socio-technical archives like GitHub. The
presented framework is extendable for most of general data
mining techniques.
V. METHODOLOGY
As the proposed frameworks describes, a service oriented
application is being developed to extract GitHub data and store
it as JSON documents in MongoDB. As a seed to the
distributed nodes of the application the URLs of object oriented
GitHub repositories are used. MongoDB itself supports
lightweight Map/Reduce which can be queried across (Hadoop
Distributed File System) HDFS. In addition a metric calculator
program is developed in Java which calculates object oriented
interactions and historical information about the source codes
and projects. Some of the applied metrics are LoC, number of
methods, number of statements in methods, depth block
complexity, number of developers who worked on a source,
number of changed lines, average of changed lines per commit,
project size, programming language and average of co-changed
files. Since it is important for Map/Reduce programming, the
67
 2015 3rd International Conference on Information and Communication Technology (ICoICT)
data should not be dependent and all of the mentioned metrics
in the key value format are not dependent on others. Finally the
result of the queries and programs are stored in My SQL and
the extracted CSV file are fed to WEKA for K-Means
clustering. The results of clustering can be analysed for
relevance to end-users for future evaluation. Currently, a proof
of concept has been developed to demonstrate the principle of
the framework, and some initial results have been derived.
Fig 2. Proposed Framework
VI. PROOF OF CONCEPT
In this section some preliminary results from the proposed
framework are introduced. The application is developed based
on the described methodology and the results of data mining
process support the concepts mentioned in this paper. We
applied K-Means clustering on source code samples to cluster
them based on their level of difficulty for developers with
different levels of experience and knowledge in GitHub to
work with. Source code from C#, Java and C++ projects was
analyzed, and the historical information and object oriented
metrics were used as clustering features. Figure 3 shows the
result of this clustering in box-plots with different perspectives;
lines of code (LoC), percentage of comment lines and number
of changes. We assume that these three metrics can impact on
the ease of modification.
These plots show that source code which belongs to
cluster_2 and cluster_3 is relatively difficult to work with
because it has changed a lot, the size is big and there have been
many changes by different developers. Cluster1 is a good
cluster because it has a limited number of developers, enough
978-1-4799-7752-9/15/$31.00 ©2015 IEEE
comments and a low rate of change. We should note the fact
that a file which changed at a low rate in the past will have a
low probability for changes in future.
Fig 3. the result of clustering on GitHub source code
VII. CONCLUSION AND FUTURE WORKS
This paper proposed usage of General Big Data Analytics
approaches and techniques like Map/Reduce and NoSQL on
Internet-scale software engineering archives. GitHub as a social
coding online environment is selected for this research. GitHub
along with its version controlling and issue tracking
repositories provides facilities for social networking among
software engineers. GitHub API is used to extract data from
GitHub and a general framework is proposed for data analysis
on GitHub based on Big Data approaches. From the extracted
source code and projects from GitHub some metrics and
features are calculated for a mining process. We have presented
some preliminary results that act as a proof on concept for the
framework. These results have applied a set of metrics that may
be useful in a recommender system for newcomers to open
source software development by identifying which source code
components may be more easily modified.
68
 2015 3rd International Conference on Information and Communication Technology (ICoICT)
For future work the result of this framework can be used on
other software analytical systems and some visualization
techniques can be applied on the extracted information. Using
the general data MSR approaches on the result of this
framework and comparing the result with specific-sized MSR
research is the other future work.
REFERENCES
[1] A. Guzzi, A. Bacchelli, M. Lanza, M. Pinzger, and A. v.
Deursen, "Communication in open source software
development mailing lists," in Proceedings of the 10th
Working Conference on Mining Software Repositories,
2013, pp. 277-286.
[2] J. J. Berman, Principles of big data: preparing, sharing,
and analyzing complex information: Newnes, 2013.
[3] M. Halkidi, D. Spinellis, G. Tsatsaronis, and M.
Vazirgiannis, "Data mining in software engineering,"
Intelligent Data Analysis, vol. 15, pp. 413-441, 2011.
[4] T. Zimmermann, A. Zeller, P. Weissgerber, and S. Diehl,
"Mining version histories to guide software changes,"
Software Engineering, IEEE Transactions on, vol. 31, pp.
429-445, 2005.
[5] M. W. Godfrey, A. E. Hassan, J. Herbsleb, G. C. Murphy,
M. Robillard, P. Devanbu, et al., "Future of mining
software archives: A roundtable," Software, IEEE, vol. 26,
pp. 67-70, 2009.
[6] A. E. Hassan and T. Xie, "Software intelligence: the future
of mining software engineering data," in Proceedings of the
FSE/SDP workshop on Future of software engineering
research, 2010, pp. 161-166.
[7] W. Shang, B. Adams, and A. E. Hassan, "An experience
report on scaling tools for mining software repositories
using mapreduce," in Proceedings of the IEEE/ACM
international conference on Automated software
engineering, 2010, pp. 275-284.
[8] S. Livieri, Y. Higo, M. Matushita, and K. Inoue, "Very-
large scale code clone analysis and visualization of open
978-1-4799-7752-9/15/$31.00 ©2015 IEEE
source programs using distributed CCFinder: D-CCFinder,"
in Proceedings of the 29th international conference on
Software Engineering, 2007, pp. 106-115.
[9] W. Shang, Z. M. Jiang, B. Adams, and A. E. Hassan,
"MapReduce as a general framework to support research in
Mining Software Repositories (MSR)," in Mining Software
Repositories, 2009. MSR'09. 6th IEEE International
Working Conference on, 2009, pp. 21-30.
[10] L. Singer, F. Figueira Filho, B. Cleary, C. Treude, M.-A.
Storey, and K. Schneider, "Mutual assessment in the social
programmer ecosystem: an empirical investigation of
developer profile aggregators," in Proceedings of the 2013
conference on Computer supported cooperative work,
2013, pp. 103-116.
[11] G. Destefanis and M. Ortu, "Position Paper: Are
Refactoring Techniques Used by Developers? A
Preliminary Empirical," presented at the International
Workshop on Refactoring & Testing (RefTest), 2014.
[12] F. Thung, T. F. Bissyandé, D. Lo, and L. Jiang, "Network
structure of social coding in GitHub," in Software
Maintenance and Reengineering (CSMR), 2013 17th
European Conference on, 2013, pp. 323-326.
[13] F. Lanubile, F. Calefato, and C. Ebert, "Group Awareness
in Global Software Engineering," IEEE software, vol. 30,
pp. 18-23, 2013.
[14] E. Kalliamvakou, G. Gousios, K. Blincoe, L. Singer, D. M.
German, and D. Damian, "The promises and perils of
mining GitHub," in Proceedings of the 11th Working
Conference on Mining Software Repositories, 2014, pp. 92-
101.
[15] G. Gousios, "The GHTorent dataset and tool suite," in
Proceedings of the 10th Working Conference on Mining
Software Repositories, 2013, pp. 233-236.
[16] L. Dabbish, C. Stuart, J. Tsay, and J. Herbsleb, "Social
coding in GitHub: transparency and collaboration in an
open software repository," in Proceedings of the ACM 2012
conference on Computer Supported Cooperative Work,
2012, pp. 1277-1286.
69