The Usage of Bots in Open Source Software
Development
1st Sankalp
2021-IMT-084
ABV-Indian Institute of Technology and Management
Gwalior, India
sankalp25103@gmail.com
3rd Gaurang Sondur
2021-IMT-036
ABV-Indian Institute of Technology and Management
Gwalior, India
gssondur28@gmail.com
I. BACKGROUND
In the realm of open-source software development, the re-
liance on automated tools and bots has become commonplace.
These automated entities play a pivotal role in managing
various aspects of open-source projects, including tasks such
as issue management, code review, and continuous integration.
Their contributions significantly enhance project efficiency
and code quality. However, amid this synergy between hu-
man contributors and automated systems, it is imperative to
distinguish between human and bot-generated contributions.
This distinction serves as the linchpin of maintaining the
integrity of open-source collaboration. The research paper
under discussion offers a detailed methodology for the identi-
fication and review of bot activities within GitHub repositories
within the context of open-source software development. This
methodology is an intricate blend of automated techniques
and manual review processes, meticulously designed to ensure
the reliability and precision of data. The approach various
technologies, including machine learning, natural language
processing, web scraping, and the GitHub API. Moreover, it
employs statistical metrics like Cohen’s kappa to gauge inter-
rater reliability, further refining the review protocol through
iterative discussions. .
II. P ROCESS AND T ECHNOLOGIES U SED
A. Automated Bot Identification with BODEGHA
Central to the methodology is the deployment of an au-
tomated bot identification method known as BODEGHA.
BODEGHA operates as a machine learning classifier, driven
by the application of Natural Language Processing (NLP) and
machine learning technologies. NLP, a field at the intersection
of linguistics and computer science, empowers the algorithm
to decipher language patterns and unearth pertinent insights
from comments. Simultaneously, machine learning algorithms
2nd Lata
2021-IMT-059
ABV-Indian Institute of Technology and Management
Gwalior, India
imlata1111@gmail.com
4th Pranav Pawar
2021-IMT-074
ABV-Indian Institute of Technology and Management
Gwalior, India
pranav@example.com or ORCID
come into play, classifying these patterns as either originating
from human or bot interactions.
B. Data Collection through the GitHub API
For the collection of activity data, the researchers have
harnessed the GitHub API. This programmatic interface serves
as the gateway to accessing and retrieving data from GitHub
repositories. By interfacing with the API, researchers can
systematically extract information concerning repositories,
commits, issues, pull requests, and user interactions. The API
emerges as a foundational technological tool for expeditious
data aggregation and analysis, ensuring the data’s accuracy
and comprehensiveness.
C. Review of GitHub Profiles and Marketplace/App Listings
To validate the bot activities flagged by BODEGHA, a
meticulous manual review of GitHub profiles and examination
of GitHub Marketplace/app listings is undertaken. This valida-
tion process necessitates the use of web scraping techniques.
Web scraping is an automation technology that facilitates the
systematic extraction of data from web pages. In this context,
it enables the gathering of critical information from user
profiles, encompassing data such as usernames, descriptions,
and affiliations with marketplace apps or services, fostering
a comprehensive understanding of the roles of specific bots
within the open-source ecosystem.
D. Manual Review of Comments and Activities
Another vital facet of the methodology encompasses a
hands-on manual review of the content found in recent com-
ments and activities associated with GitHub accounts. This
qualitative inspection leverages text analysis and sentiment
analysis techniques.
E. Data Reliability and Updating of Review Protocol
In an endeavor to ensure the reliability of data and the
robustness of the methodology, the researchers have metic-
ulously implemented statistical measures. This includes the
assessment of interrater reliability, employing Cohen’s kappa
as a metric to gauge the agreement between human raters.
Statistical software or libraries designed for data analysis and
statistical computations are instrumental in this process.
III. P ROBLEM S TATEMENT
Open Source Software (OSS) development has become a
very large domain in software engineering which involves
a huge base of contributors, users, and other stakeholders.
Various social coding platforms like GitHub, GitLab and Slack
involve a huge base of such contributors. At the same time,
the CI/CD (continuous integration, delivery, and deployment)
model is also fast-growing and requires urgent demands.
Managing such contributors and handling the requirement of
CI/CD raises the need of automation in OSS development.
Huge company projects present on various GitHub repositories
also require automation as it is infeasible for employees to
respond to every query of the contributors.
IV. S OLUTION
The report discusses the solution of this problem by ana-
lyzing the use of bots in OSS development. Bots are software
agents used to perform software engineering tasks which
help to automate the process of building any software. Bots
are seen as a promising approach and can be used to deal
with complex OSS projects. Such bots are used to automate
repetitive tasks such as managing issues and pull requests,
building and executing test sites, providing chatbot services to
the contributors and handling small queries of them. The use
of bots helps in saving human labor.
The report analyzed 1000 popular GitHub repositories and
found out that 61% of the most popular projects have adapted
bots in their projects. It can be predicted that in the near future
more and more projects will start adopting bots to perform
repetitive tasks. In the past decade, the technology has been
developed enormously such as the use of CI/CD tools and
large-scale collaboration between the contributors have caused
rapid developments to automation techniques used in bots to
handle social tasks so that people can focus on innovative
tasks.
Software engineers put significant effort towards development
of bots. Creation of these bots is done by bot frameworks.
Different types of bots need to be built for performing various
automation tasks depending upon the project’s demands, so
to separate the bot creation logic for different platforms these
frameworks are used.
V. A NALYSIS OF THE PAPER
This paper investigates the proliferation of bots in the open-
source community. Over the past decade, the open-source com-
munity has experienced significant growth, whether through
the adoption of open-source practices by new companies or
the increasing number of developers joining this community.
A major contributing factor to this growth has been the
presence of various open-source programs, such as Google
Summer of Code, Outreachy, or MLH. Concurrently, the rapid
and ongoing adoption of the continuous integration, delivery,
and deployment (CI/CD) model places urgent demands on
rapid iterations, imposing significant pressure on repository
maintainers. However, the rise of bots has provided substantial
relief to these maintainers.
This paper offers an in-depth investigation of developer bots,
and it is well-written and meticulously maintained. The authors
provide a clear and concise overview of the topic, discussing
their study’s findings in detail. To accomplish this, they
conducted thorough research on 1,000 popular repositories on
GitHub, revealing that 61% of these repositories utilize bots
for various tasks.
A. Data Collection
Data collection was a meticulous task. The first step was to
select only repositories in the sample that contained software
development projects, while excluding non-software engineer-
ing (SE) projects. To achieve this, the BODEGHA bot identi-
fication method was used to detect bot activities. The selected
repositories were then manually checked for data reliability,
achieving an interrater reliability of 0.645 (Cohen’s kappa),
indicating the effectiveness of their data collection method.
B. Analysis of Data
For each identified bot, several pieces of data were
extracted, including the bot’s service-providing medium,
main functionality, service and source code availability, de-
sign/creation, and framework.
Based on the collected data, the authors categorized bots
into six categories:
• CI task assistance
• Issue and pull request management
• Code review assistance
• Dependency and security
• Developer and user community support
• Documentation generation
The most common of these tasks is CI task assistance.
Many bots which they found are capable of performing a
wide range of tasks and can even be customized. Therefore,
it is accurate to say that they have become like butlers for
maintainers.
C. Bot Development
However, for most of the bots, their source code is not
available. The authors claim that there may be two major
reasons behind this. First, there could be security and business
concerns, as bots often have high-level permissions in repos-
itories. Making their code public could expose vulnerabilities
that attackers could easily exploit. Second, many customized
bots use existing services whose code is already available,
which gives developers little motivation to release their own
code. These reasons are well justified.
 It has also been observed that the emergence of bots has
led to the development of various frameworks for building
them, including OpenBot, Fabric Bot, and Windows Commu-
nity Tool. These frameworks have empowered developers to
concentrate on logic rather than writing extensive code. We
believe that the continued development of such frameworks
will significantly contribute to the proliferation of bots within
the open-source community.
D. Conclusion
In conclusion, the authors have conducted a comprehensive
analysis, and their paper is well-organized and structured.
In their study, they found that over 60% of repositories are
utilizing bots, indicating that bots have become prevalent in
open-source software (OSS). However, these bots are not yet
sophisticated enough to handle complex tasks. Multitasking
bots are also rapidly on the rise and are likely to dominate
the field in the near future. Additionally, they discovered that,
despite the increased use of bots, more than 65% of bot source
codes remain unpublished.
VI. P ROS A ND C ONS
A. Pros
• The paper asserts that software agents and bots automate
repetitive and routine tasks, such as managing issues and
pull requests, building and executing test suites, thereby
often substituting for human labor.
• Many bots have specialized in multiple tasks, serving
as butlers for repositories. For example, RepoKit-teh
performs various tasks, including checking the format of
a pull request, automating tests, assigning users to issues,
and merging pull requests. Some organizations have even
customized private butler bots for CI/CD practices, like
Googlebot and Facebook GitHub Bot, enhancing flexibil-
ity.
• Open source software (OSS) projects typically have few
full-time employees, making it challenging to address
queries. Trained bots capable of answering frequently
asked queries save significant time.
• During specific events like GSoC, Outreachy, and Hack-
toberfest, where contributors surge, bots prove helpful
in managing issues and reviewing PRs due to limited
maintainers.
• These automation techniques, i.e., bots, support devel-
opers through various service mediums, with options
for user profiles and platform applications. Practitioners
adopt bot services, considering performance, privacy, and
security tradeoffs.
B. Cons
• While these bots have saved substantial human effort,
they also have limitations, such as not being designed for
complex tasks and lacking interactivity during human-bot
communication.
• The paper focuses on GitHub, although alternatives like
Gitlab are also a popular choice and offer more security
for OSS projects.
• Bots primarily perform simple, rule-based automation,
limiting their application scenarios.
• Over 69% of bots in OSS remain private, raising security
and business concerns due to high permissions in repos-
itories. without transparently sharing their implementa-
tion and accessibility outside projects and organizational
ecosystems. This raises security and business concerns
due to the high level of permissions bots often possess
(above write access) in repositories.
• For data collection, the authors gathered activity data
from the most recent list of event actors in commits,
issues, and pull requests, as well as reviewing these
activities themselves. However, a broader analysis of bot
usage in OSS could have been achieved by surveying
maintainers and contributors, as they are the primary
users of the bots.
VII. I MPROVEMENT
A. Scalability and Resource Optimization
As the number of bots and their complexity increase, it’s
crucial to research ways to optimize resource usage. This
includes strategies to minimize resource-intensive operations
and ensuring that bots don’t inadvertently contribute to system
overloads. Research into resource-efficient bot design and
scaling mechanisms can be valuable.
B. Enhanced Bot Communication and Functionality
The authors mentioned that practitioners expect more in-
telligent features in bots’ communication and functionality.
To address this, future research could delve into specific im-
provements, such as natural language processing capabilities
for more interactive and context-aware communication with
users. Bots that can understand and respond to user queries
more effectively can greatly enhance their usability.
C. Bot Security Audits
Given the concerns about bot security, conducting regular
security audits of bot implementations could be beneficial.
This includes vulnerability assessments, code reviews, and
testing for potential exploits. Research efforts should focus on
developing best practices and tools for ensuring the security of
bots, especially those with elevated permissions in repositories.
C ONTRIBUTION
• Pranav Pawar: Conducted a background check and re-
viewed the history of bots in open source. Also, explained
the methodology used by the authors to extract data.
• Gaurang Sondur: Described the problem statement ad-
dressed by the authors and the reasons behind it. Addi-
tionally, outlined the proposed solution.
• Sankalp: Performed a comprehensive analysis of the
paper and provided a summary of its key points.
• Lata: Identified the pros and cons of the paper and the
bots, and offered suggestions for improving both.
 R EFERENCES
[1] Z. Wang, Y. Wang and D. Redmiles, ”From Specialized Mechanics
to Project Butlers: The Usage of Bots in Open Source Software
Development,” in IEEE Software, vol. 39, no. 5, pp. 38-43, Sept.-Oct.
2022
[2] 2022. Proceedings of the 13th Asia-Pacific Symposium on Internetware.
Association for Computing Machinery, New York, NY, USA.
[3] https://marutitech.com/complete-guide-bot-frameworks/
[4] https://github.com/googleapis/repo-automation-bots