Synergy among software, cybersecurity, and artificial intelligence

(AI) engineering disciplines will enable future critical missions in

defense, national security, and other domains. Missions of the future
will be characterized by multi-domain planning and execution, real-
time operations in dynamic environments, a broad global context in a
world that is increasingly interconnected, and the need for adaptive
human-machine interfaces to manage complexity and respond to
opportunity. The Carnegie Mellon University Software Engineering
Institute (CMU SEI) envisions that a confluence of advances in those
disciplines will support an automated and secure software lifecycle –
including the supply chain.

In this blog post, I review the origins and interactions of the software,
cybersecurity, and AI engineering disciplines and posit how their
interrelationships would contribute to the intelligent systems of the
future.

Engineering Disciplines for Software, Cybersecurity,

and AI Are in Different Stages of Development
Software engineering has evolved into a proven discipline over several
decades. The U.S. government established the SEI in 1984 to advance
the state of the practice of software engineering, and since then we
have led development of crucial software engineering elements,
including software architectural risk reduction, non-functional quality
attributes, and architectural modeling. Software engineering
practices—developed, proven, matured, and codified over many
years—foster improvement across the software lifecycle, from design
and development through testing and assurance. Thanks in large part
to the widespread transition of effective software engineering
practices into common use, today’s software-reliant systems are
increasingly affordable, trustworthy, and evolvable, and succeed in
achieving their required performance goals in delivered products.
Cybersecurity engineering is newer, dating roughly from the Morris
Worm incident in 1988, which prompted the Defense Advanced
Research Projects Agency (DARPA) to fund creation of the CERT
Coordination Center (CERT/CC, now CERT Division) at the SEI. Building
on insights from the field of software engineering, cybersecurity now
consolidates the tools and analyses used in stages of the software-
development lifecycle to ensure effective operational results. It
reduces security weaknesses through, for example, secure
coding practices; mitigates and responds to threats;
increases network situational awareness; and enables
the assurance of critical software and information systems.

Artificial intelligence was first conceived in the 1950s. Carnegie Mellon

has been on the forefront of AI since collaborating in the creation of
the first AI computer program, Logic Theorist, in 1956. It also created
perhaps the first machine-learning (ML) department, studying how
software can make discoveries and learn with experience. Carnegie
Mellon's Robotics Institute has been a leader in enabling machines to
perceive, decide, and act in the world, including a renowned
computer-vision group that explores how computers can understand
images. As occurred in the disciplines of software engineering and
cybersecurity engineering, AI practices and applications are now
evolving from origins in craft, practiced by talented early adopters. We
are seeing an explosion today of scientific and commercial
applications of AI created by skilled craftspeople applying increasingly
well-established development procedures and practices. A discipline
of AI engineering is emerging that will be practiced by educated
professionals and characterized by research-based, validated analysis
and theory. This discipline will guide the creation of AI systems that
are robust and secure, scalable, trustworthy, and
importantly, human-centered. AI engineering builds on a strong
foundation of software engineering and cybersecurity, without which
progress in this field would not be possible.
If software, cybersecurity, and AI engineering disciplines are used
together, the resulting systems could see risk reduction in the supply
chain, software/data development pipeline, and operation. Research
and development work at the SEI is investigating the interaction of
those disciplines.

Software Engineering for AI Systems

The SEI-led study and research roadmap Architecting the Future of
Software Engineering: A National Agenda for Software Engineering
Research & Development calls for empirically validated practices and
verification methods, tools, and practices to engineer AI-enabled
software. Among the SEI research projects aiming to provide
verification methods is one to automatically detect and avoid
inconsistences between assumptions and decisions that create
delays, rework, and failure in the development, deployment, and
evolution of ML-enabled systems.

In addition, a multiyear collaboration among the SEI, Georgia Tech,

Kansas State University, Galois, and Adventium Labs researchers is
developing architecture tools to analyze the impact of AI functions on
the assurance of safety-critical systems.

AI for Software Engineering

The SEI study Architecting the Future of Software Engineering: A National
Agenda for Software Engineering Research & Development notes that “AI-
enabled and other automated capabilities will enable developers to
perform their tasks better and with increased quality and accuracy.”

One area for improving developers’ tasks is in the

necessary refactoring, often on a large scale, of software code. SEI
researchers—working with experts from CMU and other
universities—developed a tool to automate the isolation of the vast
majority of connections that need to be changed for the system to be
evolved rapidly and cost-effectively.

Another area where SEI researchers apply AI to developers’ tasks in

in automating code repair. This work, undertaken with government
collaborators, is developing automated source-code transformation
tools to remediate vulnerabilities in code that are caused by violations
of rules in the CERT Secure Coding Standards.

The Architecting the Future of Software Engineering study notes, as well,

that AI can aid software architecture reconstruction for the
modernization of legacy systems, an area pertinent in DoD reliant on
established systems.

Software Engineering for Cybersecurity

In June 2023, the SEI organized the Secure Software by Design
Conference to encourage collaboration toward improving the state of
a holistic secure development approach. Participants discussed threat
modeling, security requirements development, secure software
architectures, DevSecOps, secure development platforms and
pipelines, software assurance, secure coding practices, software
testing, and other topics.

One of the presentations examined the Acquisition Security

Framework for Supply Chain Risk Management in the context of the
software bill of materials (SBOM) concept. The talk described the
potential of using a properly integrated SBOM into effective cyber risk
management processes and practices and introduced the SEI SBOM
Framework of practices for managing vulnerabilities and risks in third-
party software.

Cybersecurity for Software Engineering

In the course of creating tools for the automated prioritization of
static analysis alerts, SEI researchers developed the Source Code
Analysis Integrated Framework Environment (SCAIFE) application
programming interface (API). An architecture for classifying and
prioritizing static analysis alerts, the SCAIFE integrates a wide variety
of static analysis tools using the API. The API is pertinent to
organizations that develop or research static analysis alert auditing
tools, aggregators, and frameworks. Building on that body of work, SEI
researchers are proposing, in recently initiated research, to create a
tool that can automatically repair 80 percent of alerts in 10 categories
of code weaknesses.

Assuring software system security also means finding adversaries in

the network before they can attack from the inside using cyber threat
hunting. Unfortunately, this approach is often costly and time-
consuming, to say nothing of the particular skills needed. SEI
researchers are addressing these shortcomings by applying game
theory to the development of algorithms suitable for informing a fully
autonomous threat hunting capability.

Cybersecurity for AI
Trustworthiness is key to the acceptance of results produced by AI
systems. Those systems using ML are susceptible to attacks that
cause those results to be less reliable. SEI research is addressing
issues with the secure training of ML systems. In this collaborative
work with CMU, a team is ensuring that an ML system does not learn
the wrong thing during training (e.g., data poisoning), do the wrong
thing during operation (e.g., adversarial examples), or reveal the
wrong thing during operation (e.g., model inversion or membership
inference). To support this research, the team created the publicly
available Juneberry framework for automating the training,
evaluation, and comparison of multiple models against multiple
datasets.

AI for Cybersecurity
The use of AI and ML for cybersecurity in, for example, anomaly
detection supports faster analysis and faster response than can be
provided by human power alone. In the SEI Artificial Intelligence
Defense Evaluation project, funded by the Department of Homeland
Security’s Cybersecurity and Infrastructure Security Agency (CISA), a
team is developing a means to test AI defenses. In early work, the
research team created e virtual environment representing a typical
corporate network and used the SEI-developed GHOSTS framework to
simulate user behaviors and generate realistic network traffic.

Researchers are also seeking ways to improve human use of AI

system results, including but not limited to those for cybersecurity.
This research is developing the Human-AI Decision Evaluation System,
a test harness for investigating AI-assisted human decision making in
a variety of simulation environments. The research team has
integrated the harness into game environments to observe the effect
of AI decision-support systems on gameplaying outcomes.

How You Can Support the Evolution of the Intelligent

Systems of the Future
As the disciplines of software, cybersecurity, and AI engineering
converge and cross-pollinate, SEI looks forward to learning from pilot
projects within the software-development community about
successes and challenges that developers and users experience. The
results of real-world applications in exercises will show us where pain
points emerge that require further research and development.

Undergraduate and graduate educational curricula, as well as

continuing education and professional development, must continue
to evolve to keep pace with the rapid developments in practice that I
have outlined in this post. Degree programs, certificates, and
certifications will go a long way toward promoting the integration of AI
with software and cybersecurity engineering, taking some of the
mystery out of the craft and professionalizing the maturation of
proven, trusted practices and applications. The SEI has contributed to
establishing curricula for software engineering and cybersecurity
engineering, and we plan to apply our experience to the field of AI
engineering in the future.

Future missions will need technologically advanced and engineered

intelligent systems that can scale quickly and gracefully to adapt to
different environments, generate data to respond dynamically to
changing conditions, and evolve with new mission parameters
(i.e., cyber-physical systems driven by intelligence). Through the
synergistic combination of software, cybersecurity, and AI
engineering, these intelligent, resilient, evolvable systems will be able
to scale, adapt in real time, and generate and use data to respond to
their environments. Reduction of the risk profile of such systems will
give their users greater confidence and trust, critical factors whenever
AI is added to the functionality of mission-critical systems.