ebook

AI Governance Playbook:
Innovation Meets Security
for Responsible AI Adoption
How to Adopt AI and Multi-LLM Strategies
in a Secure, Governable Way

KongHQ.com
1 © Kong Inc. AI Governance Playbook: Innovation Meets Security for Responsible AI Adoption
 Executive summary
The breakneck pace of AI innovation and the potential risks
associated with improper usage necessitate a well-defined, robust
governance playbook for responsible adoption.
Embrace AI as a transformative force
Organizations that recognize AI’s transformative potential early on and embrace it strategically
will gain a significant competitive advantage.

Strike a balance between rapid adoption and responsible usage

A well-defined governance playbook is crucial to ensure that AI is adopted responsibly without
introducing risks to customer data, security, or compliance violations.

Implement AI as an internal platform service

To effectively manage AI adoption across teams and use cases, organizations should consider
implementing AI as an internal platform service. This centralized service should facilitate
the consumption of multiple LLMs, enforce security and governance policies, and provide
observability and analytics on AI traffic.

Master the challenge of model fine-tuning

A major challenge with implementing AI intelligence is fine-tuning the models. Organizations
must develop a strategy for extracting and sanitizing data from various sources (e.g., databases,
documents) and continuously fine-tuning their models with this data, without compromising
security or compliance. This process involves collaboration between stakeholders, including
database teams and compliance teams, to establish a responsible data usage playbook.

Embrace multi-LLM strategies and advanced prompt engineering

Organizations should adopt a multi-LLM strategy, leveraging different LLMs for different use
cases. Effective prompt engineering is also essential. Implement centralized prompt templates,
prompt firewalling, and prompt decoration to ensure compliance, improve prompt management,
and enforce contextual rules for interpreting requests and responses.

Prioritize AI governance, observability, and cost management

For AI governance, implement strict controls and approval processes for AI prompts, enforcing
compliance rules, and restricting unauthorized or potentially harmful AI usage. Consider
collecting L7 AI metrics (such as request/response tokens, LLM providers, and models used) to
gain insights into AI traffic and usage patterns. Implementing orchestration between cloud and
self-hosted LLMs can help with cost management.

By developing a comprehensive, multi-faceted AI governance playbook with these considerations

in mind, organizations can harness the power of AI while mitigating risks, ensuring responsible
usage, and maintaining a competitive edge in an AI-driven world.

2 © Kong Inc. AI Governance Playbook: Innovation Meets Security for Responsible AI Adoption
 Table of contents

02 13
Executive summary The AI governance playbook
• Benefits of an AI playbook
• Balance innovation and control
with an internal service

04 16
Introduction Introducing the AI gateway
pattern
• A free, open source AI gateway

05 22
The state of AI in the enterprise Conclusion
• Why AI adoption is taking off now
• How organizations can embrace AI
• The relationship between AI and APIs

08
AI adoption challenges
• Determining best use cases
• Multi-LLM adoption
• Cost optimization
• Fine-tuning AI models
• The need for observability

3 © Kong Inc. AI Governance Playbook: Innovation Meets Security for Responsible AI Adoption
 Introduction
AI is top of mind for nearly every executive in the world. Across
industries, organizations are scrambling to adopt and integrate
generative AI and large language models (LLMs) into their products
and services.
But in the race to innovate and keep ahead of the competition, how can organizations ensure
concerns around AI governance and security aren’t left in the rearview mirror?

In this eBook, we’ll cover the keys to properly implementing AI in your organization and
considerations to keep in mind when developing your organization’s AI governance playbook.

4 © Kong Inc. AI Governance Playbook: Innovation Meets Security for Responsible AI Adoption
 The state of AI in the
enterprise
The word “disruptive” may be overused to the point of losing all
meaning, but it would be short-sighted to consider AI anything less than
radically disruptive.
But there’s not much that can be said about
the disruptive impact of AI that you haven’t
seen before in countless articles, online
ramblings, and hyperbolic TV interviews from
Big Tech leaders — some comparing AI to fire
or electricity in terms of how important they
believe it will be to the future of humanity.
Why AI adoption taking off now
The term “artificial intelligence” was coined in
the 1950s. In the decades since, AI has been
applied in various contexts. But given the long
history, why haven’t we witnessed a massive
AI adoption until recently? And why is that only
massive organizations have primarily reaped
the benefits of AI? It’s a combination of the
right innovations and economics that finally
made sense.
More recent AI innovations are enabled by
advances in hardware. We now have enough
computing power to create compelling training
models. We can also point to a specific
technical innovation: transformers, a deep
learning architecture that came out of Google.
Another reason that AI has been rarely
embraced outside of the realm of big
enterprise players is that the economics of AI
5 © Kong Inc. AI Governance Playbook: Innovation Meets Security for Responsible AI Adoption
So let’s shift the focus from what’s down
the road to the reality of right now and how
generative AI and LLMs (e.g., OpenAI’s
ChatGPT, Google’s Bard, Meta’s Llama) can
be used. And, crucially for businesses, we’ll
talk about how they can be used securely by
developers and organizations around the world
and across industries.
have historically been less than ideal. First,
you needed data. A lot of data. Second, you
were competing with the human brain, which
has proven itself to be adept at most tasks
over the years. Add in the fact that it was
expensive to assemble the teams required
for AI projects, and you can see why AI was
something normally only explored in jumbo-
sized enterprises. But today, that’s changed.
Consider an application like AI art generator
Midjourney, where you enter a prompt to
generate an image. The inference cost to
generate an image may be something like
1/100th of a penny — and it only takes a
moment. In contrast, hiring a designer to make
the image might cost you $100 an hour, and it
would probably take longer. Compared, we’re
looking at a difference of orders of magnitude.
 AI economics have also faced historical challenges because AI tackled
problems that humans excel at — and where errors meant significant
consequences, demanding extensive investment.
So while we’re “there” with things like rendering navigate their environments, source food, and
art or summarizing documents, the economics avoid predators.
still aren’t great yet for other AI areas.
However, there are other surprising areas where
Self-driving cars are a technology with AI is already proving itself a powerful tool and
investments totaling around $75 billion, and ally. The current wave of generative AI is great
still, the unit economics aren’t superior to those at tackling some things that humans are not
of humans. It’s simply not yet competitive with so good at. Applications of AI like serving as
an offering like Lyft or Uber. Why? Because a creative companion, a coding partner, or a
despite what one might think when driving brainstorming buddy are carving out new use
during rush hour, humans are actually pretty cases, improving the economics of AI, and
good at driving — thanks to evolution and our pushing computers (and businesses) into
ancestors developing the abilities needed to entirely new directions.

How organizations can embrace AI

According to Forbes, 56% of businesses use AI tools for customer service, while 51% have
adopted AI for cybersecurity and fraud management purposes.

As more businesses come to embrace AI,
we’re likely to see similar missteps made
in the past around other unprecedented
innovations. Think back to mobile or the
cloud: not so long ago, there were Fortune
500 companies that wouldn’t allow developers
to use AWS — or employees to use mobile
devices for work purposes.
With new and disruptive technologies, it
takes time for organizations to catch up.
However, the speed with which organizations
are adopting AI outpaces these previous
breakthroughs, which means organizations
need to move faster than ever to keep on top of
the wave of AI innovation.
The key is to understand the adoption cycle:
users will use these tools, so rather than
attempting (and failing) to ban AI technologies
and LLMs, enterprises must figure out how
to address the challenges around them —
like figuring out what this means for data
governance or regulations.
One potential misstep would be to internalize
this in central IT and have an AI strategy that’s
disconnected from the actual use cases
because these are secular, organic movements.
We don’t want to be like the companies in
the 1990s that banned internet browsers
and e-mail in the workplace. Innovative
organizations need to move fast to catch up
with the adoption and figure out intelligent
ways to incorporate it.

6 © Kong Inc. AI Governance Playbook: Innovation Meets Security for Responsible AI Adoption
 The relationship between AI and APIs
APIs (application programming interfaces) are a set of functions and
procedures that act as a bridge between applications by dictating how
services interact.
APIs are essential for AI. They act as hands, eyes, and ears for AI. As the usage of AI increases,
so will the number of APIs that enable them.

The secular tailwinds of APIs are being
strengthened by the adoption of AI. Though
estimates vary on the exact percentage, we
know that the majority of internet traffic today
is API traffic. Now that we have a new category
with AI traffic, how much more will API traffic
grow in the coming years?
APIs will play a crucial role in facilitating
communication between humans and AI
systems, as well as AI systems and other
connected tools. The future will see a surge in
AI-powered agents and applications interacting
primarily through APIs, to a degree where we
can assume the users of the future will be AIs,
and they’ll need APIs to accomplish their tasks.
Let’s assume there are 4 billion people online
today. With AI adoption growing as it is, let’s
assume there may soon be 100 individual
AIs per person online. That’s 400 billion AIs.
Each of these AIs will be like a user interacting
online — but not using a keyboard, mouse, and
monitor. These AIs will interact with APIs. This
is why an API strategy is needed: in the future,
the consumer of your website or product will
almost certainly be AI, not a human.

7 © Kong Inc. AI Governance Playbook: Innovation Meets Security for Responsible AI Adoption
 AI adoption challenges
There are essentially three things we’re currently doing with AI:
We use AI. We train AI. We have AI interact with APIs to perform
operations — making AI itself the client making API calls to access
data or trigger capabilities.
In any of these cases, the operation requires an API. That’s why controlling the usage of AI at the
interface level is an API problem.

Determining best use cases

APIs allow every organization to become a platform, and AI allows for this platform to become
intelligent. It’s a further evolution of platform creation that every organization needs to be agile,
ship products quicker, and provide the best customer experience.

Today, we have data that developers use to
build applications. Developers build services on
top of this data and build APIs to turn the data
and services into a platform that others can
use. AI introduces a new dimension by bringing
intelligence to the platform to build on top of
the APIs and services you’ve already created
by making them more accessible. For example,
retailers might bring more advanced fraud
prevention systems into their e-commerce
platforms or smarter chatbots empowered with
access to a customer’s order history.
At the end of the day, AI or any tool you
implement in your business should serve a
specific purpose. Therefore, an easy way to
consider AI adoption in your organization is to
think about the end use case — or the customer
experience that you hope to build. From
there you can work backwards to determine
if intelligence can help you deliver a better
experience. This gives a much more pragmatic
view of AI adoption in the organization, and
from this vantage point we can figure out how
to implement it.

It’s up to the organization to determine how

intelligent our platform should become. This
creates an opportunity to essentially enhance
every experience that we’re building for our
customers.

8 © Kong Inc. AI Governance Playbook: Innovation Meets Security for Responsible AI Adoption
 Multi-LLM adoption
For organizations looking to leverage AI, there are many LLMs to
choose from. While ChatGPT may be the LLM that first captured the
spotlight, there’s a great deal of competition.
In addition, we’re now seeing that open source models are getting as “good” as the cloud-hosted
models we’re most familiar with. The competition and options we’re seeing are only going to
benefit us as the early adopters of AI.

The most used LLM providers via Kong’s AI Gateway March 2024

Organizations are likely to start with cloud-hosted LLMs. And cloud-hosted models are a fantastic
starting point, but in the long run, they can become expensive. This is why many organizations
begin deploying self-hosted models and allow for orchestration between cloud-hosted and local
models to cut costs, reduce latency, and retain control over potentially sensitive data.

9 © Kong Inc. AI Governance Playbook: Innovation Meets Security for Responsible AI Adoption
 As a bonus, implementing orchestration across multiple LLMs may also offer potential
guardrails to minimize hallucinations and bias and improve the quality of our AI queries.

Open source models are rapidly improving and even surpassing some private models.
Source: ARK Invest

Cost optimization
Early AI adopters are already thinking about many of these problems and looking for best
practices emerging. They already see the need for cost optimization as running millions of
queries on a cloud-hosted model can get pricey. Many of these organizations seek to deploy their
own models, which is one of the reasons open source models are taking off.

For organizations looking to reduce the total
cost of ownership or speed up return on
investment, using less AI isn’t the answer. We
need to find a way to use AI tools as needed
while finding other ways to simplify costs.
In organizations running self-hosted models,
we often see orchestration between the two
models. Many opt to fine-tune their self-hosted
LLM and make it highly available while using
cloud-hosted models as a fallback.

10 © Kong Inc. AI Governance Playbook: Innovation Meets Security for Responsible AI Adoption
 Fine-tuning AI models
Organizations typically start with more general, pre-trained models
with broad knowledge but a lack of specific domain expertise that
may be needed for our use cases.
This is a fine way to start, but we’re not going to get far with this approach. Eventually, we’ll have
to think about how we’re going to fine-tune our models to fully realize the potential benefits of
AI tools.

Extracting the right data and feeding it into
LLMs is a significant hurdle. The data may be
spread across the organization — in databases,
documents, or other unstructured sources —
and accessing and consolidating this data can
be difficult.
Additionally, there are often concerns around
data privacy, security, and compliance
that need to be addressed. The database,
compliance, and other teams may be hesitant
to allow the free use of sensitive data for model
fine-tuning.
How are we planning to fine-tune our models?
Well, this is the million (or even billion) dollar
question. It’s possibly the biggest challenge
around implementing intelligence in the
organization.
There’s no one-size-fits-all solution. Whether we
use cloud or self-hosted, whether we have an AI
service that the platform team is managing or
not, whether we have all of that infrastructure in
place — how do we extract our data and make
sure that AI can build on top of our specific and
unique data?
This intelligence differs between organizations.
This is truly the competitive advantage
that organizations have: using the data and
services they’ve built over time. How do we
train our models on top of this data? This is
a trickier problem to solve. Organizations are
experimenting with various approaches, like
dumping entire databases into the models or
mirroring internal API traffic. However, each
approach comes with its own set of challenges
and considerations.
Ultimately, fine-tuning AI models requires
a well-thought-out playbook that involves
multiple stakeholders across the organization.
It’s not just a technical problem; it’s a people
and process challenge. Creating a playbook
is crucial, but there’s no standard solution —
each organization will need to develop its own
approach based on its data, infrastructure, and
governance requirements.

11 © Kong Inc. AI Governance Playbook: Innovation Meets Security for Responsible AI Adoption
 The need for observability
While the excitement around AI is palpable,
organizations must resist the temptation
to adopt AI haphazardly. We have to focus
on observability. There’s no security if we’re
blind to our traffic. There are many things an
organization would benefit from knowing:
• What models are developers using?
• What prompts are developers using?
• What’s the semantic meaning of prompts
being used (for further analysis on those
prompts)?
12 © Kong Inc. AI Governance Playbook: Innovation Meets Security for Responsible AI Adoption
• How many tokens are being consumed?
• What’s the cost of the AI over a period of
time?
There’s much that goes into this underlying
infrastructure, and we can’t ask our developers
to build all of this from scratch every time
they want to use AI. The implementation of a
playbook has to include the creation of AI as
an internal platform service, and this internal
platform service should perform all these
different types of operations.
 The AI governance playbook
How can your organization ensure the adoption of AI doesn’t become
a Wild West? As the adoption of AI increases in the organization,
developers need to rapidly iterate and build new capabilities without
having to manage the specific cross-cutting concerns around AI usage.
We have to enforce a playbook our developers must follow to consume
AI, train AI, or have AI interact with other APIs.
The idea of implementing a playbook is no different than how many organizations implement any
other technology they may be using (or considering using) today.

However, this step can be easy to overlook given the time to respond feels so much shorter
than other breakthrough technologies. Generative AI and LLMs seemed to come into public
consciousness lightning-fast, and savvy organizations have sought to move with equal speed.
But that doesn’t mean taking shortcuts and sacrificing speed for the business-critical areas of
security and governance.

Many areas need to be addressed if we hope
to ensure responsible AI usage in our products,
including:
1. AI and data security: We must prevent
customer and sensitive data from being fed
into AI/LLMs, which could cause privacy
leaks, security escalations, and potential
data breaches.
2. AI governance: We need to be able to
manage how applications and teams are
consuming AI across all providers and
models with a single control plane to
manage, secure, and observe AI traffic
being generated by the organization.
Without a control plane for AI that gives
this level of visibility and control, the
organization is blind to how teams are
adopting AI in their products.
3. Multi-AI adoption: We should be able to
leverage the best AI for the job and lower
the time it takes to integrate different
LLMs and models. Generalist LLMs and
specialized LLMs are going to be adopted
to cater to different tasks, and developers
may adopt different cloud-hosted or
open source models based on their
requirements or to reduce costs, with OSS
models rapidly catching up in performance
and intelligence.

13 © Kong Inc. AI Governance Playbook: Innovation Meets Security for Responsible AI Adoption
 Benefits of an AI playbook
Organizations that implement their own AI playbook are going to
gain a competitive advantage. AI can accelerate the gap between
organizations in the same industry, potentially changing companies’
fortunes — in both senses of the word.
Those who move with speed and strategy in mind will create vastly superior customer
experiences and expand their businesses faster than the competition.

With an AI adoption playbook in place,
organizations can ensure they’re not slowing
down developers who want to leverage AI
and LLMs. They can create better digital
experiences while implementing standards to
minimize risks around customer data, security,
and compliance when it comes to sharing
potentially sensitive information (that could be
leaked elsewhere) with LLMs and AI tools.
Of course, developers aren’t the only people
using AI tools. You may have people in sales
using Claude to write prospecting emails
or someone in recruiting using ChatGPT to
help with LinkedIn outreach. The good news
is the use cases aren’t that different from
developers and knowledge workers. If you
have, say, an internal chatbot that you want
to allow employees to use, it’s going to be
an application that leverages the same AI
infrastructure that you’re building for your other
AI use cases.
However your people interact with AI tools, to
adopt AI responsibility you need security, you
need to validate prompts (or at least decide
how pre-formed those prompts should be), and
you need observability. By building an internal
service, you can address these concerns for all
of your people.

Balance innovation and control with an

internal service
To improve the productivity of the application teams and to securely and responsibly leverage
AI in the organization, AI needs to become a service offered by the core platform in the
organization and made available for consumption from any product that may want to use it. By
doing so, we can avoid reinventing the wheel when it comes to AI adoption across our teams.

This service can enforce security, governance,
and observability for all the AI traffic that every
team is generating.
This internal service should allow developers
to consume AI across one or more LLMs.
Why more than one LLM? We must plan for
multiple LLMs — cloud or self-hosted — in
our organizations because it’s an inevitability.
Developers may use one specialized LLM that’s
fine-tuned for a specific use case but want to
use a more generalist LLM for other operations.
Different teams may find different LLMs are
better suited for their different tasks.

14 © Kong Inc. AI Governance Playbook: Innovation Meets Security for Responsible AI Adoption
 Next, the platform service should allow the organization to see the
operations people are asking AI technologies to perform.
Typically, this involves implementing some type of advanced prompt engineering that allows us to
create a playbook determining the prompts we want people to be able to use — and which ones
we want to forbid.

We can implement security on AI either We don’t want our application engineers

by limiting the data that the LLM is being
fine-tuned with or by limiting or controlling
access to that information. And to control
the access of that information, we need to
control prompts. Prompt engineering is about
firewalling the prompts (i.e., what things are
allowed and not allowed) and setting a mutual
context for each prompt carried across LLMs.
to change their prompts constantly in their
applications; we want them to send their base
prompt from the applications and then use the
platform service to set those mutual contexts,
which will change over time and can be
managed from the platform itself.
So how do we create this internal platform
service? Enter the AI gateway.

15 © Kong Inc. AI Governance Playbook: Innovation Meets Security for Responsible AI Adoption
 Introducing the AI gateway
pattern
To accelerate the adoption of AI in the organization with the right
level of observability, security, and governance, organizations have
started adopting an AI gateway to provide a distributed AI egress to
any LLM and model that developers may want to use.
An AI gateway gives us a centralized place to manage AI consumption across every team,
whether the AI models are hosted in the cloud or self-hosted.

The AI gateway operates like a traditional API gateway, but instead of acting as a reverse proxy
for exposing our internal APIs to other clients, it’s being deployed as an egress proxy for AI traffic
generated by our applications. That traffic is directed either inside or outside the organization
depending on where the backend AI models are being hosted — in the cloud or self-hosted.

Some of the cross-cutting concerns that need to be addressed with AI adoption.

16 © Kong Inc. AI Governance Playbook: Innovation Meets Security for Responsible AI Adoption
 Without an AI gateway, we risk introducing complexity,
fragmentation, security blindspots, shadow IT, and overall lower
efficiency and higher costs.
In the same way that we adopt an API gateway to cater to API cross-cutting concerns, we can
adopt an AI gateway to do the same for all AI consumption.

To simplify the architecture, the AI gateway
should be managed by the platform team and
offered as an internal core service to every
application that needs to use it. By having
the platform team work in partnership with
the developers and the security team, we can
ensure that we provide an internal service
that is compliant and that developers can use
responsibly.
This approach provides us with a unified
control plane to manage all AI consumption
generated by the organization and therefore
gives us the opportunity to quickly implement
security policies, observability collection,
and developer pipelines for automating the
onboarding of different teams whenever they
need to access AI in their applications.

The AI gateway becomes a core platform service that every team can use.

An AI gateway will support multiple AI backends (e.g., OpenAI, Mistral, Llama, Anthropic) but still
provide one API interface that developers can use to access any AI model they need. We can now
manage the security credentials of any AI backend from one place so that our applications don’t
need to be updated whenever we rotate or revoke a credential to a third-party AI.

17 © Kong Inc. AI Governance Playbook: Innovation Meets Security for Responsible AI Adoption
 In addition, it can implement prompt security, validation, and template
generation so that the prompts themselves can be managed from one
control plane and changed without having to update the client applications.
Prompts are at the core of what we ask AI to do, and being able to control what prompts our
applications are allowed to generate is essential for responsible and compliant adoption of AI.
We wouldn’t want developers to build an AI integration around restricted topics (political, for
example), or to mistakenly set the wrong context in the prompts which can then be later exploited
by a malicious user.

Because different teams will most likely use
different AI services, the AI gateway can offer
a standardized interface to consume multiple
models, simplifying the implementation of AI
across different models and even switching
between one another.
The AI gateway could also implement security
overlays like AuthN/Z, rate-limiting, and full API
lifecycle governance to further manage how AI
is being accessed internally by the teams. At
the end of the day, AI traffic is API traffic.
AI observability can be managed from one
place and even be sent to third-party log/
metrics collectors. And since it’s being
configured in one place, we can easily capture
the entirety of AI traffic being generated to
further ensure data is compliant and that there
are no anomalies in the usage.
Last but not least, AI models can be expensive
to run. The AI gateway can be leveraged to
allow the organization to learn from its AI
usage to implement cost-reduction initiatives
and optimizations.

A free, open source AI gateway

An AI gateway should be an essential part of your AI adoption strategy, and the good news is
there are free options that can be implemented right away.

Kong Gateway — the world’s most adopted
open source API gateway — has been
enhanced with free, open source plugins that
turn every Kong Gateway deployment into an AI
gateway.
on top of Kong Gateway, it will be possible to
orchestrate AI flows in the cloud or on self-
hosted LLMs with the best performance and
the lowest latency, which are critical in AI-based
applications.

Kong’s AI Gateway allows developers who AI Gateway is natively supported by Kong’s

want to integrate one or more LLMs into their
products to be more productive and ship AI
capabilities faster while offering a solution to
architects and platform teams that ensures
visibility, control, and compliance on every AI
request sent by the teams. Because it’s built
cloud platform, Kong Konnect, which
provides the easiest and fastest way to get
started — and supports hundreds of official
and community Kong Gateway plugins,
like AuthN/Z, traffic control, rate-limiting,
transformations, and more.

18 © Kong Inc. AI Governance Playbook: Innovation Meets Security for Responsible AI Adoption
 With Kong’s AI Gateway, you can:
• Build multi-LLM integrations — The “ai-
proxy” plugin allows you to consume
multiple LLM implementations — in the
cloud or self-hosted — with the same API
interface. It ships with native support
for OpenAI, Azure AI, Cohere, Anthropic,
Mistral, and LLaMA. And because we
standardize how they can be used, you can
also easily swap between LLMs at the “flip
of a switch” without having to change your
application code — great for using multiple
specialized models in your applications and
for prototyping.
• Manage AI credentials centrally — With “ai-
proxy” you can also store all AI credentials,
including tokens and API keys, in Kong
Gateway without having to store them in
your applications, so you can easily update
and rotate them on the fly and in one
centralized place without having to update
your code.
• Collect L7 AI metrics — Using the “ai-proxy”
plugin you can now collect L7 analytics —
like the number of request and response
tokens or the LLM providers and models
used — into any third party like Datadog,
New Relic, or any logging plugin that Kong
Gateway already supports (such as TCP,
Syslog, and Prometheus). By doing this, you
not only simplify monitoring of AI traffic by
the applications, but you also get insights
as to what are the most common LLM
technologies that the developers are using
in the organization. The L7 AI observability
metrics are in addition to all other request
and response metrics already collected by
Kong Gateway.
• No-code AI integrations — You can
leverage the benefits of AI without having to
write any line of code in your applications
by using the “ai-request-transformer” and
19 © Kong Inc. AI Governance Playbook: Innovation Meets Security for Responsible AI Adoption
“ai-response-transformer” plugins that will
intercept every API request and response
and augment it with any AI prompt that
you have configured. For example, you
can translate on the fly an existing API
response for internationalization without
actually having to change the API or change
your client applications. You can enrich,
transform, and convert all existing API
traffic without lifting a finger, and more.
You can even enrich an AI request directed
to a specific LLM provider with another
LLM provider that will instantly update
the request before sending it to the final
destination.
• Advanced AI prompt engineering — AI
Gateway offers plugins fully focused
on advanced prompt engineering to
fundamentally simplify and improve how
you’re using AI in your applications. “ai-
prompt-template” gives an easy way to
create prompt templates that can be
managed centrally in Kong Gateway and
used on the fly by applications by only
sending the named values to use in your
templates. This way you can update your
prompts at a later time without having to
update your applications, or even enforce
a compliance process for adding a new
approved template to the system.
• Decorate your AI prompts — Most AI
prompts that you generate also set a
context for what the AI should or should
not do and specify rules for interpreting
requests and responses. Instead of setting
up the context every time, you can centrally
configure it using the “ai-prompt-decorator”
plugin that will prepend or append your
context on the fly on every AI request. This
is also useful to ensure compliance in the
organization by instructing AI to never
discuss — for example — restrictive topics
and more.
 • AI prompt firewall — This capability is
oriented toward teams and organizations
that want to ensure that prompts are
approved and that someone doesn’t
mistakenly use the wrong prompts in their
applications. With the “ai-prompt-guard”
plugin, you can set a list of rules to deny
or allow free-form prompts that are being
generated by the applications and that are
being received by Kong Gateway before
being sent to LLM providers.
• Create an AI egress with hundreds of
features — By leveraging these new AI
capabilities in Kong Gateway you can
20 © Kong Inc. AI Governance Playbook: Innovation Meets Security for Responsible AI Adoption
centralize how you manage, secure,
and observe all your AI traffic from one
place. You can also leverage official and
community Kong Gateway plugins to
further secure how your AI egress should
be accessed by other developers and add
other features for your unique use cases.
Every AI egress in Kong Gateway is a service
like any other, so all Kong Gateway features and
plugins are available out of the box. This makes
Kong’s AI Gateway the most capable in the
entire AI ecosystem. Kong Konnect’s developer
portal and service catalog are also supported.
 With Kong AI Gateway, we can address cross-cutting capabilities that all teams will need to
otherwise build themselves.

21 © Kong Inc. AI Governance Playbook: Innovation Meets Security for Responsible AI Adoption
 Conclusion
The speed at which both agile startups and colossal enterprises
are adopting AI is staggering. Some may move slowly and with
caution while others are racing headfirst into the unknown seeking
competitive advantage at nearly any cost.
AI and APIs are unlocking new possibilities Whatever your organization’s stance, if taking
and reshaping industries across the board. advantage of AI is important to your business,
While challenges will arise, the potential for then building a playbook for responsible AI
creative innovation and scientific discovery adoption is essential.
is boundless. As we navigate the evolving
landscape of AI and APIs, businesses must
adapt, harness AI’s creative potential, and use
APIs as a bridge to this exciting new world of
possibilities.

22 © Kong Inc. AI Governance Playbook: Innovation Meets Security for Responsible AI Adoption
 Konghq.com

Kong Inc.
contact@konghq.com

77 Geary Street, Suite 630

San Francisco, CA 94108
USA

23 © Kong Inc. AI Governance Playbook: Innovation Meets Security for Responsible AI Adoption