Professional Documents
Culture Documents
AI - Introducing ModelOps To Operationalize AI
AI - Introducing ModelOps To Operationalize AI
This PDF is only licensed for individual use when downloaded from forrester.com or reprints.forrester.com. All other distribution prohibited.
forrester.com
For Application Development & Delivery Professionals
10 Build Or Buy Your ModelOps Muscles Shatter The Seven Myths Of Machine Learning
Recommendations
Tools, technology, and practices that enable cross-functional AI teams to efficiently deploy,
monitor, retrain, and govern AI models in production systems.
Model
lifecycle
governance
Deploy Model
© 2020 Forrester Research, Inc. Unauthorized copying or distributing is a violation of copyright law. 2
Citations@forrester.com or +1 866-367-7378
For Application Development & Delivery Professionals August 13, 2020
Introducing ModelOps To Operationalize AI
The Core Capability That Enterprises Need To Deploy, Monitor, And Govern Machine
Learning Models
Developing an AI model is just the first part; now you need to get it into production. This means
connecting it to production data, integrating it into applications, and provisioning infrastructure.
Without ModelOps this can take weeks, months, or never happen at all. For example, it took a major
US bank two weeks to develop a fraud model, two weeks to document it, and eight months to
integrate into its systems — and that was considered a success story. With this type of time lag and
redevelopment, models can perform very differently than during development, which reduces their
performance and leads to additional rework. ModelOps deployment capabilities help enterprises:
›› Orchestrate deployment and serving. ModelOps helps teams orchestrate the deployment of the
model. This includes provisioning of infrastructure, staging the model, managing dependencies,
orchestrating the multiple steps that occur when a model is called, and serving the model in a
robust, scalable, and high-availability fashion. For example, a major pharmaceutical company
leveraged ModelOps pipelines to train a model that predicts pulmonary diseases in patients,
evaluates it, deploys it on a public cloud, and periodically monitors and retrains it.4
›› Deploy models from anywhere . . . Enterprises usually use multiple predictive analytics and
machine learning (PAML) tools based on the skills of their data science teams and to take
advantage of the latest innovations in AI. Hence, they need to support the deployment of models
developed using multiple tools. This includes open source ML programming languages and
frameworks like Python and R as well as commercial ML tools so enterprises can take advantage
of all best-of-breed solutions.
›› . . . to anywhere. Organizations must develop the ability to deploy models across an ever-growing
range of on-premises, private cloud, public cloud, hybrid cloud environments, and specialized
AI hardware. Not only does this help them meet the data and application requirements of their
AI use cases, but it also helps them take advantage of the cost, scalability, latency, and security
tradeoffs of each environment. For example, Datactics deploys models across a host of different
cloud environments, depending on what its customers use, and with a combination of open source
technologies, including Data Version Control for versioning, Airflow for orchestration, and MLflow to
manage the model lifecycle.
© 2020 Forrester Research, Inc. Unauthorized copying or distributing is a violation of copyright law. 3
Citations@forrester.com or +1 866-367-7378
For Application Development & Delivery Professionals August 13, 2020
Introducing ModelOps To Operationalize AI
The Core Capability That Enterprises Need To Deploy, Monitor, And Govern Machine
Learning Models
Models don’t get better on their own; they usually get worse as the world changes and the data drifts
further and further from the data the model was trained on (see Figure 2). ModelOps helps you monitor
important production model metrics and alerts you when your models may need to be retrained. It also
accelerates your ability to retrain your models by automatically capturing and selecting the new training
data. An online service provider found that without ModelOps, it took 21 days to detect that a model
was underperforming and an additional four to five weeks to assemble the necessary data to retrain it.5
Enterprises must use ModelOps to monitor their models for:
›› Data drift. The world is always changing, and the data feeding your models changes with
it, whether that is due to user behavior or your changing internal systems. By monitoring the
distribution of data feeding a model and detecting when it changes, you get early warning that
your model may not be getting the right data or may no longer be up to date (see Figure 3). An
eCommerce firm shortened the time it took to identify performance issues in its fraud detection
models from one to three months to under a day by monitoring its models for data drift and
assessing the impact, instead of waiting to see a spike in fraudulent transactions.6
›› Prediction distribution drift. Maybe your model can handle the fluctuations in the input data (e.g.,
because that data isn’t very important for the model), or maybe it can’t. By tracking your models’
predictions over time, ModelOps helps you detect when your models are making substantially
different predictions compared to normal, which can be a sign that your model needs to be
retrained or your pipeline isn’t working correctly. For example, a sudden spike in the number of
mortgage applications that your model thinks are low risk could indicate that your efforts to market
to low-risk customers are paying off, but it’s more likely a sign that your model needs retraining. By
detecting the spike, ModelOps lets you know when you should investigate.
›› Concept or business KPI drift. The ultimate proof of whether your model needs to be retrained is
when your model starts making incorrect predictions that affect your important business outcomes.
ModelOps helps you close the loop, enabling you to compare what actually happened to the
model’s prediction and its impact on business KPIs. Better still, it helps you identify the drivers
causing that incorrect prediction (see Figure 4).
›› Explainability and fairness. Enterprises can use ModelOps to provide explanations for model
predictions, using techniques like SHAP or surrogate models, or to monitor models for fairness.7
However, not all enterprise use cases will need prediction-level explanations or monitoring for
ethical considerations, and other tools can also meet these needs (see Figure 5).
© 2020 Forrester Research, Inc. Unauthorized copying or distributing is a violation of copyright law. 4
Citations@forrester.com or +1 866-367-7378
For Application Development & Delivery Professionals August 13, 2020
Introducing ModelOps To Operationalize AI
The Core Capability That Enterprises Need To Deploy, Monitor, And Govern Machine
Learning Models
FIGURE 2 Enterprises Must Monitor, Retrain, And Redevelop Machine Learning Models Over Time
Business
value
Maximum
business value
Time deployed
Retrain or
redevelop
© 2020 Forrester Research, Inc. Unauthorized copying or distributing is a violation of copyright law. 5
Citations@forrester.com or +1 866-367-7378
For Application Development & Delivery Professionals August 13, 2020
Introducing ModelOps To Operationalize AI
The Core Capability That Enterprises Need To Deploy, Monitor, And Govern Machine
Learning Models
© 2020 Forrester Research, Inc. Unauthorized copying or distributing is a violation of copyright law. 6
Citations@forrester.com or +1 866-367-7378
For Application Development & Delivery Professionals August 13, 2020
Introducing ModelOps To Operationalize AI
The Core Capability That Enterprises Need To Deploy, Monitor, And Govern Machine
Learning Models
Relevant metrics
correlated with
accuracy drop
Source: superwise.ai
© 2020 Forrester Research, Inc. Unauthorized copying or distributing is a violation of copyright law. 7
Citations@forrester.com or +1 866-367-7378
For Application Development & Delivery Professionals August 13, 2020
Introducing ModelOps To Operationalize AI
The Core Capability That Enterprises Need To Deploy, Monitor, And Govern Machine
Learning Models
AI is ultimately a type of software. As such, to be governed it must have versioning, lifecycle lineage,
and security capabilities — so that any team member can access what others have done, replicate
it, validate it, and build on it. Tools accessible by and designed for broader AI teams of business,
AD&D, IT ops, and data science enable collaboration. Most importantly, enterprises need ModelOps
governance capabilities that:
›› Track metadata, lineage, provenance, data, and other dependencies of models. Step 1 in
governing the AI lifecycle and driving collaboration among teams is tracking and documenting
what was done using what. These ModelOps capabilities help you reproduce predictions, fix faults,
accelerate your ability to both retrain models and reuse pipelines, and provide an audit trail for
regulators (see Figure 6).
›› Orchestrate workflows. Data and ML model pipelines are often complicated, involving numerous
data sources, transformations, and multiple models. The workflows to monitor, notify, and update
these pipelines as well as approve changes are similarly complex to manage. Enterprises need the
capabilities to define, manage, and update these workflows to coordinate the disparate teams. For
© 2020 Forrester Research, Inc. Unauthorized copying or distributing is a violation of copyright law. 8
Citations@forrester.com or +1 866-367-7378
For Application Development & Delivery Professionals August 13, 2020
Introducing ModelOps To Operationalize AI
The Core Capability That Enterprises Need To Deploy, Monitor, And Govern Machine
Learning Models
example, some ModelOps solutions can implement workflows that detect data drift, notify a data
engineer to investigate, determine whether a data scientist needs to retrain that model, and solicit
approval from a business user to deploy the updated model (see Figure 7).
›› Provide security and compliance. Organizations need to ensure that everyone is playing the right
role and working together effectively. That means having capabilities to determine access and
permissions for data, infrastructure, tools, model artifacts, and project repositories as well as to
both define and enforce policies for usage, sharing, data protection, and so on. For example, the
United Nations used a ModelOps platform to build a ML model repository and provide access to its
data scientists and member states globally.8
Source: Cloudera
© 2020 Forrester Research, Inc. Unauthorized copying or distributing is a violation of copyright law. 9
Citations@forrester.com or +1 866-367-7378
For Application Development & Delivery Professionals August 13, 2020
Introducing ModelOps To Operationalize AI
The Core Capability That Enterprises Need To Deploy, Monitor, And Govern Machine
Learning Models
Source: ModelOp
›› Existing predictive analytics and machine learning solutions. Odds are the PAML solutions that
your data science teams use today to develop AI models have basic ModelOps capabilities that
you can leverage immediately for the price of an additional license. Further, many PAML solutions
are rapidly expanding their ModelOps capabilities to encompass more must-have capabilities.
While most focus on deploying and managing models developed with their solution, they
increasingly support a wider array of models developed with other tools, usually the most popular
open source tools. They may not provide a full solution, in terms of supporting all your AI models or
providing all the capabilities you need — but you’ll be able to get ModelOps capabilities rapidly.9
© 2020 Forrester Research, Inc. Unauthorized copying or distributing is a violation of copyright law. 10
Citations@forrester.com or +1 866-367-7378
For Application Development & Delivery Professionals August 13, 2020
Introducing ModelOps To Operationalize AI
The Core Capability That Enterprises Need To Deploy, Monitor, And Govern Machine
Learning Models
›› Building it yourself. Enterprises with the most mature AI use cases have had to build their own
solutions, and most continue to do so today. Using an ecosystem of open source components —
often Jupyter Notebooks, Kubernetes, Data Version Control (DVC), and a few open source machine
learning frameworks — you can build your own ModelOps solutions too. For enterprises that
have already invested in homegrown solutions or that have specialized needs, this may be your
best option . . . for now. Most enterprises, however, will want to forgo the time, effort, and cost of
building and maintaining their own ModelOps platforms.
Recommendations
›› Do it now. Given the rapid developments in ModelOps offerings, many enterprises will be tempted
to wait until the offerings have matured and the market has stabilized. That is not an option.
To unlock the value of their AI initiatives today, all organizations need to rapidly improve their
ModelOps capabilities using the options that are available to them now.
›› Learn lessons from your DevOps team. Most enterprises have a DevOps functions that has
learned, perhaps the hard way, the machinations necessary to deploy software to production
environments. ModelOps is not DevOps, but there is much overlap with the working collaborations
to make ModelOps a success.
›› Put someone in charge. ModelOps is underdeveloped in most organizations because it has fallen
through the cracks, with no team having the knowledge, responsibility, or authority to develop it
into a cross-functional capability. Unless you empower someone and give them the resources and
mandate, expect slow progress. Look to leaders in DevOps, operations, and/or integrated data
science teams that have a proven track record of putting assets into production and have them
build a shared ModelOps capability for the organization.
© 2020 Forrester Research, Inc. Unauthorized copying or distributing is a violation of copyright law. 11
Citations@forrester.com or +1 866-367-7378
For Application Development & Delivery Professionals August 13, 2020
Introducing ModelOps To Operationalize AI
The Core Capability That Enterprises Need To Deploy, Monitor, And Govern Machine
Learning Models
›› Formalize and routinize your existing ModelOps processes. They might not be ideal, but if
you’ve put models into production, then you have ModelOps processes that you can incrementally
improve. Document them, look for ways to improve them quickly, and make them more widely
available so your organization can start taking advantage of AI use cases faster.
›› Evaluate the offerings regularly. If you did an assessment of ModelOps offerings six months ago,
that assessment is out of date, as nearly every solution on the market has developed significantly
since then. Assess the market’s current offerings, leverage the ones that suit your needs now, and
look again in six months to a year.
›› Have a plan if you have to build. Some enterprises will find they need to build their own
ModelOps solutions in-house; do this as a last resort after you have assessed the costs and
benefits of doing so relative to the solutions on the market. And when you do build your own
ModelOps, plan for how you’ll develop this solution to meet the needs of the broader enterprise.
Beware the temptation to let individual teams explore building their own platforms. It may look
cheap and quick in the short run, but it usually magnifies the pain later on (in terms of difficulty to
maintain and scale).
© 2020 Forrester Research, Inc. Unauthorized copying or distributing is a violation of copyright law. 12
Citations@forrester.com or +1 866-367-7378
For Application Development & Delivery Professionals August 13, 2020
Introducing ModelOps To Operationalize AI
The Core Capability That Enterprises Need To Deploy, Monitor, And Govern Machine
Learning Models
To help you put research Translate research into Join our online sessions
into practice, connect action by working with on the latest research
with an analyst to discuss an analyst on a specific affecting your business.
your questions in a engagement in the form Each call includes analyst
30-minute phone session of custom strategy Q&A and slides and is
— or opt for a response sessions, workshops, available on-demand.
via email. or speeches.
Learn more.
Learn more. Learn more.
Endnotes
Source: Forrester client inquiry.
1
fn/5NVraa31eBopO5IqYqUvx7).
Technically, machine learning ops (or MLOps) refers only to the machine learning model lifecycle, while ModelOps also
3
includes other types of models, such as decisioning models and optimization models. However, the two terms are
frequently used interchangeably.
Source: Phone interview with Ofer Razon, cofounder and CEO of superwise.ai, June 2, 2020.
5
Source: Phone interview with Ofer Razon, cofounder and CEO of superwise.ai, June 2, 2020.
6
SHAP: SHapley Additive exPlanations. Source: Scott M. Lundberg and Su-In Lee, “A Unified Approach to Interpreting
7
© 2020 Forrester Research, Inc. Unauthorized copying or distributing is a violation of copyright law. 13
Citations@forrester.com or +1 866-367-7378
For Application Development & Delivery Professionals August 13, 2020
Introducing ModelOps To Operationalize AI
The Core Capability That Enterprises Need To Deploy, Monitor, And Govern Machine
Learning Models
Source: “The United Nations Global Working Group brings machine learning to the world using
8
For evaluations of the ModelOps capabilities of the most important PAML vendors, see the Forrester report “The
9
Forrester Wave™: Notebook-Based Predictive Analytics And Machine Learning Solutions, Q3 2018” and see the
Forrester report “The Forrester Wave™: Multimodal Predictive Analytics And Machine Learning Solutions, Q3 2018.”
© 2020 Forrester Research, Inc. Unauthorized copying or distributing is a violation of copyright law. 14
Citations@forrester.com or +1 866-367-7378
forrester.com
Client support
For information on hard-copy or electronic reprints, please contact Client Support at
+1 866-367-7378, +1 617-613-5730, or clientsupport@forrester.com. We offer quantity
discounts and special pricing for academic and nonprofit institutions.
160698