EBOOK

Resilient Machine Learning

with MLOps
Observe, Diagnose, and Mitigate Issues with Production Models

The Real World Demands Agility
Today’s economy is under pressure from inflation, rising
interest rates, and disruptions in the global supply chain.
As a result, many organizations are seeking new ways to
overcome challenges — to be agile and rapidly respond to
constant change. The same challenges are supposed to
be addressed by production-grade AI systems, which are
built on data influenced by all of these real-world changes
in market and consumer behavior. But AI systems don’t
always reflect that situation, since training data deviates
from real-time data more and more. For this reason, AI
models deployed to solve existing business problems
are extremely vulnerable.
We do not know what the future holds. But we can take
the right actions to prevent failure and ensure that AI
systems perform to predictably high standards, meet
business needs, unlock additional resources for financial
sustainability, and reflect the real patterns observed in
the outside world.
To prevent deployment delays and deliver resilient,
accountable, and trusted AI systems, many organizations
invest in MLOps to monitor and manage models while
ensuring appropriate governance. According to BARC,
organizations that adopt such tools are better able to
plan their projects, as they are 4.2 times more likely to
S ource: BARC, “Driving Innovation with AI: Getting Ahead with DataOps and MLOps”
1
Resilient Machine Learning with MLOps
be able to deploy their models quickly and 3.5 times less
likely to be confronted with projects with overwhelming
complexity. However, untangling the complexities of a
machine learning lifecycle is just one piece of the puzzle.
Organizations need full visibility and automation to rapidly
correct course in their businesses and to reflect daily
changes. This task becomes increasingly hard without a
robust MLOps framework or a solution in place. In light
of this, there are many questions that data science teams
are asked to answer.
• Deployed models were accurate yesterday,
but what about today?
• How long will it take to replace a model?
• Is there a way to get a better model into
production faster?
• Is there a way to prove the value of AI to
business stakeholders?
These questions and many others are now at the top of
the agenda of every data science team. That is even more
true for organizations who are actively expanding their
production deployments.
1
 Resilient Machine Learning with MLOps

REPORT The majority of AI-enabled organizations are still

IDC MarketScape: Worldwide
Machine Learning Operations
Platforms 2022
Vendor assessment, capabilities review,
vendor strategy insights.
Get the Report
struggling to stay atop the ever-expanding repository
of production models. This poses a critical challenge
as these models continuously influence key business
decisions, such as loan provisioning in financial
services, inventory forecasting in retail, or
staffing optimization in healthcare.
A myriad of issues can interfere with the performance
and delivery of production models, resulting in poor
or incomplete predictions and ill-informed decision-
making. This is due to lack of holistic visibility into model
operations. It’s not enough to simply expose an error; it’s
essential that teams can instantly pinpoint the context of
the error, thereby enabling quicker resolution.
Fortunately, DataRobot MLOps allows data science teams
to address these and many other challenges associated
with the ML lifecycle, delivering granular model-level
insights, observability of production models, and higher
level of confidence for decisions informed by models.
This ebook highlights several features of DataRobot
MLOps that enable higher trust and observability for your
production models.
IDC2 predicts that by 2024,
60% of enterprises would
have operationalized their ML
workflows by using MLOps.
Observability Is Key
Many data scientists lack visibility into the deployment
behavior and performance of models that are in
production. Tracking the integrity of a machine learning
model in production is critical.
Model observability features within the DataRobot AI
platform help ensure that you know when something goes
wrong and understand why it went wrong. By tracking
service, drift, prediction data, training data, and custom
metrics, you can keep your models and predictions
relevant in a fast-changing world.

2
 ource: “IDC FutureScape: Worldwide Artificial Intelligence and Automation 2022 Predictions,” By: Ritu Jyoti, David Schubmehl, Hayley Sutherland,
S
Peter Haas, Sriram Subramanian, Maureen Fleming, Shari Lava, Dr. Chris Marshall, Jennifer Hamel, Neil Ward-Dutton, Jack Vernon, Mary Wardley,
2
Warren Shiau, Jennifer Cooke, Pietro Delai, Harish Dunakhe, Tim Grieser, Peter Rutten, Shane Rau, October 2021, IDC#US48298421
 EBOOK
DataRobot MLOps Agents:
Visibility For All Your Production Models
Deploy, manage, monitor, and govern machine learning
models from a single place, regardless of how they
were created or where they are located.
Get the Ebook
DataRobot Model Observability shows what went
wrong with models and why it went wrong.
Model Observability Is More Than Just Monitoring
Model observability provides an end-to-end picture
of the internal states of a system, such as its inputs,
outputs, and environment, including data drift, prediction
performance, service health, and more relevant metrics.
This means you have the ability to not only monitor the
problem but also analyze and pinpoint its source. Model
observability compounds performance stats and metrics
across the entire model lifecycle to provide context to
problems that can threaten the integrity of your models.
Holistic control over ML models is key to sustaining a
high-yield AI environment.
DataRobot MLOps provides world-class governance and
scalability for model deployment. Models across the
organization, regardless of where they were built, can
be supervised and managed under one single platform.
In addition to DataRobot models, open source models
deployed outside of DataRobot MLOps can also be
managed and monitored by the DataRobot platform.
It is not enough to just monitor performance and log
errors. To get a complete understanding of the internal
state of your AI/ML system, you also need visibility into
prediction requests and the ability to slice and dice
prediction data over time. Not knowing the context of a
performance issue delays the resolution, as the user will
have to diagnose via trial and error, which is problematic
for business critical models.
This is a key difference between model monitoring and
model observability: model monitoring exposes what the
problem is; model observability helps understand why the
problem occurred. Both must go hand in hand.
With model observability capabilities, DataRobot MLOps
users gain full visibility and the ability to track information
regarding service, drift, prediction and training data, as
well as custom metrics that are relevant to your business.
DataRobot customers now have enhanced visibility into
hundreds of models across the organization.
Resilient Machine Learning with MLOps 3

Model Observability with Custom Metrics
To quantify how well your models are doing, DataRobot
provides you with a comprehensive set of data science
metrics — from the standards (Log Loss, RMSE) to the
more specific (SMAPE, Tweedie Deviance). But many of
the things you need to measure for your business are
Challenger Insights for Multiclass and External Models
After DataRobot has delivered an optimal model,
Production Lifecycle Management capabilities of the
platform help ensure that the currently deployed model
will always be the best one even as the world changes
around it. MLOps delivers automated strategies to keep
production models at peak performance, regardless of
external conditions.
For example, DataRobot Data Drift and Accuracy
Monitoring detects when reality differs from the situation
when the training dataset was created and the model
trained. Meanwhile, DataRobot can continuously train
challenger models based on more up-to-date data.
Resilient Machine Learning with MLOps
hyper specific for your unique problems and opportunities
— specific business KPIs or data science secrets. With
DataRobot Custom Metrics, you can monitor details
specific to your business.
Once a challenger is detected to outperform the current
champion model, the DataRobot AI platform notifies you
about changing to this new candidate model.
Business processes probably require you to verify this
suggestion. Is this automatically created model actually
better than the current champion—and reliably so? To
facilitate this decision, DataRobot platform provides
challenger Insights, a deep but intuitive analysis of how
well the challenger performs and how it stacks up against
the champion. This also shows how the models compare
on standard performance metrics and informative
visualizations like Dual Lift.
4
 Resilient Machine Learning with MLOps

Manage changing market conditions. With the

DataRobot AI platform, you can see predicted values and
accuracy for various metrics for the champion, as well as
any challenger models.

On top of that, DataRobot MLOps offers Challenger

Insights for External Models, which allows you to
leverage DataRobot MLOps to monitor already existing
and deployed models, while DataRobot will construct
challengers for those models in the background. If a
DataRobot challenger model manages to beat the
external model, Challenger Insights capability allows
you to carefully compare your own models against the
candidate produced by the DataRobot AI platform.

Compare performance of multiple models 5

 Resilient Machine Learning with MLOps

Clearly know when your challenger beats your

champion. DataRobot Challenger Insights includes
a rich set of performance metrics, from standards
such as Log Loss and RMSE to the more specialized
metrics DataRobot uses for specific problems. Here the
DataRobot view shows that the challenger beats the
Challenger and champion comparison insights
champion on some metrics, but not all.

DataRobot offers more in-depth analysis in Challenger

Insights, including Dual Lift, ROC and Prediction Differences.
In the above illustrated case, DataRobot shows that the
challenger automatically retrained via AutoML handily beats
the champion on key metrics.

Champion and challenger ROC comparison

 Resilient Machine Learning with MLOps

Visualize Data Drift Over Time

to Maintain Model Integrity
Data drift is a key performance metric that data scientists
should track in order to maintain the high quality results
they expect from a model. Data drift occurs when input
data changes over time and becomes significantly
different from the data that was used during training
and validation stages of model development. When this
type of drift occurs, your model is at risk of degradation,
meaning you cannot trust the predictions anymore.

In addition to being alerted when data drift has occurred,

you need to understand how the drift score has changed
in order to get a deeper understanding of the cause and
impact of this drift.

Data drift can occur for a variety of reasons, including

seasonality, change in prediction values, or even different
volumes of predictions. The corrective action you take will
depend on the cause and context of the drift. Therefore,
you need to fully understand why and how drift occurred,
which is the ultimate goal of observability.

7
 REPORT

The Business Value of MLOps

Discover the most impactful benefits

of MLOps tools and processes for
different types of organizations.

Get the Report

Drift over time

DataRobot MLOps Offers User-Friendly Visuals to Track Data Drift Over Time

The example above shows drift (y axis) over time of
prediction (x-axis) allowing you to easily track trends. The
gray dotted line is the acceptable threshold for drift. You
can easily scan which predictions surpass this threshold
and at what time. Additionally, the gray bars at the bottom
of the chart showcase the volume of predictions so that
you can understand how many predictions were impacted
by drift. Users can slice and dice drift information by
choosing different features to investigate drift.
With the interactive ability to compound this information,
you can understand why drift is happening and quickly
take appropriate action before it impacts the business.

Resilient Machine Learning with MLOps 8

Drill Down into Drift for Rapid Model Diagnostics
A key challenge in investigating data drift is the lack of
details available to the user. Traditionally, drift is tracked
for top features by comparing scoring data to training data.
Drift can also be viewed over time to identify general drift
trends. To dive deeper into the patterns and causes of drift,
MLOps users need to be able to compare drift between
two scoring data segments (in addition to the traditional
comparison between scoring and training data), for any or
all features, and across any specified time period.
More and more organizations see the need for ad hoc
drift deep dive, especially as world economic conditions
continue to impact models at an alarming rate. Patterns
in data are changing faster than data science teams can
keep up, costing them time and visibility into deployments.
Drift drill down solves this data science challenge so that
organizations can maintain AI driven business results.
DataRobot MLOps users can compare drift of selected
features between two scoring segments of a model (or
scoring and training segments), for any time period, and
view contextual information such as prediction value over
time to further support their investigation.
Resilient Machine Learning with MLOps 9
 Resilient Machine Learning with MLOps

As highlighted in the DataRobot interface above, the Data

Drift tab is enhanced with a Drill Down section for users
to visualize drift details. Users can configure their own
display settings to select a model, date range of interest,
and time granularity. This is important as data drift can
look different at different time granularities; drift can
happen at any time and at any rate.

For example, if a model has been in production for a

year with little drift, but has only begun drifting at an
increasing rate in the last week, the overall drift view may
not represent this imminent problem. Zooming into that
last week will help the user understand how quickly data is
drifting and whether or not it’s a cause for concern.

Interactive drift drill down

10
“You might think that overall, the model’s features drifted
relatively little in production, but in reality, the model’s
drift statistics might be fluctuating quite a bit up and
down. Or there might be a concerning trend beginning to
develop over the past week that you want to keep an eye
on. That insight requires looking at specific time slices.
Granular time splits show you the true picture,”
Brian Bell
Senior Director, Product Management, DataRobot MLOps Strategy Lead
Without the ability to zoom into granular time slices,
differences in drift patterns may get lost in the overall
analysis. DataRobot drift drill down capability allows
data scientists to run quick sanity checks, investigate
accelerating or decelerating patterns in drift, and control
the level of granularity of the visuals.
The DataRobot AI platform offers fast and intuitive drift
drill down, as we focus on analyzing your data across
different dimensions in real time to answer data science
questions. You can change the parameters of analysis and
get multiple insights quickly from the user interface.
Resilient Machine Learning with MLOps 11
 Resilient Machine Learning with MLOps

Embrace Large Scale

with Confidence
While model observability and deeper understanding of
your model insights are crucial for longevity of AI projects,
it’s important to remember that scale is where the value of
machine learning models is realized. For example, if you
have a model that predicts warehouse capacity for one
store, what about global capacity? What if we can add
more segments and conditions to these parameters? Does
your system handle billions of predictions and ensure
that your models are trustworthy and data is secured?
Organizations that see more value from AI want to apply it
to more use cases, which creates a scalability challenge.

Are you making millions of predictions daily or hourly?

Do you need to ensure that you have a top-performing
model in production without sharing sensitive data? ​​You
can aggregate prediction statistics much faster while
controlling the governance and security of your sensitive
data — no need to submit their entire prediction requests
to DataRobot AI Cloud platform to get data about drift and
accuracy monitoring.

DataRobot Large Scale Monitoring allows you to

access aggregated prediction statistics. This feature
will compute some DataRobot monitoring calculations
outside of DataRobot and send the summary metadata
to DataRobot MLOps. It will let you independently control
the scale and handle billions of rows per day. These
capabilities, combined with the incredible granularity
and observability enabled by MLOps, allow organizations
to observe, diagnose, and mitigate any issues with
production models, no matter the scale.

12
 Resilient Machine Learning with MLOps

Going Forward
By implementing DataRobot MLOps, organizations can
quickly and efficiently stand up a center of excellence
(CoE) for machine learning operations and production AI.

MLOps allows organizations to:

• Put ML into production quickly through

repeatable, scalable processes.

• Observe, diagnose, and mitigate issues with

production models on an enterprise scale.

• Optimize outcomes, reduce costs, and maximize

the value of their production models.

With DataRobot MLOps, organizations can realize

significant ROI by spending less time and resources
on infrastructure, model deployment, and model
management, while ensuring resilience and observability
for their AI projects throughout their whole lifecycle.

To learn more about DataRobot’s MLOps capabilities visit

datarobot.com/platform/mlops/.

To schedule a demo, visit

datarobot.com/lp/request-demo/.

13
DataRobot is the leader in Value-Driven AI – a unique and collaborative approach to AI that
combines our open AI platform, deep AI expertise and broad use-case implementation to
improve how customers run, grow and optimize their business. The DataRobot AI Platform
is the only complete AI lifecycle platform that interoperates with your existing investments
in data, applications and business processes, and can be deployed on prem or in any
cloud environment. DataRobot and our partners have a decade of world-class AI expertise
collaborating with AI teams (data scientists, business and IT), removing common blockers
and developing best practices to successfully navigate projects that result in faster time
to value, increased revenue and reduced costs. DataRobot customers include 40% of the
Fortune 50, 8 of top 10 US banks, 7 of the top 10 pharmaceutical companies, 7 of the top
10 telcos, 5 of top 10 global manufacturers.

Learn more at datarobot.com.

© 2023 DataRobot, Inc. All rights reserved. DataRobot and the DataRobot logo are trademarks of DataRobot, Inc.
All other marks are trademarks or registered trademarks of their respective holders.