07/05/2020 Quality Assurance and Testing the ML Model - DZone AI

 

LIVE WEBINAR - Thursday, May 14th: Build Your Own CDN in Under 1 Hour
Sign Up Now 
DZone > AI Zone > Quality Assurance and Testing the ML Model

Quality Assurance and Testing the ML

Model
by Ajitesh Kumar   CORE · Aug. 09, 18 · AI Zone · Opinion

This is the irst post in the series of posts related to Quality Assurance (QA) and Testing Practices and
Data Science/Machine Learning Models, which I will release in the next few months. The goal of this
and upcoming posts is to create a tool and framework that can help you design your testing/QA
practices around data science/Machine Learning models.

QA Practices For Testing Machine Learning Models

Are you a test engineer and want to know how you can make a difference in the AI initiative being
undertaken by your current company? Are you a QA manager and looking for or researching tools and
frameworks that can help your team perform QA with Machine Learning models built by data
scientists? Are you in one of the strategic roles in your company and looking for QA practices (to
quality assure ML models built by data scientists) that you want to be adopted in your testing center of
excellence (COE) to serve your clients in a better manner?

If the answers to the above questions are yes, then keep reading. I will be presenting concepts, tools,
and frameworks that will help you achieve some of the objectives mentioned earlier.

I have seen in my experience that ML models are developed and tested by data scientists themselves.
This is not a desired situation to be in. Ideally speaking, it should be a quality assurance team that
should be performing QA by running tests as like traditional software to test the ML models from time-
to-time. However, the challenge is that ML models are not like traditional software where the behavior
of the software is pre-determined based on the different inputs. We will touch upon some of the
challenges related to testing ML models in later articles.

What Can Be Tested With ML Models?

The following are some of the aspects of a Machine Learning model that needs to be tested/quality
assured:

Quality of data
Quality of features
Quality of ML algorithms

https://dzone.com/articles/quality-assurancetesting-the-machine-learning-mode 1/3
07/05/2020 Quality Assurance and Testing the ML Model - DZone AI

Quality Assurance of Data Used for Training the Model

One of the most overlooked (or ignored) aspects of building a Machine Learning model is to check
whether the data used for training and testing the model are sanitized or if they belong to an adversary
data set. The adversary data sets are the ones that can be used to skew the results of the model by
training the model using incorrect data. This is also termed as data poisoning attacks.

The role of the QA is to put test mechanisms in place to validate whether the data used for training is
sanitized. In other words, the tests need to be performed to identify whether there are instances of data
poisoning attacks intentionally or unintentionally.

In order to achieve the above, one of the techniques could be to have QA/Test engineers work with
product management and product consultant teams for some of the following:

Understand the statistics related with data (mean, median, mode etc)
Understand the data and their relationships at a high-level
Build tests (using scripts) to check the above statistics and relationships.
Run the tests at regular intervals

The parameters listed above would need to be tracked at regular intervals and veri ied with the help of
PMs/consultants before every release. We will go into the details in later articles.

Quality Assurance of Features

Many a time, one or more features could cease to be important or become redundant/irrelevant, and,
in turn, impact the prediction error rates. This is where a set of QA/testing practices should be in place
to proactively evaluate features using feature engineering techniques such as feature selection,
dimensionality reduction, etc. We will go into the details in later articles.

Quality Assurance of ML Algorithms

Evolving datasets as a result of business evolution or data poisoning attacks could result in increased
prediction error rates. As the ML model gets retrained (manually or in an automated manner), the
increased prediction error rates result in the re-evaluation of ML models, which could result in the
discovery of new algorithms that could give improved accuracy over the previous ones.

One of the ways to go about testing ML algorithms with new data is the following:

Keep all the ML models based on different algorithms handy. Many times, I have seen that ML
d l b l d ff l h d
https://dzone.com/articles/quality-assurancetesting-the-machine-learning-mode d d d df ll f h 2/3
07/05/2020 Quality Assurance and Testing the ML Model - DZone AI
models are built using different algorithms and get discarded once and for all after the most
accurate model gets selected.
Retrain all of the models and evaluate the performance
Track the performance of all the models with new data set at regular intervals.
Raise the defect if another model starts giving greater accuracy or performing better than the
existing model.

We will go into further details in later articles.

References
Keeping Your Machine Learning Models Up-To-Date
An introduction to feature selection

Summary
In this post, you learned about the need for QA practices for Data Science/ML models and also the
different aspects of testing the ML models. Please feel free to suggest or share your thoughts in the
comments section.

Like This Article? Read More From DZone

related DZone Article related DZone Article
article QA: Blackbox Testing for Machine article Deploying Machine Learning Models in
thumbnail Learning Models thumbnail the Cloud

related DZone Article related Free DZone Refcard

article Testing Features of ML Models refcard Introduction to TensorFlow
thumbnail thumbnail

Topics: ARTIFICIAL INTELLIGENCE , DEEP LEARNING , MACHINE LEARNING , ML ALGORITHMS , ML MODEL , QUALITY ASSURANCE ,
TESTING ML MODELS

Published at DZone with permission of Ajitesh Kumar , DZone MVB. See the original article here. 
Opinions expressed by DZone contributors are their own.

https://dzone.com/articles/quality-assurancetesting-the-machine-learning-mode 3/3