KPI

Lab

Data Science for Supply Chain Forecast by Nicholas Vandeput

KPI’s to Consider
• Bias (mean error)

• MAE (mean absolute error)

• RMSE (root mean square error)

• MAPE (mean absolute percentage error)

Data Science for Supply Chain Forecast by Nicholas Vandeput
KPI
• Compared to MAE, RMSE does not treat each error the same.
• It gives more importance to the biggest errors.
• That means that one big error is enough to get a very bad RMSE.
• Let's imagine we want to compare two slightly different forecasts.

• The only difference is the forecast on the latest demand observation: forecast #1
undershot it by 7 units and forecast #2 by only 6 units.
• Calculate the KPI: MAE and RMSE for both the forecasts.

Data Science for Supply Chain Forecast by Nicholas Vandeput

 Which KPI to Choose?
• Lets do another problem to understand this.
• Take a product with a rather flat weekly demand that has from time to time
a big order.

• Lets look at 3 different forecasts for this product.

• The first one predicts 2 pieces/day,
• The second one 4
• The last one 6.
• Plot the demand we observed and these forecasts.
• Calculate bias, MAPE, MAE, and RMSE on the historical period for all the 3
forecasts.
Data Science for Supply Chain Forecast by Nicholas Vandeput
MAE or RMSE - Which one to choose?

• Is it worse to aim for the median or the average of the demand?

• The answer is not black and white.
• Each technique has some benefits and some risks.
• Only experimentation will reveal which technique works best for a
dataset.
• You can even choose to use both MAE and RMSE.
• Let’s look at the impact of choosing either RMSE or MAE on the bias,
the sensitivity to outliers, and the intermittent demand.

Data Science for Supply Chain Forecast by Nicholas Vandeput

 Bias
• For most of the products, the median is not the same as
the average/mean demand.
• The demand will have some peaks here and there
resulting in a skewed distribution.
• These skewed demand distributions are very common in
supply chain
• The peaks can be due to periodic promotions
• Clients ordering in bulk.
• This will cause the demand median to be below the
average demand
• This means that a forecast that is minimizing MAE will
result in a bias.
• On the other hand, the forecast that is minimizing RMSE
will not result in a bias (as it aims for the average).
• This is MAE's main weakness.
Data Science for Supply Chain Forecast by Nicholas Vandeput
 Sensitivity to Outliers
• Let’s look at another example
• Calculate median and average for both the datasets
• The only difference between the 2 datasets is that
there is a sudden increase in the demand in period
10 (may be an outlier)
• Which KPI? (MAE or RMSE)
• Would they be different for both the dataset?
• Observations
• The median is more robust to outliers than the average.
• In a supply chain environment, this is rather important as
we can have many outliers due to encoding mistakes or
demand peaks (marketing, promotions, spot deals).
Data Science for Supply Chain Forecast by Nicholas Vandeput
Intermittent Demand
• We sell a product to a single client. It
is a highly profitable product and our
unique client seems to make an order
one week out of three without any
kind of pattern. The client always
orders the product by batches of 100.
• Create a weekly forecast for this
product
• #Forecast 1 will be the average demand
• #Forecast 2 will be the demand median
• Calculate MAE and RMSE in both the
situations

Data Science for Supply Chain Forecast by Nicholas Vandeput

Conclusion
• MAE provides a protection against outliers whereas RMSE provides
the assurance to get an unbiased forecast.
• Which indicator should you use?
• There is unfortunately no definitive answer.
• As a supply chain data scientist, you should experiment:
• if using MAE as a KPI results in a high bias, you might want to use RMSE.
• If the dataset contains many outliers, resulting in a skewed forecast, you
might want to use MAE.

Data Science for Supply Chain Forecast by Nicholas Vandeput

Problems
• Demand problem
• Assume we have demand data upto 80 periods
• We use moving average of 10 periods to calculate our forecast for
period 81
• Once we have forecast for period 81, demand for 81 occurs
• We then forecast for period 82, and demand for period 82 occurs and
so on
• Calculate MAE, RMSE, and MAPE
• Do this with moving average of 20 periods
Data Science for Supply Chain Forecast by Nicholas Vandeput