Case Study: Using machine learning to forecast demand despite limited data

Supply Chain Optimization – Pharma Distributor

Part 1: Demand Forecasting
This is the first part of a supply chain optimization project. This first part focuses on demand
forecasting.

Executive Summary
In this two-part proof-of-concept, SupChains and DragonRitter (a forecasting-as-a-service
platform) helped a Pharma Distributor to reduce its inventory while optimizing its service
level. The POC showed a forecast error reduction of 25% and an expected inventory
excess reduction of 40% within six months while securing higher service levels (>95%
compared to the current 80%). In addition, the model provided the client with a list of dead
stock so they could take action immediately.
Based on these preliminary results (presented in November 2022), the client decided to use
our forecast engine as of December 2022 and plan its purchases based on the model as of
January 2023.

Business Situation
Our client is a pharma distributor active in Latin America with a dozen active warehouses and
around 10,000 unique products. In 2021, they reported a total revenue of 1B$, employing more
than a thousand employees.
As many supply chains worldwide, they suffer from both dead stocks and shortages: despite
stocking the equivalent of 45 days of sales, they only achieved a 60% fill rate in 2022 (up to
80% in the warehouse we used in the POC). They solicited the help of DragonRitter (a
forecasting-as-a-service platform) and SupChains to propose a joint end-to-end solution.
Our client’s objective is to increase their service levels while reducing their total inventory.

Our End-to-End Solution: Demand Forecasting and Inventory Optimization

To achieve this double objective of inventory reduction and service level increase, SupChains
recommended a 2-step approach:
1. Improve the forecasting quality by implementing a machine learning forecasting
engine.
2. Compute adequate inventory levels capturing both demand and supply variability to
achieve the desired service level targets.
As our client didn’t have a proper forecast engine, we started the project by developing a
custom forecasting engine leveraging our machine learning expertise.

Forecast Challenges
To support its supply chain decisions, the distributor needs an accurate forecast per
warehouse, product, and week for the upcoming 26 weeks. This represents around 100,000
combinations. (The POC focused on a single warehouse.)
We have access to historical sales (per client channel) – which is the main requirement. But
the information fed into the model could be enriched by promotions, backorders, shortages,

nicolas.vandeput@supchains.com Confidential
Case Study: Using machine learning to forecast demand despite limited data

product information, prices, etc. Unfortunately, this project dataset presented two main
difficulties: erratic demand behavior and a lack of rich data. Let’s discuss these two in
detail.

Erratic Demand Behaviour

This dataset is especially skewed, with an average weekly sales of 30 units but a median of 0
units (see the figure below).

Figure 1 More than 50% of observed periods have zero sales.

Moreover, 99% of the sales observations are below 400 units, but the remaining 1% go up to
80,000 units sold in a single week.

Figure 2 99% of sales observations are below 400 units, but some products sold for up to 80,000 units in a single week.

To illustrate this erratic demand, you can see the anonymized demand pattern of one of their
best sellers in the figure below. This pattern is wildly irregular: the average order numbers
are four times higher than the median – and the demand coefficient of variation is around
150% (whereas, usually, top sellers enjoy more stable demand patterns).

nicolas.vandeput@supchains.com Confidential
Case Study: Using machine learning to forecast demand despite limited data

Figure 3 Even the top sellers display erratic demand patterns (anonymized data).

Lack of Rich Data

Machine learning models are especially suited to deal with rich demand datasets (including
demand drivers, such as promotions and pricing; and extra information, such as historical
shortages). Unfortunately, as detailed in the following paragraphs, we didn’t have access to
this additional information for this POC.

Limited Historical Data

The client’s ERP system only allows the conservation of historical sales for up to 3.5 years.
This limitation is, unfortunately, only too common in supply chains and a significant
drawback to achieving the best accuracy possible.1

No Demand Drivers
Machine learning models are especially suited to gather insights from various demand
drivers (pricing, promotions, shortages) to generate accurate predictions.2
For this specific case, we could only gather historical prices.

No Product Transitions (Life-Cycle)

Unfortunately, in this initial POC, we didn’t have the chance to access product transitions
data. As displayed in the figure below, flagging product transitions is a crucial best practice
that will always result in higher forecasting accuracy and more adequate stock targets for
new products.

1
We greatly encourage all supply chain practitioners to keep as much historical data as possible and
store it in a clean, consistent way. Applying data management best practices will show a return on
investment in forecasting accuracy, as shown in our previous case studies (here and here).
2
To see how our models deal with demand drivers such as promotions and pricing, see our previous
case studies here (manufacturer with promotions only) and here (retailer with promotions and
pricing).

nicolas.vandeput@supchains.com Confidential
Case Study: Using machine learning to forecast demand despite limited data

Figure 4 - Combination of the old and new versions of a similar product. The combined data will be used to forecast future
sales (data: chemical manufacturer).

No Data about Shortages

Our client, like any supply chain, is suffering shortages. During shortages, their clients
(retailers and drug stores) won’t place orders. This means that if you use historical order
figures to forecast future demand (see the example in the figure below), you will cripple your
time series with zero-periods due to these shortages.

Figure 5 Impact of shortages on sales numbers (data: online retailer).

Unfortunately, we didn’t get precise data on historical shortages. (A rigorous analysis of

historical inventory levels will be completed later in 2023.)
In summary, this dataset and forecasting setup is especially challenging as the demand
patterns are erratic, and we lack rich data (demand drivers, shortages, product transitions).

SupChains Solution
Forecasting Metrics
The dataset being erratic, we have chosen a simple combination of Median Absolute Error
(MAE, or simply forecast error in the figures) and bias to assess the forecasting quality of our

nicolas.vandeput@supchains.com Confidential
Case Study: Using machine learning to forecast demand despite limited data

forecasts. This combination has the advantage of being simple to interpret while looking both
at accuracy and bias.3

Machine Learning Forecast Engine

We developed a tree-based machine learning model (based on those introduced in Data
Science for Supply Chain Forecasting). This model leverages the latest technology (CPU-
multithreading and GPU computation) to generate a forecast in 5 minutes using a laptop.

Results
As illustrated hereunder, our machine learning models delivered more than 25% forecast
added value compared to a benchmark (a 12-weeks moving average).4

Figure 6 Our machine learning delivers an 27.9% FVA compared to the benchmark. The absolute error of the benchmark is
well above 100% due to the erratic demand patterns.

We could deliver this amazing 25% added value despite the lack of rich data. If the dataset
was enriched with demand drivers (marketing & promotions), life-cycle information
(predecessors and successors), and shortage data, we could achieve even better results.

Results over the Forecasting Horizon

As shown in the figure below, our machine learning model delivered accurate forecasts over
the 26-week horizon. Our model loses around 0.5% accuracy per week, whereas the
benchmark loses about 0.25%.5 Looking at the bias, machine learning is delivering stable
results – properly capturing long-term trends, cycles, and seasonality.

3
More information about forecasting metrics here.
4
We compute the added value as the % reduction of the scoring metric (the combination of MAE and
Bias). More information about forecast value added here. For more info, see «How do we select our
benchmark ?» and «How do we test our models ?» at the end of the document.
5
This difference can be mechanically explained by the evolution of the bias: the absolute error is
usually reduced when the bias is lower. As the benchmark achieves a lower bias for long-term
predictions, it also benefits from a reduced absolute error.

nicolas.vandeput@supchains.com Confidential
Case Study: Using machine learning to forecast demand despite limited data

Project Timeline
As only little data cleaning was needed, only two weeks of work were needed to create the
model and test it. A final week was dedicated to analyzing the models’ performance and
creating the final report.

Do you have a similar case?

Let’s get in touch to see how much value advanced forecasting can deliver to your supply
chain.

How do we select our benchmark?

For multiple reasons, we do not recommend using external industry benchmarks when
assessing forecasting accuracy (more info here). Instead, we compare our forecasting
models against a moving average (and so should you!). In the past, simple naïve forecasts
used to be considered suitable benchmarks. But they are too easy to beat. So instead, we
look for the best moving average by trying out different horizons (3, 6, 12, and 24 months).

How do we test our models?

We follow data science best practices: our models are tuned, trained, optimized, and
selected using a specific training dataset. Then, once we have chosen our favorite model,
we assess it against a test set (that is entirely unknown to the model). By doing so, we ensure
that the results obtained on the test set are as close as possible to real-life results.

nicolas.vandeput@supchains.com Confidential