# Intermittent or Sparse Data

In this notebook, we’ll implement models for intermittent or sparse data

Intermittent or sparse data has very few non-zero observations. This type of data is hard to forecast because the zero values increase the uncertainty about the underlying patterns in the data. Furthermore, once a non-zero observation occurs, there can be considerable variation in its size. Intermittent time series are common in many industries, including finance, retail, transportation, and energy. Given the ubiquity of this type of series, special methods have been developed to forecast them. The first was from Croston (1972), followed by several variants and by different aggregation frameworks.

StatsForecast has implemented several models to forecast intermittent time series. By the end of this tutorial, you’ll have a good understanding of these models and how to use them.

**Outline:**

- Install libraries
- Load and explore the data
- Train models for intermittent data
- Plot forecasts and compute accuracy

TipYou can use Colab to run this Notebook interactively

TipFor forecasting at scale, we recommend you check this notebook done on Databricks.

## Install libraries

We assume that you have StatsForecast already installed. If not, check this guide for instructions on how to install StatsForecast

Install the necessary packages using `pip install statsforecast`

```
pip install statsforecast -U
```

## Load and explore the data

For this example, we’ll use a subset of the M5 Competition dataset. Each time series represents the unit sales of a particular product in a given Walmart store. At this level (product-store), most of the data is intermittent. We first need to import the data, and to do that, we’ll need datasetsforecast.

```
pip install datasetsforecast -U
```

```
from datasetsforecast.m5 import M5
```

The function to load the data is `M5.load()`

. It requieres the following
argument: - `directory`

: (str) The directory where the data will be
downloaded.

This function returns multiple outputs, but only the first one with the unit sales is needed.

```
df_total, *_ = M5.load('./data')
```

```
df_total.head()
```

unique_id | ds | y | |
---|---|---|---|

0 | FOODS_1_001_CA_1 | 2011-01-29 | 3.0 |

1 | FOODS_1_001_CA_1 | 2011-01-30 | 0.0 |

2 | FOODS_1_001_CA_1 | 2011-01-31 | 0.0 |

3 | FOODS_1_001_CA_1 | 2011-02-01 | 1.0 |

4 | FOODS_1_001_CA_1 | 2011-02-02 | 4.0 |

From this dataset, we’ll select the first 8 time series. You can select
any number you want by changing the value of `n_series`

.

```
n_series = 8
uids = df_total['unique_id'].unique()[:n_series]
df = df_total.query('unique_id in @uids')
```

We can plot these series using the `plot`

method from the
StatsForecast
class. This method has multiple parameters, and the required ones to
generate the plots in this notebook are explained below.

`df`

: A`pandas`

dataframe with columns [`unique_id`

,`ds`

,`y`

].`forecasts_df`

: A`pandas`

dataframe with columns [`unique_id`

,`ds`

] and models.`plot_random`

: (bool =`True`

) Plots the time series randomly.`max_insample_length`

: (int) The maximum number of train/insample observations to be plotted.`engine`

: (str =`plotly`

). The library used to generate the plots. It can also be`matplotlib`

for static plots.

```
from statsforecast import StatsForecast
```

```
StatsForecast.plot(df, plot_random = False, max_insample_length = 100)
```

Here we only plotted the last 100 observations, but we can visualize the
complete history by removing `max_insample_length`

. From these plots, we
can confirm that the data is indeed intermittent since it has multiple
periods with zero sales. In fact, in all cases but one, the median value
is zero.

```
df.groupby('unique_id')[['y']].median().query('unique_id in @uids')
```

y | |
---|---|

unique_id | |

FOODS_1_001_CA_1 | 0.0 |

FOODS_1_001_CA_2 | 1.0 |

FOODS_1_001_CA_3 | 0.0 |

FOODS_1_001_CA_4 | 0.0 |

FOODS_1_001_TX_1 | 0.0 |

FOODS_1_001_TX_2 | 0.0 |

FOODS_1_001_TX_3 | 0.0 |

FOODS_1_001_WI_1 | 0.0 |

## Train models for intermittent data

Before training any model, we need to separate the data in a train and a test set. The M5 Competition used the last 28 days as test set, so we’ll do the same.

```
dates = df['ds'].unique()[-28:] # last 28 days
train = df.query('ds not in @dates')
test = df.query('ds in @dates')
```

StatsForecast has efficient implementations of multiple models for intermittent data. The complete list of models available is here. In this notebook, we’ll use:

- Agregate-Dissagregate Intermittent Demand Approach (ADIDA)
- Croston Classic
- Intermittent Multiple Aggregation Prediction Algorithm (IMAPA)
- Teunter-Syntetos-Babai (TSB)

To use these models, we first need to import them from
`statsforecast.models`

and then we need to instantiate them.

```
from statsforecast.models import (
ADIDA,
CrostonClassic,
IMAPA,
TSB
)
# Create a list of models and instantiation parameters
models = [
ADIDA(),
CrostonClassic(),
IMAPA(),
TSB(alpha_d = 0.2, alpha_p = 0.2)
]
```

To instantiate a new StatsForecast object, we need the following parameters:

`df`

: The dataframe with the training data.`models`

: The list of models defined in the previous step.`freq`

: A string indicating the frequency of the data. See pandas’ available frequencies.`n_jobs`

: An integer that indicates the number of jobs used in parallel processing. Use -1 to select all cores.

```
sf = StatsForecast(
df = train,
models = models,
freq = 'D',
n_jobs = -1
)
```

Now we’re ready to generate the forecast. To do this, we’ll use the
`forecast`

method, which requires the forecasting horizon (in this case, 28 days)
as argument.

The models for intermittent series that are currently available in StatsForecast can only generate point-forecasts. If prediction intervals are needed, then a probabilisitic model should be used.

```
horizon = 28
forecasts = sf.forecast(h=horizon)
forecasts = forecasts.reset_index()
forecasts.head()
```

unique_id | ds | ADIDA | CrostonClassic | IMAPA | TSB | |
---|---|---|---|---|---|---|

0 | FOODS_1_001_CA_1 | 2016-05-23 | 0.791852 | 0.898247 | 0.705835 | 0.434313 |

1 | FOODS_1_001_CA_1 | 2016-05-24 | 0.791852 | 0.898247 | 0.705835 | 0.434313 |

2 | FOODS_1_001_CA_1 | 2016-05-25 | 0.791852 | 0.898247 | 0.705835 | 0.434313 |

3 | FOODS_1_001_CA_1 | 2016-05-26 | 0.791852 | 0.898247 | 0.705835 | 0.434313 |

4 | FOODS_1_001_CA_1 | 2016-05-27 | 0.791852 | 0.898247 | 0.705835 | 0.434313 |

Finally, we’ll merge the forecast with the actual values.

```
test = test.merge(forecasts, how='left', on=['unique_id', 'ds'])
```

## Plot forecasts and compute accuracy

We can generate plots using the `plot`

described above.

```
StatsForecast.plot(train, test, plot_random = False, max_insample_length = 100)
```

To compute the accuracy of the forecasts, we’ll use the Mean Average
Error (MAE), which is the sum of the absolute errors divided by the
number of forecasts. We’ll create a function to compute the MAE, and for
that, we’ll need to import `numpy`

.

```
import numpy as np
def mae(y_hat, y_true):
return np.mean(np.abs(y_hat-y_true))
```

```
y_true = test['y'].values
adida_preds = test['ADIDA'].values
croston_preds = test['CrostonClassic'].values
imapa_preds = test['IMAPA'].values
tsb_preds = test['TSB'].values
print('ADIDA MAE: \t %0.3f' % mae(adida_preds, y_true))
print('Croston Classic MAE: \t %0.3f' % mae(croston_preds, y_true))
print('IMAPA MAE: \t %0.3f' % mae(imapa_preds, y_true))
print('TSB MAE: \t %0.3f' % mae(tsb_preds, y_true))
```

```
ADIDA MAE: 0.949
Croston Classic MAE: 0.944
IMAPA MAE: 0.957
TSB MAE: 1.023
```

Hence, on average, the forecasts are one unit off.