Skip to main content

​

​
M5

M5(source_url='https://github.com/Nixtla/m5-forecasts/raw/main/datasets/m5.zip')

​
M5.download

download(directory)
Downloads M5 Competition Dataset. Parameters:
NameTypeDescriptionDefault
directorystrDirectory path to download dataset.required

​
M5.load

load(directory, cache=True)
Downloads and loads M5 data. Parameters:
NameTypeDescriptionDefault
directorystrDirectory where data will be downloaded.required
cacheboolIf True saves and loads.True
Returns:
TypeDescription
Tuple[DataFrame, DataFrame, DataFrame]Tuple[pd.DataFrame, pd.DataFrame, pd.DataFrame]: Target time series with columns [‘unique_id’, ‘ds’, ‘y’], Exogenous time series with columns [‘unique_id’, ‘ds’, ‘y’], Static exogenous variables with columns [‘unique_id’, ‘ds’] and static variables.

​
M5.source_url

source_url: str = 'https://github.com/Nixtla/m5-forecasts/raw/main/datasets/m5.zip'

​
Evaluation class

​
M5Evaluation

​
M5Evaluation.aggregate_levels

aggregate_levels(y_hat, categories=None)
Aggregates the 30_480 series to get 42_840. Parameters:
NameTypeDescriptionDefault
y_hatDataFrameForecasts as wide pandas dataframe with columns [‘unique_id’].required
categoriesDataFrameCategories of M5 dataset (not used). Defaults to None.None
Returns:
TypeDescription
DataFramepd.DataFrame: Aggregated forecasts as wide pandas dataframe with columns [‘unique_id’].

​
M5Evaluation.evaluate

evaluate(directory, y_hat, validation=False)
Evaluates y_hat according to M4 methodology. Parameters:
NameTypeDescriptionDefault
directorystrDirectory where data will be downloaded.required
validationboolWheter perform validation evaluation. Default False, return test evaluation.False
y_hatUnion[DataFrame, str]Forecasts as wide pandas dataframe with columns [‘unique_id’] and forecasts or benchmark url from https://github.com/Nixtla/m5-forecasts/tree/main/forecasts.required
Returns:
TypeDescription
DataFramepd.DataFrame: DataFrame with columns OWA, SMAPE, MASE and group as index.
Examples: 
m5_winner_url = 'https://github.com/Nixtla/m5-forecasts/raw/main/forecasts/0001 YJ_STU.zip'
winner_evaluation = M5Evaluation.evaluate('data', m5_winner_url)

m5_second_place_url = 'https://github.com/Nixtla/m5-forecasts/raw/main/forecasts/0002 Matthias.zip'
m5_second_place_forecasts = M5Evaluation.load_benchmark('data', m5_second_place_url)
second_place_evaluation = M5Evaluation.evaluate('data', m5_second_place_forecasts)

​
M5Evaluation.levels

levels: dict = dict(Level1=['total'], Level2=['state_id'], Level3=['store_id'], Level4=['cat_id'], Level5=['dept_id'], Level6=['state_id', 'cat_id'], Level7=['state_id', 'dept_id'], Level8=['store_id', 'cat_id'], Level9=['store_id', 'dept_id'], Level10=['item_id'], Level11=['state_id', 'item_id'], Level12=['item_id', 'store_id'])

​
M5Evaluation.load_benchmark

load_benchmark(directory, source_url=None, validation=False)
Downloads and loads a bechmark forecasts. Parameters:
NameTypeDescriptionDefault
directorystrDirectory where data will be downloaded.required
source_urlstrOptional benchmark url obtained from https://github.com/Nixtla/m5-forecasts/tree/master/forecasts. If None returns the M5 winner.None
validationboolWheter return validation forecasts. Default False, return test forecasts.False
Returns:
TypeDescription
ndarraynp.ndarray: Numpy array of shape (n_series, horizon).
Example: 
winner_benchmark = M5Evaluation.load_benchmark('data')
winner_evaluation = M5Evaluation.evaluate('data', winner_benchmark)

​
URL-based evaluation

The method evaluate from the class M5Evaluation can receive a url of a submission to the M5 competiton. The results compared to the on-the-fly evaluation were obtained from the official evaluation. 
m5_winner_url = 'https://github.com/Nixtla/m5-forecasts/raw/main/forecasts/0001 YJ_STU.zip'
winner_evaluation = M5Evaluation.evaluate('data', m5_winner_url)
# Test of the same evaluation as the original one
test_close(winner_evaluation.loc['Total'].item(), 0.520, eps=1e-3)
winner_evaluation

​
Pandas-based evaluation

Also the method evaluate can recevie a pandas DataFrame of forecasts. 
m5_second_place_url = 'https://github.com/Nixtla/m5-forecasts/raw/main/forecasts/0002 Matthias.zip'
m5_second_place_forecasts = M5Evaluation.load_benchmark('data', m5_second_place_url)
second_place_evaluation = M5Evaluation.evaluate('data', m5_second_place_forecasts)
# Test of the same evaluation as the original one
test_close(second_place_evaluation.loc['Total'].item(), 0.528, eps=1e-3)
second_place_evaluation
By default you can load the winner benchmark using the following. 
winner_benchmark = M5Evaluation.load_benchmark('data')
winner_evaluation = M5Evaluation.evaluate('data', winner_benchmark)
# Test of the same evaluation as the original one
test_close(winner_evaluation.loc['Total'].item(), 0.520, eps=1e-3)
winner_evaluation

​
Validation evaluation

You can also evaluate the official validation set. 
winner_benchmark_val = M5Evaluation.load_benchmark('data', validation=True)
winner_evaluation_val = M5Evaluation.evaluate('data', winner_benchmark_val, validation=True)
winner_evaluation_val

​
Kaggle-Competition-M5 References

The evaluation metric of the Favorita Kaggle competition was the normalized weighted root mean squared logarithmic error (NWRMSLE). Perishable items have a score weight of 1.25; otherwise, the weight is 1.0. NWRMSLE=∑i=1nwi(log(y^i+1)−log(yi+1))2∑i=1nwi NWRMSLE = \sqrt{\frac{\sum^{n}_{i=1} w_{i}\left(log(\hat{y}_{i}+1) - log(y_{i}+1)\right)^{2}}{\sum^{n}_{i=1} w_{i}}}
Kaggle Competition Forecasting Methods16D ahead NWRMSLE
LGBM [1]0.5091
Seq2Seq WaveNet [2]0.5129
  1. Corporación Favorita. Corporación favorita grocery sales forecasting. Kaggle Competition Leaderboard, 2018.
  2. Glib Kechyn, Lucius Yu, Yangguang Zang, and Svyatoslav Kechyn. Sales forecasting using wavenet within the framework of the Favorita Kaggle competition. Computing Research Repository, abs/1803.04037, 2018.