Datasets
M5 dataset
Download and evaluate the M5 dataset.
Download data class
source
M5
M5 (source_url:str='https://github.com/Nixtla/m5- forecasts/raw/main/datasets/m5.zip')
Test number of series
Evaluation class
source
M5Evaluation
M5Evaluation ()
Initialize self. See help(type(self)) for accurate signature.
URL-based evaluation
The method evaluate
from the class
M5Evaluation
can receive a url of a submission to the M5
competiton.
The results compared to the on-the-fly evaluation were obtained from the official evaluation.
m5_winner_url = 'https://github.com/Nixtla/m5-forecasts/raw/main/forecasts/0001 YJ_STU.zip'
winner_evaluation = M5Evaluation.evaluate('data', m5_winner_url)
# Test of the same evaluation as the original one
test_close(winner_evaluation.loc['Total'].item(), 0.520, eps=1e-3)
winner_evaluation
Pandas-based evaluation
Also the method evaluate
can recevie a pandas DataFrame of forecasts.
m5_second_place_url = 'https://github.com/Nixtla/m5-forecasts/raw/main/forecasts/0002 Matthias.zip'
m5_second_place_forecasts = M5Evaluation.load_benchmark('data', m5_second_place_url)
second_place_evaluation = M5Evaluation.evaluate('data', m5_second_place_forecasts)
# Test of the same evaluation as the original one
test_close(second_place_evaluation.loc['Total'].item(), 0.528, eps=1e-3)
second_place_evaluation
By default you can load the winner benchmark using the following.
winner_benchmark = M5Evaluation.load_benchmark('data')
winner_evaluation = M5Evaluation.evaluate('data', winner_benchmark)
# Test of the same evaluation as the original one
test_close(winner_evaluation.loc['Total'].item(), 0.520, eps=1e-3)
winner_evaluation
Validation evaluation
You can also evaluate the official validation set.
winner_benchmark_val = M5Evaluation.load_benchmark('data', validation=True)
winner_evaluation_val = M5Evaluation.evaluate('data', winner_benchmark_val, validation=True)
winner_evaluation_val
Kaggle-Competition-M5 References
The evaluation metric of the Favorita Kaggle competition was the normalized weighted root mean squared logarithmic error (NWRMSLE). Perishable items have a score weight of 1.25; otherwise, the weight is 1.0.
Kaggle Competition Forecasting Methods | 16D ahead NWRMSLE |
---|---|
LGBM [1] | 0.5091 |
Seq2Seq WaveNet [2] | 0.5129 |
- Corporación Favorita. Corporación favorita grocery sales forecasting. Kaggle Competition Leaderboard, 2018.
- Glib Kechyn, Lucius Yu, Yangguang Zang, and Svyatoslav Kechyn. Sales forecasting using wavenet within the framework of the Favorita Kaggle competition. Computing Research Repository, abs/1803.04037, 2018.