API Reference
Evaluation
Model performance evaluation
source
evaluate
Evaluate forecast using different metrics.
Type | Default | Details | |
---|---|---|---|
df | AnyDFType | Forecasts to evaluate. Must have id_col , time_col , target_col and models’ predictions. | |
metrics | List | Functions with arguments df , models , id_col , target_col and optionally train_df . | |
models | Optional | None | Names of the models to evaluate. If None will use every column in the dataframe after removing id, time and target. |
train_df | Optional | None | Training set. Used to evaluate metrics such as mase . |
level | Optional | None | Prediction interval levels. Used to compute losses that rely on quantiles. |
id_col | str | unique_id | Column that identifies each serie. |
time_col | str | ds | Column that identifies each timestep, its values can be timestamps or integers. |
target_col | str | y | Column that contains the target. |
agg_fn | Optional | None | Statistic to compute on the scores by id to reduce them to a single number. |
Returns | AnyDFType | Metrics with one row per (id, metric) combination and one column per model. If agg_fn is not None , there is only one row per metric. |
unique_id | metric | model0 | model1 | |
---|---|---|---|---|
0 | 0 | mae | 0.158108 | 0.163246 |
1 | 1 | mae | 0.160109 | 0.143805 |
2 | 2 | mae | 0.159815 | 0.170510 |
3 | 3 | mae | 0.168537 | 0.161595 |
4 | 4 | mae | 0.170182 | 0.163329 |
… | … | … | … | … |
175 | 5 | scaled_crps | 0.034202 | 0.035472 |
176 | 6 | scaled_crps | 0.034880 | 0.033610 |
177 | 7 | scaled_crps | 0.034337 | 0.034745 |
178 | 8 | scaled_crps | 0.033336 | 0.032459 |
179 | 9 | scaled_crps | 0.034766 | 0.035243 |
metric | model0 | model1 | |
---|---|---|---|
0 | calibration_q0.025 | 0.000000 | 0.000000 |
1 | calibration_q0.1 | 0.000000 | 0.000000 |
2 | calibration_q0.9 | 0.833993 | 0.815833 |
3 | calibration_q0.975 | 0.853991 | 0.836949 |
4 | coverage_level80 | 0.833993 | 0.815833 |
5 | coverage_level95 | 0.853991 | 0.836949 |
6 | mae | 0.161286 | 0.162281 |
7 | mape | 0.048894 | 0.049624 |
8 | mase | 0.966846 | 0.975354 |
9 | mqloss | 0.056904 | 0.056216 |
10 | mse | 0.048653 | 0.049198 |
11 | quantile_loss_q0.025 | 0.019990 | 0.019474 |
12 | quantile_loss_q0.1 | 0.067315 | 0.065781 |
13 | quantile_loss_q0.9 | 0.095510 | 0.093841 |
14 | quantile_loss_q0.975 | 0.044803 | 0.045767 |
15 | rmse | 0.220357 | 0.221543 |
16 | scaled_crps | 0.035003 | 0.034576 |
17 | smape | 0.024475 | 0.024902 |