Hyperparameter Optimization
Machine Learning forecasting methods are defined by many hyperparameters that control their behavior, with effects ranging from their speed and memory requirements to their predictive performance. For a long time, manual hyperparameter tuning prevailed. This approach is time-consuming, automated hyperparameter optimization methods have been introduced, proving more efficient than manual tuning, grid search, and random search.
The BaseAuto
class offers shared API connections to hyperparameter optimization algorithms like Optuna, HyperOpt, Dragonfly among others through ray
, which gives you access to grid search, bayesian optimization and other state-of-the-art tools like hyperband.
Comprehending the impacts of hyperparameters is still a precious skill, as it can help guide the design of informed hyperparameter spaces that are faster to explore automatically.
BaseAuto
BaseAuto (cls_model, h, loss, valid_loss, config, search_alg=<ray.tune.search.basic_variant.BasicVariantGenerator object at 0x7f9557bee4a0>, num_samples=10, cpus=4, gpus=0, refit_with_val=False, verbose=False, alias=None, backend='ray', callbacks=None)
*Class for Automatic Hyperparameter Optimization, it builds on top of
ray
to give access to a wide variety of hyperparameter optimization
tools ranging from classic grid search, to Bayesian optimization and
HyperBand algorithm.
The validation loss to be optimized is defined by the config['loss']
dictionary value, the config also contains the rest of the
hyperparameter search space.
It is important to note that the success of this hyperparameter optimization heavily relies on a strong correlation between the validation and test periods.*
Type | Default | Details | |
---|---|---|---|
cls_model | PyTorch/PyTorchLightning model | See neuralforecast.models collection here. | |
h | int | Forecast horizon | |
loss | PyTorch module | Instantiated train loss class from losses collection. | |
valid_loss | PyTorch module | Instantiated valid loss class from losses collection. | |
config | dict or callable | Dictionary with ray.tune defined search space or function that takes an optuna trial and returns a configuration dict. | |
search_alg | BasicVariantGenerator | <ray.tune.search.basic_variant.BasicVariantGenerator object at 0x7f9557bee4a0> | For ray see https://docs.ray.io/en/latest/tune/api_docs/suggestion.html For optuna see https://optuna.readthedocs.io/en/stable/reference/samplers/index.html. |
num_samples | int | 10 | Number of hyperparameter optimization steps/samples. |
cpus | int | 4 | Number of cpus to use during optimization. Only used with ray tune. |
gpus | int | 0 | Number of gpus to use during optimization, default all available. Only used with ray tune. |
refit_with_val | bool | False | Refit of best model should preserve val_size. |
verbose | bool | False | Track progress. |
alias | NoneType | None | Custom name of the model. |
backend | str | ray | Backend to use for searching the hyperparameter space, can be either ‘ray’ or ‘optuna’. |
callbacks | NoneType | None | List of functions to call during the optimization process. ray reference: https://docs.ray.io/en/latest/tune/tutorials/tune-metrics.html optuna reference: https://optuna.readthedocs.io/en/stable/tutorial/20_recipes/007_optuna_callback.html |
BaseAuto.fit
BaseAuto.fit (dataset, val_size=0, test_size=0, random_seed=None, distributed_config=None)
*BaseAuto.fit
Perform the hyperparameter optimization as specified by the BaseAuto
configuration dictionary config
.
The optimization is performed on the
TimeSeriesDataset
using temporal cross validation with the validation set that
sequentially precedes the test set.
Parameters:
dataset
: NeuralForecast’s
TimeSeriesDataset
see details
here
val_size
: int, size of temporal validation set (needs to be bigger
than 0).
test_size
: int, size of temporal test set (default
0).
random_seed
: int=None, random_seed for hyperparameter
exploration algorithms, not yet implemented.
Returns:
self
: fitted instance of BaseAuto
with best hyperparameters and
results
.*
BaseAuto.predict
BaseAuto.predict (dataset, step_size=1, **data_kwargs)
*BaseAuto.predict
Predictions of the best performing model on validation.
Parameters:
dataset
: NeuralForecast’s
TimeSeriesDataset
see details
here
step_size
: int, steps between sequential predictions, (default 1).
**data_kwarg
: additional parameters for the dataset module.
random_seed
: int=None, random_seed for hyperparameter exploration
algorithms (not implemented).
Returns:
y_hat
: numpy
predictions of the
NeuralForecast
model.
*
class RayLogLossesCallback(tune.Callback):
def on_trial_complete(self, iteration, trials, trial, **info):
result = trial.last_result
print(40 * '-' + 'Trial finished' + 40 * '-')
print(f'Train loss: {result["train_loss"]:.2f}. Valid loss: {result["loss"]:.2f}')
print(80 * '-')
config = {
"hidden_size": tune.choice([512]),
"num_layers": tune.choice([3, 4]),
"input_size": 12,
"max_steps": 10,
"val_check_steps": 5
}
auto = BaseAuto(h=12, loss=MAE(), valid_loss=MSE(), cls_model=MLP, config=config, num_samples=2, cpus=1, gpus=0, callbacks=[RayLogLossesCallback()])
auto.fit(dataset=dataset)
y_hat = auto.predict(dataset=dataset)
assert mae(Y_test_df['y'].values, y_hat[:, 0]) < 200
def config_f(trial):
return {
"hidden_size": trial.suggest_categorical('hidden_size', [512]),
"num_layers": trial.suggest_categorical('num_layers', [3, 4]),
"input_size": 12,
"max_steps": 10,
"val_check_steps": 5
}
class OptunaLogLossesCallback:
def __call__(self, study, trial):
metrics = trial.user_attrs['METRICS']
print(40 * '-' + 'Trial finished' + 40 * '-')
print(f'Train loss: {metrics["train_loss"]:.2f}. Valid loss: {metrics["loss"]:.2f}')
print(80 * '-')
auto2 = BaseAuto(h=12, loss=MAE(), valid_loss=MSE(), cls_model=MLP, config=config_f, search_alg=optuna.samplers.RandomSampler(), num_samples=2, backend='optuna', callbacks=[OptunaLogLossesCallback()])
auto2.fit(dataset=dataset)
assert isinstance(auto2.results, optuna.Study)
y_hat2 = auto2.predict(dataset=dataset)
assert mae(Y_test_df['y'].values, y_hat2[:, 0]) < 200
References
- James Bergstra, Remi Bardenet, Yoshua Bengio, and Balazs Kegl (2011). “Algorithms for Hyper-Parameter Optimization”. In: Advances in Neural Information Processing Systems. url: https://proceedings.neurips.cc/paper/2011/file/86e8f7ab32cfd12577bc2619bc635690-Paper.pdf
- Kirthevasan Kandasamy, Karun Raju Vysyaraju, Willie Neiswanger, Biswajit Paria, Christopher R. Collins, Jeff Schneider, Barnabas Poczos, Eric P. Xing (2019). “Tuning Hyperparameters without Grad Students: Scalable and Robust Bayesian Optimisation with Dragonfly”. Journal of Machine Learning Research. url: https://arxiv.org/abs/1903.06694
- Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, Ameet Talwalkar (2016). “Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization”. Journal of Machine Learning Research. url: https://arxiv.org/abs/1603.06560