Skip to main content
Machine Learning forecasting methods are defined by many hyperparameters that control their behavior, with effects ranging from their speed and memory requirements to their predictive performance. For a long time, manual hyperparameter tuning prevailed. This approach is time-consuming, automated hyperparameter optimization methods have been introduced, proving more efficient than manual tuning, grid search, and random search.

The BaseAuto class offers shared API connections to hyperparameter optimization algorithms like Optuna, HyperOpt, Dragonfly among others through ray, which gives you access to grid search, bayesian optimization and other state-of-the-art tools like hyperband. Comprehending the impacts of hyperparameters is still a precious skill, as it can help guide the design of informed hyperparameter spaces that are faster to explore automatically. Figure 1. Example of dataset split (left), validation (yellow) and test (orange). The hyperparameter optimization guiding signal is obtained from the validation set.

​

​
BaseAuto

BaseAuto(
    cls_model,
    h,
    loss,
    valid_loss,
    config,
    search_alg=BasicVariantGenerator(random_state=1),
    num_samples=10,
    cpus=cpu_count(),
    gpus=torch.cuda.device_count(),
    refit_with_val=False,
    verbose=False,
    alias=None,
    backend="ray",
    callbacks=None,
)
Bases: LightningModule Class for Automatic Hyperparameter Optimization, it builds on top of ray to give access to a wide variety of hyperparameter optimization tools ranging from classic grid search, to Bayesian optimization and HyperBand algorithm. The validation loss to be optimized is defined by the config['loss'] dictionary value, the config also contains the rest of the hyperparameter search space. It is important to note that the success of this hyperparameter optimization heavily relies on a strong correlation between the validation and test periods. Parameters:
NameTypeDescriptionDefault
cls_modelPyTorch/PyTorchLightning modelSee neuralforecast.models collection here.required
hintForecast horizonrequired
lossPyTorch moduleInstantiated train loss class from losses collection.required
valid_lossPyTorch moduleInstantiated valid loss class from losses collection.required
configdict or callableDictionary with ray.tune defined search space or function that takes an optuna trial and returns a configuration dict.required
search_algray.tune.search variant or optuna.samplerFor ray see https://docs.ray.io/en/latest/tune/api_docs/suggestion.html For optuna see https://optuna.readthedocs.io/en/stable/reference/samplers/index.html.BasicVariantGenerator(random_state=1)
num_samplesintNumber of hyperparameter optimization steps/samples.10
cpusintNumber of cpus to use during optimization. Only used with ray tune.cpu_count()
gpusintNumber of gpus to use during optimization, default all available. Only used with ray tune.device_count()
refit_with_valboolRefit of best model should preserve val_size.False
verboseboolTrack progress.False
aliasstrCustom name of the model.None
backendstrBackend to use for searching the hyperparameter space, can be either ‘ray’ or ‘optuna’.‘ray’
callbackslist of callableList of functions to call during the optimization process. ray reference: https://docs.ray.io/en/latest/tune/tutorials/tune-metrics.html optuna reference: https://optuna.readthedocs.io/en/stableNone

​
BaseAuto.fit

fit(
    dataset, val_size=0, test_size=0, random_seed=None, distributed_config=None
)
BaseAuto.fit Perform the hyperparameter optimization as specified by the BaseAuto configuration dictionary config. The optimization is performed on the TimeSeriesDataset using temporal cross validation with the validation set that sequentially precedes the test set. Parameters:
NameTypeDescriptionDefault
datasetNeuralForecast’s TimeSeriesDatasetNeuralForecast’s TimeSeriesDataset see details hererequired
val_sizeintSize of temporal validation set (needs to be bigger than 0).0
test_sizeintSize of temporal test set (default 0).0
random_seedintRandom seed for hyperparameter exploration algorithms, not yet implemented.None
Returns:
NameTypeDescription
selfFitted instance of BaseAuto with best hyperparameters and results.

​
BaseAuto.predict

predict(dataset, step_size=1, h=None, **data_kwargs)
BaseAuto.predict Predictions of the best performing model on validation. Parameters:
NameTypeDescriptionDefault
datasetNeuralForecast’s TimeSeriesDatasetNeuralForecast’s TimeSeriesDataset see details hererequired
step_sizeintSteps between sequential predictions, (default 1).1
hintPrediction horizon, if None, uses the model’s fitted horizon. Defaults to None.None
**data_kwargAdditional parameters for the dataset module.required
Returns:
NameTypeDescription
y_hatNumpy predictions of the NeuralForecast model.

​
Usage Example

class RayLogLossesCallback(tune.Callback):
    def on_trial_complete(self, iteration, trials, trial, **info):
        result = trial.last_result
        print(40 * '-' + 'Trial finished' + 40 * '-')
        print(f'Train loss: {result["train_loss"]:.2f}. Valid loss: {result["loss"]:.2f}')
        print(80 * '-')

config = {
    "hidden_size": tune.choice([512]),
    "num_layers": tune.choice([3, 4]),
    "input_size": 12,
    "max_steps": 10,
    "val_check_steps": 5
}
auto = BaseAuto(h=12, loss=MAE(), valid_loss=MSE(), cls_model=MLP, config=config, num_samples=2, cpus=1, gpus=0, callbacks=[RayLogLossesCallback()])
auto.fit(dataset=dataset)
y_hat = auto.predict(dataset=dataset)
assert mae(Y_test_df['y'].values, y_hat[:, 0]) < 200

def config_f(trial):
    return {
        "hidden_size": trial.suggest_categorical('hidden_size', [512]),
        "num_layers": trial.suggest_categorical('num_layers', [3, 4]),
        "input_size": 12,
        "max_steps": 10,
        "val_check_steps": 5
    }

class OptunaLogLossesCallback:
    def __call__(self, study, trial):
        metrics = trial.user_attrs['METRICS']
        print(40 * '-' + 'Trial finished' + 40 * '-')
        print(f'Train loss: {metrics["train_loss"]:.2f}. Valid loss: {metrics["loss"]:.2f}')
        print(80 * '-')

auto2 = BaseAuto(h=12, loss=MAE(), valid_loss=MSE(), cls_model=MLP, config=config_f, search_alg=optuna.samplers.RandomSampler(), num_samples=2, backend='optuna', callbacks=[OptunaLogLossesCallback()])
auto2.fit(dataset=dataset)
assert isinstance(auto2.results, optuna.Study)
y_hat2 = auto2.predict(dataset=dataset)
assert mae(Y_test_df['y'].values, y_hat2[:, 0]) < 200

​
References