Dilated RNN
The Dilated Recurrent Neural Network
(DilatedRNN
)
addresses common challenges of modeling long sequences like vanishing
gradients, computational efficiency, and improved model flexibility to
model complex relationships while maintaining its parsimony. The
DilatedRNN
builds a deep stack of RNN layers using skip conditions on the temporal
and the network’s depth dimensions. The temporal dilated recurrent skip
connections offer the capability to focus on multi-resolution inputs.The
predictions are obtained by transforming the hidden states into contexts
, that are decoded and adapted into
through MLPs.
where , is the hidden state for time , is the input at time and is the hidden state of the previous layer at , are static exogenous inputs, historic exogenous, are future exogenous available at the time of the prediction.
References
-Shiyu Chang, et al. “Dilated Recurrent Neural
Networks”.
-Yao Qin, et al. “A
Dual-Stage Attention-Based recurrent neural network for time series
prediction”.
-Kashif Rasul, et
al. “Zalando Research: PyTorch Dilated Recurrent Neural
Networks”.
source
DilatedRNN
DilatedRNN (h:int, input_size:int=-1, inference_input_size:int=-1, cell_type:str='LSTM', dilations:List[List[int]]=[[1, 2], [4, 8]], encoder_hidden_size:int=200, context_size:int=10, decoder_hidden_size:int=200, decoder_layers:int=2, futr_exog_list=None, hist_exog_list=None, stat_exog_list=None, loss=MAE(), valid_loss=None, max_steps:int=1000, learning_rate:float=0.001, num_lr_decays:int=3, early_stop_patience_steps:int=-1, val_check_steps:int=100, batch_size=32, valid_batch_size:Optional[int]=None, step_size:int=1, scaler_type:str='robust', random_seed:int=1, num_workers_loader:int=0, drop_last_loader:bool=False, optimizer=None, optimizer_kwargs=None, lr_scheduler=None, lr_scheduler_kwargs=None, **trainer_kwargs)
*DilatedRNN
Parameters:
h
: int, forecast horizon.
input_size
: int,
maximum sequence length for truncated train backpropagation. Default -1
uses all history.
inference_input_size
: int, maximum sequence
length for truncated inference. Default -1 uses all history.
cell_type
: str, type of RNN cell to use. Options: ‘GRU’, ‘RNN’,
‘LSTM’, ‘ResLSTM’, ‘AttentiveLSTM’.
dilations
: int list, dilations
betweem layers.
encoder_hidden_size
: int=200, units for the RNN’s
hidden state size.
context_size
: int=10, size of context vector
for each timestamp on the forecasting window.
decoder_hidden_size
:
int=200, size of hidden layer for the MLP decoder.
decoder_layers
:
int=2, number of layers for the MLP decoder.
futr_exog_list
: str
list, future exogenous columns.
hist_exog_list
: str list, historic
exogenous columns.
stat_exog_list
: str list, static exogenous
columns.
loss
: PyTorch module, instantiated train loss class from
losses
collection.
valid_loss
: PyTorch module=loss
, instantiated valid loss class from
losses
collection.
max_steps
: int, maximum number of training steps.
learning_rate
:
float, Learning rate between (0, 1).
num_lr_decays
: int, Number of
learning rate decays, evenly distributed across max_steps.
early_stop_patience_steps
: int, Number of validation iterations before
early stopping.
val_check_steps
: int, Number of training steps
between every validation loss check.
batch_size
: int=32, number of
different series in each batch.
valid_batch_size
: int=None, number
of different series in each validation and test batch.
step_size
:
int=1, step size between each window of temporal data.
scaler_type
: str=‘robust’, type of scaler for temporal inputs
normalization see temporal
scalers.
random_seed
: int=1, random_seed for pytorch initializer and numpy
generators.
num_workers_loader
: int=os.cpu_count(), workers to be
used by TimeSeriesDataLoader
.
drop_last_loader
: bool=False, if
True TimeSeriesDataLoader
drops last non-full batch.
alias
: str,
optional, Custom name of the model.
optimizer
: Subclass of
‘torch.optim.Optimizer’, optional, user specified optimizer instead of
the default choice (Adam).
optimizer_kwargs
: dict, optional, list
of parameters used by the user specified optimizer
.
lr_scheduler
: Subclass of ‘torch.optim.lr_scheduler.LRScheduler’,
optional, user specified lr_scheduler instead of the default choice
(StepLR).
lr_scheduler_kwargs
: dict, optional, list of parameters
used by the user specified lr_scheduler
.
**trainer_kwargs
: int,
keyword trainer arguments inherited from PyTorch Lighning’s
trainer.
*
Usage Example
import pandas as pd
import matplotlib.pyplot as plt
from neuralforecast import NeuralForecast
from neuralforecast.models import DilatedRNN
from neuralforecast.losses.pytorch import DistributionLoss
from neuralforecast.utils import AirPassengersPanel, AirPassengersStatic
Y_train_df = AirPassengersPanel[AirPassengersPanel.ds<AirPassengersPanel['ds'].values[-12]] # 132 train
Y_test_df = AirPassengersPanel[AirPassengersPanel.ds>=AirPassengersPanel['ds'].values[-12]].reset_index(drop=True) # 12 test
fcst = NeuralForecast(
models=[DilatedRNN(h=12,
input_size=-1,
loss=DistributionLoss(distribution='Normal', level=[80, 90]),
scaler_type='robust',
encoder_hidden_size=100,
max_steps=200,
futr_exog_list=['y_[lag12]'],
hist_exog_list=None,
stat_exog_list=['airline1'],
)
],
freq='M'
)
fcst.fit(df=Y_train_df, static_df=AirPassengersStatic)
forecasts = fcst.predict(futr_df=Y_test_df)
Y_hat_df = forecasts.reset_index(drop=False).drop(columns=['unique_id','ds'])
plot_df = pd.concat([Y_test_df, Y_hat_df], axis=1)
plot_df = pd.concat([Y_train_df, plot_df])
plot_df = plot_df[plot_df.unique_id=='Airline1'].drop('unique_id', axis=1)
plt.plot(plot_df['ds'], plot_df['y'], c='black', label='True')
plt.plot(plot_df['ds'], plot_df['DilatedRNN-median'], c='blue', label='median')
plt.fill_between(x=plot_df['ds'][-12:],
y1=plot_df['DilatedRNN-lo-90'][-12:].values,
y2=plot_df['DilatedRNN-hi-90'][-12:].values,
alpha=0.4, label='level 90')
plt.legend()
plt.grid()
plt.plot()