Re-using fine-tuned models

1. Import packages

First, we import the required packages and initialize the Nixtla client

import pandas as pd
from nixtla import NixtlaClient
from utilsforecast.losses import rmse
from utilsforecast.evaluation import evaluate

nixtla_client = NixtlaClient(
    # defaults to os.environ["NIXTLA_API_KEY"]
    api_key = 'my_api_key_provided_by_nixtla'
)

2. Load data

df = pd.read_parquet('https://datasets-nixtla.s3.amazonaws.com/m4-hourly.parquet')

h = 48
valid = df.groupby('unique_id', observed=True).tail(h)
train = df.drop(valid.index)
train.head()

	unique_id	ds	y
0	H1	1	605.0
1	H1	2	586.0
2	H1	3	586.0
3	H1	4	559.0
4	H1	5	511.0

3. Zero-shot forecast

We can try forecasting without any finetuning to see how well TimeGPT does.

fcst_kwargs = {'df': train, 'freq': 1, 'model': 'timegpt-1-long-horizon'}
fcst = nixtla_client.forecast(h=h, **fcst_kwargs)
zero_shot_eval = evaluate(fcst.merge(valid), metrics=[rmse], agg_fn='mean')
zero_shot_eval

INFO:nixtla.nixtla_client:Validating inputs...
INFO:nixtla.nixtla_client:Preprocessing dataframes...
INFO:nixtla.nixtla_client:Querying model metadata...
WARNING:nixtla.nixtla_client:The specified horizon "h" exceeds the model horizon, this may lead to less accurate forecasts. Please consider using a smaller horizon.
INFO:nixtla.nixtla_client:Restricting input...
INFO:nixtla.nixtla_client:Calling Forecast Endpoint...

	metric	TimeGPT
0	rmse	1504.474342

4. Fine-tune

We can now fine-tune TimeGPT a little and save our model for later use. We can define the ID that we want that model to have by providing it through output_model_id.

first_model_id = 'my-first-finetuned-model'
nixtla_client.finetune(output_model_id=first_model_id, **fcst_kwargs)

INFO:nixtla.nixtla_client:Validating inputs...
INFO:nixtla.nixtla_client:Preprocessing dataframes...
INFO:nixtla.nixtla_client:Calling Fine-tune Endpoint...

'my-first-finetuned-model'

We can now forecast using this fine-tuned model by providing its ID through the finetuned_model_id argument.

first_finetune_fcst = nixtla_client.forecast(h=h, finetuned_model_id=first_model_id, **fcst_kwargs)
first_finetune_eval = evaluate(first_finetune_fcst.merge(valid), metrics=[rmse], agg_fn='mean')
zero_shot_eval.merge(first_finetune_eval, on=['metric'], suffixes=('_zero_shot', '_first_finetune'))

INFO:nixtla.nixtla_client:Validating inputs...
INFO:nixtla.nixtla_client:Preprocessing dataframes...
WARNING:nixtla.nixtla_client:The specified horizon "h" exceeds the model horizon, this may lead to less accurate forecasts. Please consider using a smaller horizon.
INFO:nixtla.nixtla_client:Restricting input...
INFO:nixtla.nixtla_client:Calling Forecast Endpoint...

	metric	TimeGPT_zero_shot	TimeGPT_first_finetune
0	rmse	1504.474342	1472.024619

We can see the error was reduced.

5. Further fine-tune

We can now take this model and fine-tune it a bit further by using the NixtlaClient.finetune method but providing our already fine-tuned model as finetuned_model_id, which will take that model and fine-tune it a bit more. We can also change the fine-tuning settings, like using finetune_depth=3, for example.

second_model_id = nixtla_client.finetune(finetuned_model_id=first_model_id, finetune_depth=3, **fcst_kwargs)
second_model_id

INFO:nixtla.nixtla_client:Validating inputs...
INFO:nixtla.nixtla_client:Preprocessing dataframes...
INFO:nixtla.nixtla_client:Calling Fine-tune Endpoint...

'468b13fb-4b26-447a-bd87-87a64b50d913'

Since we didn’t provide output_model_id this time, it got assigned an UUID.

We can now use this model to forecast.

second_finetune_fcst = nixtla_client.forecast(h=h, finetuned_model_id=second_model_id, **fcst_kwargs)
second_finetune_eval = evaluate(second_finetune_fcst.merge(valid), metrics=[rmse], agg_fn='mean')
first_finetune_eval.merge(second_finetune_eval, on=['metric'], suffixes=('_first_finetune', '_second_finetune'))

INFO:nixtla.nixtla_client:Validating inputs...
INFO:nixtla.nixtla_client:Preprocessing dataframes...
WARNING:nixtla.nixtla_client:The specified horizon "h" exceeds the model horizon, this may lead to less accurate forecasts. Please consider using a smaller horizon.
INFO:nixtla.nixtla_client:Restricting input...
INFO:nixtla.nixtla_client:Calling Forecast Endpoint...

	metric	TimeGPT_first_finetune	TimeGPT_second_finetune
0	rmse	1472.024619	1435.365211

We can see the error was reduced a bit more.

6. Listing fine-tuned models

We can list our fine-tuned models with the NixtlaClient.finetuned_models method.

finetuned_models = nixtla_client.finetuned_models()
finetuned_models

[FinetunedModel(id='468b13fb-4b26-447a-bd87-87a64b50d913', created_at=datetime.datetime(2024, 12, 30, 17, 57, 31, 241455, tzinfo=TzInfo(UTC)), created_by='user', base_model_id='my-first-finetuned-model', steps=10, depth=3, loss='default', model='timegpt-1-long-horizon', freq='MS'),
 FinetunedModel(id='my-first-finetuned-model', created_at=datetime.datetime(2024, 12, 30, 17, 57, 16, 978907, tzinfo=TzInfo(UTC)), created_by='user', base_model_id='None', steps=10, depth=1, loss='default', model='timegpt-1-long-horizon', freq='MS')]

While that representation may be useful for programmatic use, in this exploratory setting it’s nicer to see them as a dataframe, which we can get by providing as_df=True.

nixtla_client.finetuned_models(as_df=True)

	id	created_at	created_by	base_model_id	steps	depth	loss	model	freq
0	468b13fb-4b26-447a-bd87-87a64b50d913	2024-12-30 17:57:31.241455+00:00	user	my-first-finetuned-model	10	3	default	timegpt-1-long-horizon	MS
1	my-first-finetuned-model	2024-12-30 17:57:16.978907+00:00	user	None	10	1	default	timegpt-1-long-horizon	MS

We can seee that the base_model_id of our second model is our first model, along with other metadata.

7. Deleting fine-tuned models

In order to keep things organized, and since there’s a limit of 50 fine-tuned models, you can delete models that weren’t so promising to make room for more experiments. For example, we can delete our first finetuned model. Note that even though it was used as the base for our second model, they’re saved independently so removing it won’t affect our second model, except for the dangling metadata.

nixtla_client.delete_finetuned_model(first_model_id)

True

We can verify that our first model model doesn’t show up anymore in our available models.

nixtla_client.finetuned_models(as_df=True)

	id	created_at	created_by	base_model_id	steps	depth	loss	model	freq
0	468b13fb-4b26-447a-bd87-87a64b50d913	2024-12-30 17:57:31.241455+00:00	user	my-first-finetuned-model	10	3	default	timegpt-1-long-horizon	MS

On this page

1. Import packages
2. Load data
3. Zero-shot forecast
4. Fine-tune
5. Further fine-tune
6. Listing fine-tuned models
7. Deleting fine-tuned models

1. Import packages

First, we import the required packages and initialize the Nixtla client

import pandas as pd
from nixtla import NixtlaClient
from utilsforecast.losses import rmse
from utilsforecast.evaluation import evaluate

nixtla_client = NixtlaClient(
    # defaults to os.environ["NIXTLA_API_KEY"]
    api_key = 'my_api_key_provided_by_nixtla'
)

2. Load data

df = pd.read_parquet('https://datasets-nixtla.s3.amazonaws.com/m4-hourly.parquet')

h = 48
valid = df.groupby('unique_id', observed=True).tail(h)
train = df.drop(valid.index)
train.head()

	unique_id	ds	y
0	H1	1	605.0
1	H1	2	586.0
2	H1	3	586.0
3	H1	4	559.0
4	H1	5	511.0

3. Zero-shot forecast

We can try forecasting without any finetuning to see how well TimeGPT does.

fcst_kwargs = {'df': train, 'freq': 1, 'model': 'timegpt-1-long-horizon'}
fcst = nixtla_client.forecast(h=h, **fcst_kwargs)
zero_shot_eval = evaluate(fcst.merge(valid), metrics=[rmse], agg_fn='mean')
zero_shot_eval

INFO:nixtla.nixtla_client:Validating inputs...
INFO:nixtla.nixtla_client:Preprocessing dataframes...
INFO:nixtla.nixtla_client:Querying model metadata...
WARNING:nixtla.nixtla_client:The specified horizon "h" exceeds the model horizon, this may lead to less accurate forecasts. Please consider using a smaller horizon.
INFO:nixtla.nixtla_client:Restricting input...
INFO:nixtla.nixtla_client:Calling Forecast Endpoint...

	metric	TimeGPT
0	rmse	1504.474342

4. Fine-tune

We can now fine-tune TimeGPT a little and save our model for later use. We can define the ID that we want that model to have by providing it through output_model_id.

first_model_id = 'my-first-finetuned-model'
nixtla_client.finetune(output_model_id=first_model_id, **fcst_kwargs)

INFO:nixtla.nixtla_client:Validating inputs...
INFO:nixtla.nixtla_client:Preprocessing dataframes...
INFO:nixtla.nixtla_client:Calling Fine-tune Endpoint...

'my-first-finetuned-model'

We can now forecast using this fine-tuned model by providing its ID through the finetuned_model_id argument.

first_finetune_fcst = nixtla_client.forecast(h=h, finetuned_model_id=first_model_id, **fcst_kwargs)
first_finetune_eval = evaluate(first_finetune_fcst.merge(valid), metrics=[rmse], agg_fn='mean')
zero_shot_eval.merge(first_finetune_eval, on=['metric'], suffixes=('_zero_shot', '_first_finetune'))

INFO:nixtla.nixtla_client:Validating inputs...
INFO:nixtla.nixtla_client:Preprocessing dataframes...
WARNING:nixtla.nixtla_client:The specified horizon "h" exceeds the model horizon, this may lead to less accurate forecasts. Please consider using a smaller horizon.
INFO:nixtla.nixtla_client:Restricting input...
INFO:nixtla.nixtla_client:Calling Forecast Endpoint...

	metric	TimeGPT_zero_shot	TimeGPT_first_finetune
0	rmse	1504.474342	1472.024619

We can see the error was reduced.

5. Further fine-tune

second_model_id = nixtla_client.finetune(finetuned_model_id=first_model_id, finetune_depth=3, **fcst_kwargs)
second_model_id

INFO:nixtla.nixtla_client:Validating inputs...
INFO:nixtla.nixtla_client:Preprocessing dataframes...
INFO:nixtla.nixtla_client:Calling Fine-tune Endpoint...

'468b13fb-4b26-447a-bd87-87a64b50d913'

Since we didn’t provide output_model_id this time, it got assigned an UUID.

We can now use this model to forecast.

second_finetune_fcst = nixtla_client.forecast(h=h, finetuned_model_id=second_model_id, **fcst_kwargs)
second_finetune_eval = evaluate(second_finetune_fcst.merge(valid), metrics=[rmse], agg_fn='mean')
first_finetune_eval.merge(second_finetune_eval, on=['metric'], suffixes=('_first_finetune', '_second_finetune'))

INFO:nixtla.nixtla_client:Validating inputs...
INFO:nixtla.nixtla_client:Preprocessing dataframes...
WARNING:nixtla.nixtla_client:The specified horizon "h" exceeds the model horizon, this may lead to less accurate forecasts. Please consider using a smaller horizon.
INFO:nixtla.nixtla_client:Restricting input...
INFO:nixtla.nixtla_client:Calling Forecast Endpoint...

	metric	TimeGPT_first_finetune	TimeGPT_second_finetune
0	rmse	1472.024619	1435.365211

We can see the error was reduced a bit more.

6. Listing fine-tuned models

We can list our fine-tuned models with the NixtlaClient.finetuned_models method.

finetuned_models = nixtla_client.finetuned_models()
finetuned_models

[FinetunedModel(id='468b13fb-4b26-447a-bd87-87a64b50d913', created_at=datetime.datetime(2024, 12, 30, 17, 57, 31, 241455, tzinfo=TzInfo(UTC)), created_by='user', base_model_id='my-first-finetuned-model', steps=10, depth=3, loss='default', model='timegpt-1-long-horizon', freq='MS'),
 FinetunedModel(id='my-first-finetuned-model', created_at=datetime.datetime(2024, 12, 30, 17, 57, 16, 978907, tzinfo=TzInfo(UTC)), created_by='user', base_model_id='None', steps=10, depth=1, loss='default', model='timegpt-1-long-horizon', freq='MS')]

While that representation may be useful for programmatic use, in this exploratory setting it’s nicer to see them as a dataframe, which we can get by providing as_df=True.

nixtla_client.finetuned_models(as_df=True)

	id	created_at	created_by	base_model_id	steps	depth	loss	model	freq
0	468b13fb-4b26-447a-bd87-87a64b50d913	2024-12-30 17:57:31.241455+00:00	user	my-first-finetuned-model	10	3	default	timegpt-1-long-horizon	MS
1	my-first-finetuned-model	2024-12-30 17:57:16.978907+00:00	user	None	10	1	default	timegpt-1-long-horizon	MS

We can seee that the base_model_id of our second model is our first model, along with other metadata.

7. Deleting fine-tuned models

nixtla_client.delete_finetuned_model(first_model_id)

True

We can verify that our first model model doesn’t show up anymore in our available models.

nixtla_client.finetuned_models(as_df=True)

	id	created_at	created_by	base_model_id	steps	depth	loss	model	freq
0	468b13fb-4b26-447a-bd87-87a64b50d913	2024-12-30 17:57:31.241455+00:00	user	my-first-finetuned-model	10	3	default	timegpt-1-long-horizon	MS

On this page

1. Import packages
2. Load data
3. Zero-shot forecast
4. Fine-tune
5. Further fine-tune
6. Listing fine-tuned models
7. Deleting fine-tuned models

​1. Import packages

​2. Load data

​3. Zero-shot forecast

​4. Fine-tune

​5. Further fine-tune

​6. Listing fine-tuned models

​7. Deleting fine-tuned models

​1. Import packages

​2. Load data

​3. Zero-shot forecast

​4. Fine-tune

​5. Further fine-tune

​6. Listing fine-tuned models

​7. Deleting fine-tuned models

1. Import packages

2. Load data

3. Zero-shot forecast

4. Fine-tune

5. Further fine-tune

6. Listing fine-tuned models

7. Deleting fine-tuned models

1. Import packages

2. Load data

3. Zero-shot forecast

4. Fine-tune

5. Further fine-tune

6. Listing fine-tuned models

7. Deleting fine-tuned models