> ## Documentation Index > Fetch the complete documentation index at: https://nixtlaverse.nixtla.io/llms.txt > Use this file to discover all available pages before exploring further. # Pooled lag transforms > Compute lag features across all series, or within groups of series, > using SQL-style RANGE semantics. Most lag transforms in `mlforecast` are computed **independently per series**: a rolling mean for series `A` only ever sees `A`’s own history. That works well when every series carries enough signal on its own, but it breaks down in two common situations: * **Short-history series.** A newly launched product, store, or SKU has too little history for per-series rolling statistics to stabilise. * **Cross-series signals.** Total demand across a brand, region, or category at a given timestamp is often a stronger feature than any one series’ lag. **Pooled lag transforms** address both. By passing `global_=True` (across all series) or `groupby=["col", ...]` (within a static-feature group) to any rolling, expanding, seasonal, or exponentially weighted transform, you ask `mlforecast` to compute the statistic over a **bucket of series** aggregated by timestamp. The result is a single value per timestamp per bucket that every series in the bucket then receives as a feature. ## Data setup ```python theme={null} import warnings import numpy as np import pandas as pd from mlforecast import MLForecast from mlforecast.lag_transforms import ( ExpandingMean, ExponentiallyWeightedMean, RollingMean, ) from mlforecast.utils import generate_daily_series ``` We’ll start from a small synthetic panel and attach a static `brand` column so we have something to group by. ```python theme={null} series = generate_daily_series( n_series=6, min_length=60, max_length=60, equal_ends=True, static_as_categorical=False, seed=0, ) brand_map = {f"id_{i}": "A" if i < 3 else "B" for i in range(6)} series["brand"] = series["unique_id"].map(brand_map) series.head() ``` ## Global features with `global_=True` Set `global_=True` on any built-in lag transform to compute the statistic **across all series**, aggregated by timestamp. Every series receives the same value at a given timestamp. ```python theme={null} fcst = MLForecast( models=[], freq="D", lag_transforms={ 1: [RollingMean(window_size=7, global_=True)], }, ) prep = fcst.preprocess(series, static_features=["brand"]) prep[prep["ds"] == prep["ds"].min()].head(6) ``` Notice that `global_rolling_mean_lag1_window_size7` is identical for every `unique_id` at a given `ds`: it’s the rolling mean of the **pooled** observations across all series, lagged by one day. The feature name is automatically prefixed with `global_` to make the pooling explicit. ## Group features with `groupby=[...]` Use `groupby` to compute the statistic **within each level of one or more static features**. Series in the same group share the feature value at each timestamp; series in different groups get different values. Any column used in `groupby` must be declared as a static feature when fitting. ```python theme={null} fcst = MLForecast( models=[], freq="D", lag_transforms={ 1: [RollingMean(window_size=7, groupby=["brand"])], }, ) prep = fcst.preprocess(series, static_features=["brand"]) prep[prep["ds"] == prep["ds"].min()].sort_values(["brand", "unique_id"]).head(6) ``` Series within brand `A` share one rolling-mean value; series within brand `B` share another. The column name is prefixed with `groupby_brand_` to record which static feature drove the pooling. ## RANGE semantics: staggered series and gaps Pooled transforms use **`RANGE BETWEEN ... PRECEDING`** semantics, the same model SQL window functions use. The window is defined by **timestamp distance**, not row position, and only **actual observations** are aggregated — no synthetic zeros are injected for series that haven’t started yet. Concretely, with `RollingMean(window_size=2, global_=True)` over the data below: | `unique_id` | `ds` | `y` | | ----------- | ---- | ---- | | a | 1 | 1.0 | | a | 2 | 2.0 | | a | 3 | 3.0 | | b | 2 | 20.0 | | b | 3 | 30.0 | Series `b` starts at `ds=2`; it does **not** contribute a phantom zero at `ds=1`. At each timestamp the window looks back over the last two days of *real* values across all series, applied at lag 1. ```python theme={null} staggered = pd.DataFrame( { "unique_id": ["a", "a", "a", "b", "b"], "ds": pd.to_datetime( ["2024-01-01", "2024-01-02", "2024-01-03", "2024-01-02", "2024-01-03"] ), "y": [1.0, 2.0, 3.0, 20.0, 30.0], } ) fcst = MLForecast( models=[], freq="D", lag_transforms={1: [RollingMean(window_size=2, global_=True, min_samples=1)]}, ) fcst.preprocess(staggered, dropna=False) ``` Because pooled transforms assume a **continuous, gap-free time grid** within each series, you should validate your data (the default) before relying on these features. We come back to this in the `validate_data` section below. ## `min_samples` in pooled mode There’s an important semantic divergence between local and pooled modes: * **Local mode** (per series): `min_samples` is capped at `window_size` by `coreforecast`. It controls how many *non-NaN values in the window* the series needs. * **Pooled mode** (`global_=True` or `groupby=...`): `min_samples` counts the **total non-NaN observations across all series in the bucket**, with no capping at `window_size`. This makes it useful as a *coverage threshold*. For example, `RollingMean(window_size=1, min_samples=2, groupby=['brand'])` only produces a value when at least two series in the brand contributed an observation in the window — useful to suppress noise from sparsely populated groups. The default when `min_samples=None` also differs by mode. Everywhere else it defaults to `window_size`, but in **local partition mode** (`partition_by` without `global_`/`groupby`, covered below) it defaults to **1**: the window spans `window_size` calendar steps while only same-partition observations count toward `min_samples`, so requiring a full window is rarely attainable — `RollingMean(7, partition_by=['promo'])` on a panel with interleaved promo days would otherwise be almost entirely NaN. The default of 1 matches SQL `RANGE` window semantics, where the result is NULL only for empty windows. When `partition_by` is combined with `global_` or `groupby`, the default remains `window_size`, counted across all series in the (group, partition) bucket — pass `min_samples` explicitly if you want a different threshold. ```python theme={null} fcst = MLForecast( models=[], freq="D", lag_transforms={ 1: [RollingMean(window_size=1, min_samples=2, groupby=["brand"])], }, ) prep = fcst.preprocess(series, static_features=["brand"], dropna=False) prep[ [ "unique_id", "ds", "brand", "groupby_brand_rolling_mean_lag1_window_size1_min_samples2", ] ].head(6) ``` ## Aggregating rows per timestamp with `time_agg` By default a pooled transform treats every observation as an individual sample: `RollingMean(window_size=7, groupby=['brand'])` pools *all rows* of the brand that fall in the window and averages them, weighting a timestamp by how many series reported on it. Sometimes you instead want to first collapse all rows sharing a timestamp into a single value — for example the brand’s **daily total** — and then apply the transform over that per-timestamp series. `time_agg` does exactly that. Set it to one of `'sum'`, `'count'`, `'mean'`, `'min'`, or `'max'` and each `(bucket, timestamp)` is reduced to one value before the rolling/expanding/seasonal/EWM statistic runs: * `time_agg='sum'` — rolling mean of the brand’s daily *sums* (total demand) * `time_agg='count'` — e.g. expanding max of the number of active series per day * `time_agg='max'` — rolling max of the daily *peak* across the brand `time_agg` requires `global_=True` or `groupby=[...]` (optionally combined with `partition_by`). It is rejected in local or `partition_by`-only mode, where each `(bucket, timestamp)` already has a single row and the aggregation would be a no-op. With `time_agg=None`, each observation remains an individual pooled sample. `ExponentiallyWeightedMean` is the exception: its `time_agg` defaults to `'mean'`, and `None` is not accepted. ```python theme={null} fcst = MLForecast( models=[], freq="D", lag_transforms={ 1: [RollingMean(window_size=3, groupby=["brand"], time_agg="sum")], }, ) prep = fcst.preprocess(series, static_features=["brand"], dropna=False) prep[ [ "unique_id", "ds", "brand", "groupby_brand_rolling_mean_lag1_window_size3_time_aggsum", ] ].head(6) ``` Two semantics worth calling out: * With `time_agg`, `min_samples` counts **observed timestamps** in the window (each timestamp contributes at most one aggregated value), not rows. `RollingMean(window_size=3, min_samples=3, groupby=['brand'], time_agg='sum')` needs three distinct timestamps of history, regardless of how many series reported on each. * `RollingMean(..., time_agg='mean')` differs from `RollingMean(...)` without `time_agg`: the former is an unweighted mean of the per-timestamp means, the latter a row-weighted pooled mean (they coincide only when every timestamp has the same number of observations). On `ExponentiallyWeightedMean`, `time_agg='mean'` is the default because pooled EWM consumes each timestamp’s bucket-aggregate mean exactly once. ## Disabling validation: the `validate_data=False` warning Pooled transforms are correctness-sensitive to timestamp gaps because they rely on the *actual* timestamps when computing RANGE windows. If you bypass validation (`validate_data=False`) and your data has gaps, the features will be silently wrong. To make this hazard explicit, `mlforecast` emits a `UserWarning` when you combine `validate_data=False` with any pooled transform: ```python theme={null} fcst = MLForecast( models=[], freq="D", lag_transforms={1: [RollingMean(window_size=7, global_=True)]}, ) with warnings.catch_warnings(record=True) as caught: warnings.simplefilter("always") fcst.preprocess(series, static_features=["brand"], validate_data=False) [str(w.message) for w in caught if issubclass(w.category, UserWarning)] ``` If you genuinely need to skip validation (e.g. on a very large pre-cleaned dataset), make sure your time grid is continuous and gap-free before doing so. ## Pooling with `partition_by` `partition_by=[col, ...]` splits a pooled bucket further along a **dynamic** column — typically a feature whose value changes over time (e.g. `promo`, `regime`, `store_format` flips). Each unique combination of partition values gets its own bucket; rolling/expanding statistics are computed per bucket. Unlike `groupby`, `partition_by` columns **do not need to be static**. They can vary across timestamps for the same `unique_id`. At prediction time you must supply their future values via `X_df`. Three combinations are supported: | Combination | Bucket key | | ----------------------------------- | --------------------------------------------------------------------------------------------------------- | | `partition_by=[...]` alone | `(unique_id, *partition_values)` — *local partition*, one bucket per series-partition pair | | `global_=True, partition_by=[...]` | `(*partition_values)` — *nonlocal*, aggregates across all series sharing the same partition values | | `groupby=[...], partition_by=[...]` | `(*group_values, *partition_values)` — *nonlocal*, aggregates within each group, separately per partition | Unlike `global_` and `groupby` (which are mutually exclusive), `partition_by` composes with either one — or stands alone. Let’s add a time-varying `promo` column to our panel and walk through the three combinations. ```python theme={null} rng = np.random.default_rng(0) series["promo"] = rng.choice([0, 1], size=len(series), p=[0.7, 0.3]) series.head() ``` ### Local partition: per-(id, partition\_vals) buckets When `partition_by` is used alone (no `global_`, no `groupby`), each `(unique_id, partition_value)` combination becomes its own bucket. The rolling window is computed within that bucket only — so a series sees a different rolling history depending on which `promo` regime it is currently in. ```python theme={null} fcst = MLForecast( models=[], freq="D", lag_transforms={1: [RollingMean(window_size=7, partition_by=["promo"])]}, ) fcst.preprocess(series).head(8) ``` The feature name is prefixed `partby_promo_` to indicate the partition. Each row’s feature is the 7-day rolling mean of `y` lagged by 1, restricted to rows of the same series with the same `promo` value. ### Global + `partition_by`: cross-series aggregates within each partition Combine with `global_=True` to aggregate across all series, but separately for each `promo` value: ```python theme={null} fcst = MLForecast( models=[], freq="D", lag_transforms={ 1: [RollingMean(window_size=7, global_=True, partition_by=["promo"])] }, ) fcst.preprocess(series).head(8) ``` Every series with `promo=0` at a given timestamp sees the same `global_partby_promo_...` value; every series with `promo=1` sees a different (also shared) value. ### Groupby + `partition_by`: group aggregates within each partition Combine with `groupby=[...]` to aggregate per group (e.g. per `brand`), separately for each partition value: ```python theme={null} fcst = MLForecast( models=[], freq="D", lag_transforms={ 1: [RollingMean(window_size=7, groupby=["brand"], partition_by=["promo"])] }, ) fcst.preprocess(series, static_features=["brand"]).head(8) ``` The bucket key here is `(brand, promo)`. There are at most `n_brands × n_promo_values` buckets. ## Dynamic partition values at prediction time Because partition columns are *not* static, you must supply their future values when forecasting. Pass an `X_df` containing `unique_id`, `ds`, and every partition column for every horizon step: ```python theme={null} from sklearn.linear_model import LinearRegression fcst = MLForecast( models=[LinearRegression()], freq="D", lag_transforms={ 1: [RollingMean(window_size=7, global_=True, partition_by=["promo"])] }, ) # `brand` plays no role in this global_ + partition_by example, so we drop it # rather than feed a raw string column to LinearRegression fcst.fit(series.drop(columns="brand")) last_date = series["ds"].max() unique_ids = sorted(series["unique_id"].unique()) future_promo = pd.DataFrame( [ { "unique_id": uid, "ds": last_date + pd.Timedelta(days=h), "promo": int(rng.integers(0, 2)), } for uid in unique_ids for h in range(1, 4) ] ) fcst.predict(h=3, X_df=future_promo).head() ``` A few notes on dynamic prediction: * If `X_df` introduces a partition value never seen during fit, mlforecast creates a fresh bucket on the fly. The first few predictions for that bucket may be `NaN` if its history is shorter than `window_size`/`min_samples`. * Subset prediction (`predict(ids=[...])`) is rejected when nonlocal `partition_by` is in play: the missing series would still contribute to the bucket aggregates, so partial subsets are ambiguous. * Partition columns are **automatically excluded from `static_features`** even when `static_features=None`. They must come from `X_df` at predict time. ## Understanding ordinals and parent calendars A `partition_by` bucket only contains the rows where the partition value is active for a series. When the partition is sparse — e.g. `promo=1` is only on at timestamps `[1, 3, 5]` while the parent calendar (the global or group scope) is `[1, 2, 3, 4, 5]` — the bucket’s *ordinals* come from the parent calendar: * Bucket has observations at parent positions `[0, 2, 4]`, not `[0, 1, 2]`. * `RollingMean(window_size=2)` at parent position 4 looks back to positions `[3, 4]`. The bucket has no observation at position 3, so only position 4 contributes to the window. This preserves SQL `RANGE BETWEEN ...` semantics across gaps: the window is defined in *time*, not in *number of rows*. Without parent-calendar ordinals, a partition with gaps would silently collapse to row-based semantics, mixing observations across non-adjacent timestamps. ## Aligned-end requirement All pooled modes — `global_`, `groupby`, and `partition_by` (including local, alone) — require every series to **end at the same timestamp**. This is checked at `fit` and `predict` time and raises if violated: ```text theme={null} ValueError: Pooled lag transforms require all series to end at the same timestamp (recursive prediction advances all series in lockstep). ``` Local `partition_by` shares this requirement because parent calendars (per-id, for local mode) must advance in lockstep during recursive prediction. If your series have ragged ends, pre-align them (pad to a common last timestamp, or drop series that end early) before fitting. ## Exponentially Weighted Mean with partitions `ExponentiallyWeightedMean` with `partition_by` has subtly different semantics from a regular EWM: * The decay is applied across the bucket’s **observed timestamp aggregates** (one contribution per timestamp, regardless of how many rows the aggregate covered). * Timestamps where the partition bucket has **no observation** are skipped — they do not contribute to the running EWM and do not advance the decay step. This means EWM treats each observed timestamp uniformly and ignores gaps. mlforecast emits a `UserWarning` when you construct an `ExponentiallyWeightedMean` with `partition_by` to make this explicit. ```python theme={null} with warnings.catch_warnings(record=True) as caught: warnings.simplefilter("always") tfm = ExponentiallyWeightedMean(alpha=0.3, partition_by=["promo"]) [str(w.message) for w in caught if issubclass(w.category, UserWarning)] ``` ## `keep_last_n` and pooled history `keep_last_n` controls how much history is carried into the recursive prediction loop. It applies to pooled transforms too, **per state** (one state per `(mode, groupby, partition_by)` combination): * A state whose transforms are **all finite-window** — `Lag`, any `Rolling*` or `SeasonalRolling*`, and `Offset`/`Combine` built from them — is trimmed to its last `max(keep_last_n, W_state)` parent-calendar ordinals, where `W_state` is the state’s widest window. The dropped prefix can never enter a window, so predictions are unchanged. * A state containing any **unbounded** transform — `Expanding*` or `ExponentiallyWeightedMean` — keeps its **full** history. Unlike the local (coreforecast) path, pooled expanding/EWM carry no running accumulator: they recompute over the entire aggregate vectors at every step, so trimming them would change predictions. This applies whether `keep_last_n` is set explicitly or inferred — when you leave it unset it is inferred as the largest window across all transforms — mirroring how the per-series arrays used by local transforms are trimmed. ### Divergence from local transforms Pooled trimming is **floored at the state’s widest window** (`W_state`). A local rolling transform survives an explicit `keep_last_n` *smaller* than its window because coreforecast keeps a separate per-transform window buffer. Pooled mode has no such buffer — the per-timestamp aggregates *are* the buffer — so a `keep_last_n` below a pooled window is raised to that window for the affected state, keeping predictions correct. When `keep_last_n` is left to be inferred it already equals the largest window, so the floor never changes anything. ## Occurrence lookups with `LookupLag` `LookupLag` returns the target from the **previous matching occurrence** within each `(unique_id, partition_by...)` bucket, regardless of the calendar gap between occurrences. This is useful for moving events — e.g. “last year’s Easter value” — where the relevant previous target is not a fixed calendar lag. `partition_by` is required and defines the buckets. Because it is a pooled transform, the partition columns may change over time and must be supplied via `X_df` at prediction. Since the lookup is computed inside `mlforecast`, it stays aligned with any target transforms (e.g. `LocalStandardScaler`). ```python theme={null} from mlforecast.lag_transforms import LookupLag fcst = MLForecast( models=[], freq='D', lag_transforms={1: [LookupLag(partition_by=['promo'])]}, ) prep = fcst.preprocess(series, static_features=['brand']) prep.head() ``` ## Constraints to keep in mind * `global_=True` and `groupby=[...]` are **mutually exclusive** on the same transform. Use two separate transforms if you want both feature families. `partition_by` composes with either one, or stands alone. * Columns in `groupby` must be declared in `static_features` when you call `fit` / `preprocess`. Columns in `partition_by` must **not** be — they are dynamic and supplied via `X_df` at prediction. * `min_samples=0` in pooled mode produces NaN for timestamps with no observations in the window and triggers a warning at construction time — prefer `min_samples=1` if you want “as soon as there is any observation”. * All pooled modes — `global_`, `groupby`, and `partition_by` (including local, alone) — require **series to end at the same timestamp**. * Subset prediction (`predict(ids=[...])`) is rejected when nonlocal pooled transforms are in play: omitted series would still contribute to the bucket aggregates. * All supported rolling, expanding, seasonal-rolling, and exponentially weighted transforms accept `global_`, `groupby`, and `partition_by`. `Offset` and `Combine` delegate to their wrapped transforms. * **Performance**: `RollingQuantile`, `ExpandingQuantile` and the `SeasonalRolling*` transforms have no aggregate-cache fast path in pooled modes. They fall back to a row-level pass whose cost grows with `unique timestamps × bucket rows` at fit, and aggregates are rebuilt at every recursive prediction step. For large panels prefer the mean/std/min/max/EWM transforms, which use cached per-timestamp aggregates. * **Memory**: finite-window pooled states (`Lag`/`Rolling*`/`SeasonalRolling*`) are trimmed under `keep_last_n` just like the per-series arrays (see the [`keep_last_n` and pooled history](#keep_last_n-and-pooled-history) section above); a state containing an `Expanding*`/`ExponentiallyWeightedMean` transform keeps the full training history because it recomputes over all of it at predict. Each `predict` call also backs up the pooled state per model to isolate recursive mutations. Expect higher peak memory with unbounded pooled transforms on large panels. * `time_agg` (`'sum'`/`'count'`/`'mean'`/`'min'`/`'max'`) pre-aggregates rows sharing a timestamp within each bucket before the transform runs; it requires `global_` or `groupby` and counts observed timestamps for `min_samples`. All pooled-capable transforms support it, including the quantile and seasonal ones (via the row-level path). ## End-to-end example Pooled transforms behave like any other lag transform in the full `MLForecast` lifecycle: `fit`, `predict`, and `cross_validation` all carry the pooled features through automatically. ```python theme={null} from sklearn.linear_model import LinearRegression fcst = MLForecast( models=[LinearRegression()], freq="D", lags=[1, 7], lag_transforms={ 1: [ RollingMean(window_size=7, global_=True), RollingMean(window_size=7, groupby=["brand"]), ExpandingMean(groupby=["brand"]), ExponentiallyWeightedMean(alpha=0.3, global_=True), ], }, ) # `promo` was only needed for the partition_by examples above; this end-to-end # model groups by `brand` and doesn't use it, so drop it to keep predict X_df-free fcst.fit(series.drop(columns="promo"), static_features=["brand"]) fcst.predict(h=7).head(6) ``` ## Where to next * The general [Lag transformations](./lag_transforms_guide.html) guide covers the built-in transforms, `Combine`, `Offset`, and custom numba-based transforms. * The [`mlforecast.lag_transforms`](../../lag_transforms.html) API reference documents every parameter, including `global_`, `groupby`, and `min_samples`, for each transform class.