Skip to main content
Accuracy benchmarks ask whether synthetic data helps a model. Coverage asks a narrower question first: do the generator pools produce series whose statistical summaries land near those of real series at all? If a whole region of real behaviour has no synthetic neighbour, no amount of pretraining on that corpus can teach it. This page walks through the frozen twelve-feature schema, the distance-based scoring protocol, the results across the six M4 frequencies, and the one frequency where the pools fall short. Every number on this page is read from a benchmark summary rather than transcribed by hand. Those summaries and the scripts that produce them are not in the repository yet, so this page publishes stored outputs and only the first example runs from a fresh checkout; the scripts are listed at the end.
Summary
  • Twelve native features (native_v1) are defined for every one of the 100,000 M4 training series under declared period and window recipes, and they respond in the expected direction to controlled changes in dependence, seasonality, spikes, level shifts, and variance shifts.
  • M4, Yearly to Daily: the interpretable and pretraining pools cover 52–88% of fresh held-out real queries at the real-calibrated radius, against real-to-real baselines of 80–91%. No M4 panel other than Hourly shows a material gap over three generation seeds and four query groups.
  • M4 Hourly is the largest gap: 1–2% coverage against a 91% baseline, replicated on 288 distinct real queries. The missing combination is a daily cycle with slow level and amplitude changes; trend strength alone carries about half the remaining distance.
  • Independent panels change the picture: on Monash tourism quarterly and Australian weather daily the pools also fall 35 points or more short, and hospital monthly is 27 points short, while tourism yearly and traffic weekly are covered as well as M4. Gaps are panel-specific, with strong regular seasonality the recurring theme.
  • Configuring existing generators closes most of the M4 Hourly gap (67% coverage) but transfers only partly to an independent traffic panel (19%). A moderated seasonal corpus, added to the interpretable pool at equal candidate count, raises coverage on all five quarterly and monthly panels tested, three of them held out, by 3 to 32 points. That corpus ships as the opt-in seasonal_pool() preset; a pre-declared test on four more panels passes on M1 monthly, is neutral on M4, and shows no benefit on intermittent data, so the default pools are unchanged.

​
What is measured

compute_features returns one row of twelve summaries per series: spectral entropy, trend and seasonal strength from a classical decomposition, four autocorrelation summaries, spike and lumpiness, the largest level and variance shift between adjacent windows, and the number of median crossings. The definitions are frozen as native_v1; the seasonal period and window width are extraction parameters that you declare per panel, not part of the schema. The example below extracts features for a small hourly panel and compares two synthetic sources in that space with compare_feature_coverage. It needs no network access, so the reference here is itself synthetic; the benchmarks further down use real M4 and traffic data.
The CoverageResult reports which features were used, how many series each side retained, and grid-based miscoverage in a two-dimensional PCA map fitted on the reference alone. The map is a convenient picture, but two dimensions keep only 54–84% of the feature variance on M4 panels, so the benchmarks score coverage on all standardized features and use the map only for illustration.
How the benchmarks score coverage
  1. Fit the feature mean and scale on 64 real reference series only. Synthetic data never shifts the space.
  2. For each held-out real query, find its nearest synthetic series and its nearest real candidate in standardized Euclidean distance.
  3. Calibrate a radius as the 90th percentile of nearest distances from 32 separate real calibration queries to the real candidates.
  4. Coverage is the fraction of queries whose nearest synthetic series lies within that radius. The real baseline is the same fraction using real candidates, and bounds what any corpus of that size can reach.
  5. Synthetic and real candidates are matched one-to-one in count and length, so candidate density cannot favour either side.
Queries are always disjoint from the fitting and calibration series, criteria are declared before results are inspected, and repeated settings use fresh queries rather than the ones that informed a design. Coverage is conditional on the twelve features, the distance, and the radius; Manhattan distance and single-feature omissions are reported as sensitivity checks.

​
Coverage across frequencies

The cross-frequency benchmark samples 640 training series per M4 frequency (all 359 for Weekly), forms four disjoint groups of 64 fresh queries, and scores the unchanged interpretable_pool (stored under the key balanced_pool in the benchmark artifacts, its name at the time), pretraining_pool, and a random MAR corpus over three generation seeds with 240 matched candidates each. Hourly comes from the earlier fresh-split validation, which used the same protocol.
Each dot is the median over four query groups and three generation seeds; the bar behind it spans the twelve fold–seed pairs. Three things stand out.
  • Yearly and Quarterly are essentially covered. The pools sit within a few points of what real candidates achieve. Monthly falls about 5 points short and Weekly about 20, the latter flagged as a minor gap. Daily is the widest of the three, roughly 25 to 30 points below the real-to-real baseline, and depends strongly on which real series are queried (two folds gap by more than 25 points, two by less than 10).
  • The two pools are interchangeable here. They stay within five points of each other on every panel, so the meta-generators added by the pretraining pool do not buy coverage in this feature space. Random MAR is far behind on every panel, Quarterly included, where it reaches 34% against 88% for the pools.
  • Hourly is a different regime. Coverage is 1–2% against a 91% baseline, and the twelve fold–seed pairs never exceed 16%. The rest of this page is about that gap.

​
What covered queries look like

Coverage numbers summarize distances between feature vectors, so it is worth checking that a covered query actually resembles its nearest synthetic neighbour. The examples below are real M4 series from the first fold and first seed of the benchmark above, chosen at fixed rank percentiles (10th, 50th, 90th) of their distance to the interpretable pool rather than for visual effect, and shown next to their nearest neighbour from each pool. Every series is standardized so that shape, not scale, is compared; the label gives the generator prototype and the standardized feature distance.
The close and median queries resemble their neighbours: gradual or stepwise level movements are matched by fractional Brownian motion and Lévy-process series, which is why those two prototypes dominate the nearest-neighbour counts on Monthly and Daily. Neighbours are matched on the twelve features, not on the path itself, so lengths differ and the comparison is one of texture and level movement rather than exact shape. The 90th-percentile queries show where the pools thin out. The Monthly query has irregular seasonal swings on a rising level; its nearest neighbours offer either the rise without the swings or the swings without the rise. The Daily query is shorter and drops sharply in level midway; its neighbours are rough random walks without such a break. These are the queries that the real baseline covers and the pools do not.

​
Independent panels

M4 is one source, so each non-hourly frequency was also scored on a public Monash panel from another domain with the identical protocol: tourism demand (yearly and quarterly), hospital patient counts (monthly), San Francisco freeway occupancy (weekly), and Australian weather stations (daily). Hourly’s independent panel is the traffic dataset used in the composition rounds below.
Filled dots are the pools on the independent panel; the hollow blue circle is the interpretable pool’s M4 result at the same frequency. Tourism yearly and traffic weekly are covered as well as, or better than, M4. Tourism quarterly is the sharpest change: the interpretable pool falls from 88% on M4 Quarterly to 47%, with every fold–seed pair between 34% and 59% while real candidates still cover 91%. Weather daily sits 34–38 points below its baseline and hospital monthly 27 points below. Both tourism quarterly and hospital monthly are flagged by the declared gates because the real series are more strongly seasonal than any pool neighbour, by 0.2–0.6 standardized units of seasonal strength; on weather the differences are first-difference autocorrelation and level shifts across a panel that mixes rain, temperature, and solar radiation. Coverage is therefore panel-specific rather than frequency-specific. Within M4 only Hourly is short, but across domains, strongly seasonal quarterly and monthly panels and heterogeneous daily weather are also under-covered, and the recurring theme is strong regular seasonality combined with level movement, the same combination the Hourly chain below identifies.
The tourism series show what the seasonal-strength attribution means: a regular four-quarter cycle whose amplitude grows with a rising level. Even the closest pool neighbour, a regime-switching series with a level step, has no regular cycle, and the median and far queries are matched to sensor and vital-sign presets that only share the upward drift and the noise level. The pools contain seasonal generators, but none of the 42 prototypes produces multiplicative period-4 seasonality of this strength, so the nearest series are seasonal in the features’ terms without looking seasonal. The weather panel fails for different reasons. The two temperature-like queries carry a slow annual swing under day-to-day noise; the neighbours, a fractional Brownian motion and a regime-switching series, reproduce the roughness but not the smooth swing, and the declared weekly recipe cannot represent a cycle longer than the 512-day window. The far query is rainfall: long dry runs broken by isolated spikes, which the nearest GARCH and TCM series only resemble in being spiky. These are the two behaviours behind the Daily flag, intermittent rain and slow annual cycles, and neither is a matter of a single missing component.

​
The Hourly gap

The Hourly result was replicated on fixed queries with ten fresh generation seeds and four times more candidates, then on 256 additional queries in four disjoint groups. Neither more draws nor more candidates closed it. Decomposing the nearest-neighbour distances shows that trend strength contributes about half of the mean squared distance: real hourly series have far stronger trend components than their nearest synthetic neighbours, whose daily cycles sit on a flat level. Omitting trend strength from the distance raises pool coverage to about 44%, so the gap is not only trend, but trend is where it starts. To test that reading, paired diagnostic controls were generated from a unit daily sine plus noise, adding one component at a time: a slow level drift, a weekly amplitude envelope, and a second daily harmonic. The controls use no real values and share their random components within each slot, so each row below differs from the previous one by exactly one ingredient.
Adding slow level drift to a plain daily cycle lifts full-feature coverage from 2% to 57%, and it does so in all twelve paired fold–seed comparisons. Amplitude variation alone changes little, but combined with drift it adds a few more points, and the second harmonic helps the separate daily-profile shape score. The existing pools, in grey, sit where the plain cycle does: they contain smooth daily cycles but not the combination of a daily cycle with a moving level. Real hourly series, by contrast, drift, shift, and change amplitude over a three-week window.
For all three real queries, the nearest interpretable-pool neighbour is the same periodic Gaussian-process prototype: a regular sine on a flat level. The real series drift, change amplitude from week to week, and in the 90th-percentile query have sharp, non-sinusoidal daily peaks. The first-round composition supplies daily cycles on a moving level, mostly from the daily-plus-weekly TSIGenerator prototype, and its neighbours share the real series’ slow variation; the targeted round’s noisier variant picks up the jagged query. The standardized feature distances fall from 2.8–5.4 for the pool to 1.0–2.4 for the compositions. The daily shapes remain smoother than the real ones, which is the daily-profile score the compositions did not improve.

​
Can existing generators close it?

The controls are diagnostic recipes, not generators. The next question was whether the shipped classes can express the same combination through configuration alone. A benchmark-only corpus of eight prototypes was declared before inspection: TSIGenerator restricted to a period-24 harmonic with overtones, a slow amplitude envelope, and a drifting trend; ETSGenerator with multiplicative period-24 seasonality on a random-walk level; SeasonalGenerator with injected slope and level changepoints; and EnergyLoadGenerator profiles with a slope change. It was scored on the same fresh M4 splits and then on 862 hourly traffic series from the Monash repository, a panel that had not informed the design. A second, targeted round then added noisier and level-shifting variants and was scored on traffic queries never used before, with M4 kept as a retention check.
On M4 the configured corpus reproduces the control result: 67% coverage against 1–2% for the pools, within a point of the best control, and higher than the best pool in all twelve paired comparisons. The two TSIGenerator prototypes supply three quarters of the nearest neighbours; the ETSGenerator prototypes are never nearest. It does not reproduce the controls’ improvement in average daily shape. On traffic the same corpus improves coverage in every paired comparison but only to about 10%, and the targeted second round roughly doubles that to 19% on queries never used before, while keeping the M4 result. Both rounds fall short of the declared threshold of a 25-point median gain over the best pool in three of four folds. The remaining traffic distance concerns crossing points and ten-lag autocorrelation: real traffic series combine low within-day autocorrelation with long excursions on one side of the mean, which none of the configured generators produce together.
Traffic occupancy has flat, low nights and sharp commuting peaks, so its daily shape is far from a sine, and weekend days show lower peaks. The pool neighbour is again a flat sine. The composition neighbours reproduce level changes and amplitude envelopes but not the asymmetric peak-and-floor shape; the noisy variant gets closer to the median query by adding irregularity rather than the right shape. The 90th-percentile query alternates between quiet weeks near its floor and weeks of tall spikes, and no source approaches it. This is what the remaining crossing-point and autocorrelation differences look like in the raw series: long stretches on one side of the mean, punctuated by brief excursions.

​
Can configuration close the seasonal gaps?

The same approach was applied to the seasonal theme. Two benchmark-only corpora of configured TSIGenerator, ETSGenerator, and SeasonalGenerator instances at the panel period were scored on the two panels that motivated them, tourism quarterly and hospital monthly, and then on three panels that did not: M3 quarterly, M3 monthly, and tourism monthly. The first recipe was declared before any result and aimed at strong seasonality; it turned out far too clean, with seasonal strength near one and almost no noise, while the real series sit at moderate seasonal strength under heavy noise. The second, moderated recipe was declared after that diagnosis on the design panels only, and nothing was changed once the validation panels were scored.
Scored as a replacement for the pools, the moderated corpus fails the declared 25-point criterion on every validation panel: it beats the pools on the two tourism panels but, being narrow, covers far fewer queries than they do on the broad M3 panels. That criterion answers the wrong question for a corpus meant to fill a gap. The question a preset change poses is whether adding the corpus helps, so the chart compares the interpretable pool plus the moderated corpus with the interpretable pool plus the pretraining pool, both at 480 candidates and the same real-calibrated radius. The addition is ahead on all five panels, by 3 points where the pools were already near the baseline and by about 30 points on the tourism panels, and higher in 53 of 60 fold–seed pairs. This comparison was specified after the replacement results were known, so it is a post-hoc analysis and is labelled as such in the benchmark report; the three validation panels were nevertheless never used to tune the recipe. Tourism monthly, one of the held-out panels, is itself a new finding: the pools cover 11% of its queries against an 86% real baseline. The examples below show why, and what the moderated corpus adds.
Real tourism series carry a strong twelve-month cycle whose amplitude grows with the level, on a trend with occasional level shifts. The pools’ nearest neighbours are Holt-Winters ETS series with a much weaker cycle, or physiological and load presets that only share the noise texture. The moderated corpus supplies noisy Holt-Winters and sine-with-breaks series with a visible cycle, drawn roughly equally from its additive, multiplicative, and level-break prototypes, and those become the nearest neighbours. Its distances are still not small, and seasonal strength remains the largest remaining difference: the corpus narrows the gap rather than closing it.

​
The shipped preset and its pre-declared test

The moderated corpus now ships as the opt-in seasonal_pool() preset, with exactly the configuration benchmarked above. Because the augmented comparison had been chosen after the fact, it was declared as the criterion before testing the preset on four panels never used in any earlier step: fresh M4 Quarterly and Monthly samples, M1 monthly, and intermittent car-parts sales as a no-harm check. The gates were a median paired gain of at least 5 points in three of four folds, and no fold below −2 points, for interpretable + preset against interpretable + pretraining at 480 candidates each.
The verdict is mixed and is reported as such. M1 monthly, where the pools sit 24 points below the real baseline, passes both gates with a 12.5-point paired gain and is the only clear pass. The two fresh M4 samples have no seasonal gap to fill and come out neutral, though M4 Quarterly misses the no-harm gate by one point in one fold. Car parts fails no-harm: its gap is an intermittent-demand gap that the seasonal preset does not address, while the pretraining pool’s meta-generators help a little, so displacing their share costs 3 points. Added on top of both pools rather than in place of a share, a post-hoc comparison against adding random MAR shows the preset helps where seasonality is strong and is within a point either way elsewhere, including −0.8 points on car parts. That is how the preset is documented: an opt-in addition on top of a pool, kept out of the defaults. A last pre-declared test asked whether the addition should become the pretraining_pool default, through the new include_seasonal flag, by comparing the pool with and without it as equal-count corpora on fifteen panels at every frequency but yearly. The flag raised coverage on every quarterly and monthly panel, by 2 to 14 points, and lowered it on weekly, daily, and intermittent panels, by up to about 6 points on M4 weekly and daily and about 3 on car parts, while traffic weekly and weather daily were essentially unchanged; it builds a cycle those series do not have and displaces breadth. Six panels failed the no-harm gate, so the default stays off, and the flag is recommended for quarterly and monthly targets only. The table below is from that run.

​
What this means for the presets

Recommendation
  • Keep the production pools as they are for now. The two pools are interchangeable for coverage, and no single configuration has yet been shown to close a gap on a panel that did not inform it, so there is no evidence-backed change to interpretable_pool or pretraining_pool defaults.
  • The gap is about strong regular seasonality on a moving level, not one frequency. Within M4 only Hourly is short, but tourism quarterly and monthly, hospital monthly, and weather daily are also under-covered on independent panels.
  • Ship gap-filling corpora as opt-in additions, not replacements. seasonal_pool() is now available for that purpose: it raised coverage on tourism, hospital, and M1 monthly panels, was neutral on M4, and gave nothing on intermittent data, so it belongs on top of a pool and outside the defaults. A comparable hourly preset would need the same pre-declared test on a third hourly panel first.
  • Evaluate on held-out panels, not M4 alone. M4 Quarterly looked covered at 88% while tourism quarterly is at 47%; a benchmark on one source can miss a gap that another source shows.
What coverage does not tell you
  • Coverage is conditional on the twelve native_v1 features, standardized Euclidean distance, a q90 radius, and the declared period and window per panel. Manhattan distance and feature omissions shift absolute levels by up to ten points, and by far more on Hourly.
  • Fold–seed measurements share real series across roles and are not independent replications; there are no confidence intervals in these summaries.
  • A covered query has a synthetic neighbour with similar summaries, not a similar series. Nothing here measures forecasting accuracy, raw-series distance, or privacy.

​
Reproduce

The benchmark scripts and their artifacts are not yet in the repository; they land in a follow-up pull request that keeps this one focused on the library change. The commands below are recorded now so the protocol is on the record, and the numbers on this page come from a local run of exactly these steps. Once the scripts land they run in sequence, each reading the previous one’s summary.json, with detailed per-query traces written alongside and a README per directory documenting protocol, results, and limits.
Summaries are written under benchmarks/data/feature_coverage_*/, which is untracked. The API used above, compute_features and compare_feature_coverage, is documented in the evaluation reference.