Skip to content
risshabh.com

Memory-constrained forecasting

Measured

19% lower tail-forecast error at equal memory

Same bytes. At the best size, 2 learned summaries beat 14× more raw history (28 past values).

Same memory, two ways to spend it

Slide the memory size. Both forecasters always get exactly the same bytes.

Raw history

28 past values

Learned summaries

2 slots of 14 numbers

Tail-forecast error, lower is better

Raw history0.01174
Learned summaries0.00949

19% lower, p < 0.001

75.8 KB: 2 learned slots against 28 raw values. Tail loss 0.01174 against 0.00949, 19% lower, p < 0.001.

Does it hold at every size?

11 of 12 comparisons are significant on their own, and the learned side is ahead in all 12. The exception is recency at the largest size (p = 0.205). Tap a size to move the slider.

How much lower the learned summaries’ tail error is than raw history, for each memory size and keep-policy. A filled dot marks p below 0.05.
Recency13% lower, significant, p < 0.00119% lower, significant, p < 0.00118% lower, significant, p < 0.00114% lower, significant, p < 0.0019% lower, significant, p = 0.0115% lower, not significant, p = 0.205
Tail oracle40% lower, significant, p = 0.00145% lower, significant, p < 0.00146% lower, significant, p < 0.00145% lower, significant, p < 0.00143% lower, significant, p < 0.00134% lower, significant, p = 0.002

Cells show the cut in q99 pinball loss. Compare the two sides within a cell, not across rows: the tail oracle’s losses are inflated by a readout mismatch. One market, one year of real electricity prices (499 nodes, 35,035 intervals, 3,504 held-out queries); other markets are not tested.

Exact numbers for all six sizes
Measured q99 pinball loss and Diebold–Mariano p-values per memory size and keep-policy
Learned / raw slotsBytesRecency rawRecency learnedRecency pTail rawTail learnedTail p
1 / 1461,8760.011610.010090.00002830.075520.045010.00142
2 / 2875,8480.011740.0094867.66e-100.077630.042690.000364
3 / 4289,8200.011730.0096493.57e-80.07760.042150.000289
4 / 56103,7920.011750.01010.00003650.076230.041610.000326
5 / 70117,7640.011760.010680.01120.072670.041110.000535
6 / 84131,7360.011570.011010.2050.061870.040590.00153

How this works

What was compared

Two forecasters get exactly the same bytes. One spends them on a few 14-number summary slots: a node’s latest price plus a summary of what the whole grid is doing. The other spends them on 14 times as many slots of that node’s own recent prices. The slider moves both through six memory sizes, from about 62 KB to 132 KB.

How it was tested

The comparison was written down before the test data was scored. Each memory size gets its own Diebold–Mariano test, corrected for correlated errors (HAC variance, lag 24), under two keep-policies: recency and a tail oracle. That is 12 comparisons. 11 of the 12 are significant at p < 0.05 on their own; recency at the largest size is not (p = 0.205). The learned side is ahead in all 12.

What the numbers mean

Price spikes are rare and carry the risk, so forecasters are scored on the 99th-percentile tail (q99 pinball loss), not the average. The big 19% is the cut at the best size for the recency policy: 0.01174 for raw history against 0.00949 for learned summaries, at 2 slots against 28. Every number on the page is measured on the held-out test split; nothing is illustrative. The tail oracle’s losses are inflated by a readout mismatch, so compare its two sides and do not read its levels.

Where it does not hold

One market and one year: 499 nodes of real electricity prices, 35,035 intervals, 3,504 held-out queries. Other markets are not tested. A learned memory keeper did not beat a simple baseline at any of three scales, and an earlier claim that streaming scenario selection would beat batch selection was not supported. Both negative results are kept.

Measured. The numbers on this page come from real runs and are shown as measured.