Phase 1 Results — Baseline Anchor

Headline numbers#

Test below val (1.26 < 1.27) — no overfitting; 50k steps is on the undertrained side of the budget, not the overfit side.

What this proves (and doesn't)#

Proved:

• Byte-level cross-entropy in nats, divided by ln(2) per scored byte, is the bits/byte arithmetic-coded code length. Identity, not approximation. No exotic loss, no gradient through the coder, no learnable bits — and the number falls out of standard training cleanly.
• The DeepMind 2023 "Language Modeling Is Compression" framing is real on hardware we own: 14.7M params crush gzip by a factor of ~2.3.
• The metric pipeline is honest. gzip baseline lands where independent sources put it; the model number sits inside the published range for ~10M-param byte-level models on enwik8 (typically 1.0–1.4 bpb at comparable budgets).

NOT proved:

• The actual thesis. The thesis is "small parametric reasoner + external memory + bits-as-output beats a same-size transformer." Tonight's run is the baseline transformer in that comparison. The experiment that decides the bet is the next one — same model + frozen retrieval — and we haven't run it yet.
• Anything about inference cost. That's Phase 2.

Compute and wall time#

• Hardware: RTX 5070 Ti (Blackwell, sm_120), 16 GB VRAM, on WSL2.
• PyTorch 2.12 nightly + CUDA 12.8 (stable wheels don't yet ship sm_120 kernels — Blackwell needs the nightly).
• ~13 train steps/sec sustained, ~14 eval batches/sec. 50k training steps in ~68 min. Full val eval in ~11 s.
• Peak readings: 250 W power draw, 72 °C die temp, 77% fan. Watchdog thresholds (88 °C / 100 °C / 115% TGP / fan-stuck) never approached.

What we changed mid-run#

One real bug found and fixed: eval.py was iterating ByteSequenceDataset with stride 1, scoring every token ~seq_len times. Fixed by tiling non-overlapping (seq_len + 1)-length chunks and dividing total nats by the actual prediction count. The first eval run didn't finish; the second (with the fix) ran in 11 s and produced the numbers above.

Lesson worth keeping: training-time datasets and eval-time iteration have opposite sampling needs. Catching it required noticing eval was queueing ~150k batches when it should have been ~150. Always sanity-check eval batch count against total_bytes / seq_len.

What's locked vs. tentative#

Locked (these are reproducible now and shouldn't change for the rest of Phase 1):

• Byte-level 257-symbol vocab.
• Plain CE loss; bits/byte computed in eval, not training.
• Non-overlapping eval; gzip on the same source bytes.
• 14.7M-param transformer baseline (a hair over the 10M target, but the closest "round" config — d=384, h=6, L=8, ff=1536 — and the right ballpark).

Tentative (likely to change as we run more variants):

• 50k steps. The training curve was still dropping at the end. A 4-hour overnight run at 200k steps would give us a less undertrained anchor.
• seq_len=1024. Fine for a transformer; we'll revisit when the SSM variant lands.

Anchor for next runs#

Every future variant gets compared back to 1.2727 (val) and 1.2565 (test) at this same param/step budget. Variants that don't beat both are losing.

Next experiment on deck: same trained model, same eval, plus a frozen kNN-style retriever blending its next-byte distribution with the model's. That isolates the "retrieval-as-free-context" claim without retraining. If it drops bits/byte, the thesis is alive. If not, we know the retrieval framing needs more than plug-in blending — likely full RETRO-style retraining.