Phase 1 Results — Baseline Anchor

Headline numbers

Test below val (1.26 < 1.27) — no overfitting; 50k steps is on the

undertrained side of the budget, not the overfit side.

What this proves (and doesn't)

Proved:

• Byte-level cross-entropy in nats, divided by ln(2) per scored byte, is

the bits/byte arithmetic-coded code length. Identity, not approximation.

No exotic loss, no gradient through the coder, no learnable bits — and

the number falls out of standard training cleanly.

• The DeepMind 2023 "Language Modeling Is Compression" framing is real on

hardware we own: 14.7M params crush gzip by a factor of ~2.3.

• The metric pipeline is honest. gzip baseline lands where independent

sources put it; the model number sits inside the published range for

~10M-param byte-level models on enwik8 (typically 1.0–1.4 bpb at

comparable budgets).

NOT proved:

• The actual thesis. The thesis is "small parametric reasoner + external

memory + bits-as-output beats a same-size transformer." Tonight's run is

the baseline transformer in that comparison. The experiment that

decides the bet is the next one — same model + frozen retrieval — and we

haven't run it yet.

• Anything about inference cost. That's Phase 2.

Compute and wall time

• Hardware: RTX 5070 Ti (Blackwell, sm_120), 16 GB VRAM, on WSL2.
• PyTorch 2.12 nightly + CUDA 12.8 (stable wheels don't yet ship sm_120

kernels — Blackwell needs the nightly).

• ~13 train steps/sec sustained, ~14 eval batches/sec. 50k training steps

in ~68 min. Full val eval in ~11 s.

• Peak readings: 250 W power draw, 72 °C die temp, 77% fan. Watchdog

thresholds (88 °C / 100 °C / 115% TGP / fan-stuck) never approached.

What we changed mid-run

One real bug found and fixed: eval.py was iterating

ByteSequenceDataset with stride 1, scoring every token ~seq_len times.

Fixed by tiling non-overlapping (seq_len + 1)-length chunks and dividing

total nats by the actual prediction count. The first eval run didn't

finish; the second (with the fix) ran in 11 s and produced the numbers

above.

Lesson worth keeping: training-time datasets and eval-time iteration have

opposite sampling needs. Catching it required noticing eval was queueing

~150k batches when it should have been ~150. Always sanity-check eval

batch count against total_bytes / seq_len.

What's locked vs. tentative

Locked (these are reproducible now and shouldn't change for the rest

of Phase 1):

• Byte-level 257-symbol vocab.
• Plain CE loss; bits/byte computed in eval, not training.
• Non-overlapping eval; gzip on the same source bytes.
• 14.7M-param transformer baseline (a hair over the 10M target, but the

closest "round" config — d=384, h=6, L=8, ff=1536 — and the right ballpark). Tentative (likely to change as we run more variants):

• 50k steps. The training curve was still dropping at the end. A 4-hour

overnight run at 200k steps would give us a less undertrained anchor.

• seq_len=1024. Fine for a transformer; we'll revisit when the SSM

variant lands.

Anchor for next runs

Every future variant gets compared back to 1.2727 (val) and 1.2565

(test) at this same param/step budget. Variants that don't beat both

are losing.

Next experiment on deck: same trained model, same eval, plus a frozen

kNN-style retriever blending its next-byte distribution with the model's.

That isolates the "retrieval-as-free-context" claim without retraining.

If it drops bits/byte, the thesis is alive. If not, we know the retrieval

framing needs more than plug-in blending — likely full RETRO-style

retraining.