bmo.guru/ ~
← all posts
Development

Post-LLM Architecture: A Compression-Native Thought Experiment

·3 min read

Thesis#

Current LLM efficiency work (TurboQuant, 1-bit quantization, GPTQ) compresses the representation of a fundamentally redundant architecture. It doesn't fix the underlying waste: transformers do dense matmul over learned dense vectors when the underlying signal (language) is sparse, structured, and has ~1 bit/char of Shannon entropy.

We're betting on a different stack with one coherent claim:

A small parametric reasoner, paired with external memory, with output measured directly in bits, can match or beat a same-size transformer on compression — and dominate it on inference cost.

Compression and intelligence are formally linked (Solomonoff, Hutter, Schmidhuber; DeepMind 2023 "Language Modeling Is Compression"). If that link is real, optimizing directly for bits/char is a more honest objective than cross-entropy-on-tokens, and a model that does it well should be smaller.

Architecture (composed, not piled)#

Four pieces, each attacking a different source of waste:

  1. Mamba / SSM backbone — linear-time sequence modeling, no KV cache. Kills quadratic attention and the inference-time memory blowup.
  2. KAN-style MLP replacement — learnable activations on edges (splines) instead of fixed activations on nodes. Smaller and more interpretable per layer. Drop if it slows research iteration.
  3. Retrieval + tiny reasoner — knowledge lives in an external index, not in weights. The parametric model learns how to think, not what to recall.
  4. Compression-native output head — model emits an arithmetic-coded stream. Trained directly on bits/char, not token cross-entropy. This is the least-standard piece and the one that makes the thesis testable.

Explicitly not combining: energy-based models (#5 from earlier discussion). Different training paradigm; doesn't compose with autoregressive SSM.

Phase 1 — PyTorch prototype (research velocity)#

Goal: find out whether the architecture works at all before investing in a performance rewrite.

  • Scale: ~10M parameters
  • Corpus: enwik8 (the Hutter Prize benchmark — natural fit for a compression thesis)
  • Components:
    • Mamba/SSM backbone
    • KAN-style MLPs (start simple; ablate if it slows us down)
    • Arithmetic-coded output head
    • Retrieval interface (stub for Phase 1; real index in Phase 2)
  • Baselines (same parameter budget, same corpus):
    • gzip — the trivial floor
    • Vanilla transformer — the real benchmark

Phase 1 success criteria (decided before building)#

  • Must: beat gzip on bits/char.
  • Must: match the vanilla transformer baseline within 10% on bits/char.
  • If both hold → proceed to Phase 2.
  • If not → the thesis needs revision before we port anything to Rust.

Phase 2 — Rust inference port (thesis demonstration)#

Only triggered if Phase 1 succeeds.

  • Forward pass on Candle (Hugging Face Rust ML framework)
  • Native Rust arithmetic coder — bit-twiddling tight loops, where Rust genuinely dominates Python (10–100× realistic)
  • Native Rust retrieval index (tantivy / qdrant-style)
  • Target hardware: laptop CPU, no CUDA
  • Measure: tokens/sec, peak RAM, bits/char (must match Phase 1)

Why Rust is for Phase 2, not Phase 1#

~95% of training time is inside CUDA kernels. Rust doesn't make cuBLAS faster. The Rust win is in the outer loop (arithmetic coding, retrieval, inference orchestration) and on no-GPU targets (edge inference). Both of those are Phase 2 concerns. Doing pure-Rust from day one would burn months reimplementing autograd before we learn whether the architecture works at all — wrong order of operations.

Open questions (to resolve before coding)#

  1. How does the arithmetic-coded head train end-to-end? This is the least-standard piece. Need to nail down the loss function and gradient path before writing the model.
  2. Retrieval interface in Phase 1 — what does the stub look like such that swapping in a real index in Phase 2 doesn't require a model rewrite?
  3. KAN keep-or-cut decision rule — what ablation result kills KAN from Phase 1?
  4. Tokenization — byte-level (cleanest for bits/char) vs. BPE (standard, but muddies the compression metric).

Non-goals#

  • Beating GPT-4 on anything. This is a thesis test at 10M params.
  • A general-purpose framework. We're building one specific architecture to test one specific claim.
  • Energy-based / diffusion / non-autoregressive variants. Different bet.
  • Rust autograd from scratch. Use Candle if/when we get to Phase 2.

Status#

Plan locked. Next step: resolve open question #1 (arithmetic-coded head training) before writing model code.