In June I wrote about augur, my from-scratch compressor built on one idea: compression is prediction. Predict each bit, code only the surprise. Back then the win was beating xz and zstd on structured data. That bar turned out to be low. Context mixing beats xz on almost everything if you put the work in.
So I moved the bar. The goal now is the best compression for every file type, and the honest reference for that is paq8px, the research compressor that has topped the benchmarks for years. This post covers how far augur got, where it still loses, and the newest piece: a neural network that learns the file while it compresses it.
Since June, augur picked up models for nearly everything I could throw at it:
Audio: 8 to 32-bit WAV. Each sample is predicted by a least-squares fit that re-solves every 16 samples, then a cascade of adaptive filters, and only the residual is coded. It beats FLAC -8 by 8–31%, and it beats paq8px on 24-bit audio by 29%, because paq8px has no 24-bit model.
Images: PPM, BMP, TGA, and the pixels inside PNGs. Two dozen predictors per pixel, a colour transform, contexts on actual pixel values. 25.6% smaller than JPEG XL -e 9 on the Kodak set.
Already-compressed files: this is the fun one. A PNG, a ZIP, a PDF or a gzip is mostly deflate, which looks like noise to a modeller. augur unpacks the deflate stream, compresses what's inside, and stores a recipe to rebuild the original bytes exactly. For zlib that means a bit-exact clone of zlib's encoder (checked against the real thing on 3,480 cases). For other encoders (GNU gzip, Info-ZIP, PNG optimisers) it records only the places where their choices differ from a predicted policy.
GIF: same trick for LZW, including encoders that emit "wrong" codes on purpose.
JPEG: the scan is modelled bit by bit through the Huffman codes, predicting each coefficient from its neighbours across block edges. About 20% off the original JPEG, losslessly.
x86 code: an instruction model with its own opcode tables.
Every one of these round-trips byte for byte, and every number in this post comes from a run that decompressed and compared.
Being honest about paq8px meant running it on everything. It is very good. On dickens, 10 MB of Charles Dickens, paq8px -8 gets 1,896,704 bytes. augur was 9% behind on prose, 25% behind on executables, 10% behind on photos. paq8px also unpacks zlib streams, so on JARs and PDFs it wins by 30%+.
augur does win in places: 24-bit audio, gzip and ZIP files made by non-zlib encoders (paq8px can't reproduce those, so it can't unpack them: on a GNU-gzipped tarball augur is 2× smaller), and structured data like logs and JSON.
To find out where the text gap came from, I built an instrumented paq8px that can switch off any of its models and measured what each one is worth. Two surprises:
augur's core is stronger than paq8px's core. Strip paq8px down to its order and match models and augur beats it. The whole gap was paq8px's word model.
No single word context matters. The word model's 46 contexts are worth 4.7% together, but the best single one is worth 0.19%. The value is spread across many redundant overlapping contexts. And when I added more contexts to augur, they helped less than they should have, because augur's mixer was paying a tax on redundant inputs. Skipping training on tiny errors fixed part of that, and improved nearly every file type.
That still left a gap of tenths of a percent per experiment. Time for something bigger.
Teaching it to read: an LSTM inside the compressor#
Context models are lookup tables: "what came after the qu last time?" They're fast and sharp, but they only know contexts they've seen exactly. An LSTM is a small recurrent neural network that carries a continuous state through the file, so it can generalise to contexts it has never seen verbatim.
augur's LSTM reads one byte at a time and predicts a probability for each of the 256 possible next bytes. There's no pre-training: it starts from random weights at byte zero and learns as it goes. The decompressor runs the exact same learning in the exact same order, so both sides always hold identical networks.
The compressor codes bits, though, not bytes. The trick is a tree of partial sums: once you know the first few bits of the byte, only the bytes that start with those bits are still possible, and the next bit's probability is just "mass of the ones that continue with 1" over "mass of everything still possible." Click through the bits to see it (16 symbols here instead of 256, so it fits):
From a byte distribution to bit probabilities
<style> .wrap { font-family: var(--mono); color: var(--ink); max-width: 760px } .row { display: flex; gap: 10px; align-items: center; flex-wrap: wrap; margin: 6px 0 12px } .bits span { display: inline-block; min-width: 22px; text-align: center; padding: 2px 4px; border-bottom: 2px solid var(--rule-strong); margin-right: 4px } .bits span.known { color: var(--accent); border-color: var(--accent) } .stat { font-family: var(--serif); font-size: 15px; color: var(--mid) } .stat b { color: var(--ink); font-family: var(--mono) } svg { width: 100%; height: auto; display: block; overflow: visible } .tip { position: absolute; pointer-events: none; background: var(--surface-2); color: var(--ink); border: 1px solid var(--rule-strong); padding: 4px 8px; font-size: 12px; border-radius: 4px; display: none; white-space: nowrap }</style><div class="wrap" style="position:relative"> <div class="row"> <button id="b0">next bit = 0</button> <button id="b1">next bit = 1</button> <button id="reset">reset</button> <span class="bits" id="bits"></span> </div> <div class="stat" id="stat"></div> <svg id="chart" viewBox="0 0 760 230" role="img" aria-label="Bar chart of the predicted probability of each of 16 symbols; symbols still possible are highlighted"></svg> <div class="tip" id="tip"></div></div><script> // a made-up next-symbol distribution, as the LSTM's softmax would give const P = [0.02,0.01,0.05,0.03, 0.30,0.06,0.02,0.01, 0.04,0.02,0.18,0.09, 0.03,0.06,0.05,0.03]; const css = n => getComputedStyle(document.documentElement).getPropertyValue(n).trim(); let known = []; const svg = document.getElementById('chart'), tip = document.getElementById('tip'); const W = 760, H = 230, y0 = 10, base = 190, bw = W / 16; function range() { let lo = 0, size = 16; for (const b of known) { size /= 2; if (b) lo += size; } return [lo, lo + size]; } function draw() { const [lo, hi] = range(); const mid = lo + (hi - lo) / 2; const sum = (a, b) => P.slice(a, b).reduce((x, y) => x + y, 0); const left = sum(lo, mid), all = sum(lo, hi); const done = known.length === 4; document.getElementById('bits').innerHTML = [0,1,2,3].map(i => i < known.length ? `<span class="known">${known[i]}</span>` : '<span>?</span>').join(''); document.getElementById('stat').innerHTML = done ? `Symbol <b>${lo.toString(2).padStart(4,'0')}</b> decoded. It had probability <b>${P[lo].toFixed(2)}</b>, so it cost <b>${(-Math.log2(P[lo])).toFixed(2)} bits</b>.` : `Still possible: symbols ${lo}–${hi - 1}. P(next bit = 1) = mass on the right / mass remaining = <b>${(all - left).toFixed(2)} / ${all.toFixed(2)} = ${((all - left) / all).toFixed(3)}</b>`; const max = Math.max(...P); let g = `<line x1="0" x2="${W}" y1="${base}" y2="${base}" stroke="${css('--rule-strong')}" stroke-width="1"/>`; P.forEach((p, i) => { const h = (p / max) * (base - y0 - 14), x = i * bw + 2, w = bw - 4; const live = i >= lo && i < hi; const right = live && !done && i >= mid; const fill = live ? (right ? css('--accent') : css('--accent-2')) : css('--rule-strong'); g += `<path d="M${x},${base} v${-(h - 4)} q0,-4 4,-4 h${w - 8} q4,0 4,4 v${h - 4} z" fill="${fill}" opacity="${live ? 1 : 0.45}" data-i="${i}"/>`; g += `<rect x="${i * bw}" y="${y0}" width="${bw}" height="${base - y0}" fill="transparent" data-i="${i}"/>`; g += `<text x="${i * bw + bw / 2}" y="${base + 16}" text-anchor="middle" font-size="11" fill="${css('--mute')}">${i.toString(2).padStart(4,'0')}</text>`; }); if (!done) { const xm = mid * bw; g += `<line x1="${xm}" x2="${xm}" y1="${y0}" y2="${base}" stroke="${css('--mid')}" stroke-dasharray="3 3"/>`; g += `<text x="${(lo * bw + xm) / 2}" y="${y0 + 4}" text-anchor="middle" font-size="11" fill="${css('--mid')}">bit 0</text>`; g += `<text x="${(xm + hi * bw) / 2}" y="${y0 + 4}" text-anchor="middle" font-size="11" fill="${css('--mid')}">bit 1</text>`; } svg.innerHTML = g; document.getElementById('b0').disabled = document.getElementById('b1').disabled = done; } svg.addEventListener('mousemove', e => { const i = e.target.dataset && e.target.dataset.i; if (i === undefined) { tip.style.display = 'none'; return; } const r = svg.getBoundingClientRect(), wr = svg.parentNode.getBoundingClientRect(); tip.textContent = `symbol ${(+i).toString(2).padStart(4,'0')}: P = ${P[i].toFixed(2)}`; tip.style.display = 'block'; tip.style.left = Math.min(e.clientX - wr.left + 12, wr.width - 150) + 'px'; tip.style.top = (e.clientY - wr.top - 30) + 'px'; }); svg.addEventListener('mouseleave', () => tip.style.display = 'none'); document.getElementById('b0').onclick = () => { known.push(0); draw(); }; document.getElementById('b1').onclick = () => { known.push(1); draw(); }; document.getElementById('reset').onclick = () => { known = []; draw(); }; draw();</script>
My first LSTM was bad: 3.2 bits per character on its own, where the context models manage about 1.9. It took a while to learn what actually matters:
Training was chaotic. Changing nothing but the order floats got added in moved the result by 1.4%. Changing the random seed moved it 4%. Before I could tune anything I had to make training stable, then compare every setting across three seeds.
paq8px's LSTM showed the way. paq8px is GPL and augur is Apache-2.0, so I read its design but wrote my own code. The ideas that mattered: normalise each gate's inputs (RMS norm), give every parameter group its own learning rate, and start with tiny training windows that grow. A fresh network updates every 4 bytes; a mature one looks 100 bytes back. After that rebuild, my LSTM alone codes text as well as paq8px's: 81,543 bytes vs 81,496 on the same 300 KB, at the same speed.
Determinism is the hard part. The decompressor has to reproduce every float exactly, or one bit flips and the rest of the file is garbage. Everything is plain f32 with a fixed summation order and a hand-written exp, so the result doesn't depend on the CPU's SIMD width or the system's math library.
Training runs on a second thread. Backpropagation is two-thirds of the work. It now runs in the background while the next window is predicted with the previous weights, so each update lands one window late. Encoder and decoder follow the same schedule, so they still agree bit for bit. That cut the time by 23% for 0.07% in size.
I expected the LSTM to help prose most. It helps binary data more:
What the LSTM saves, by kind of data (1 MB slices of the Silesia corpus)
<style> .c { font-family: var(--mono); color: var(--ink); position: relative; max-width: 760px } svg { width: 100%; height: auto; display: block } .tip { position: absolute; pointer-events: none; background: var(--surface-2); color: var(--ink); border: 1px solid var(--rule-strong); padding: 4px 8px; font-size: 12px; border-radius: 4px; display: none; white-space: nowrap } table { display: none }</style><div class="c"> <svg id="s" viewBox="0 0 760 300" role="img" aria-label="Horizontal bar chart of the size reduction from the LSTM per file; data in the table below"></svg> <div class="tip" id="t"></div> <table id="tbl"><thead><tr><th>file</th><th>kind</th><th>without</th><th>with LSTM</th><th>saved</th></tr></thead><tbody></tbody></table></div><script> const D = [ ['osdb', 'database tables', 231387, 213748], ['ooffice', 'x86 executable', 301570, 289750], ['x-ray', 'medical image', 421741, 408731], ['nci', 'chemical database', 37279, 36483], ['mr', 'MRI volume', 182720, 179778], ['sao', 'star catalogue', 532213, 525167], ['samba', 'source tarball', 128703, 127892], ['xml', 'XML', 72350, 72083], ]; const css = n => getComputedStyle(document.documentElement).getPropertyValue(n).trim(); const s = document.getElementById('s'), t = document.getElementById('t'); const L = 230, R = 700, row = 34, y0 = 10, max = 8; let g = ''; for (let v = 0; v <= max; v += 2) { const x = L + (v / max) * (R - L); g += `<line x1="${x}" x2="${x}" y1="${y0}" y2="${y0 + D.length * row}" stroke="${css('--rule')}"/>`; g += `<text x="${x}" y="${y0 + D.length * row + 18}" text-anchor="middle" font-size="11" fill="${css('--mute')}">${v}%</text>`; } D.forEach(([f, k, a, b], i) => { const pct = (a - b) / a * 100, y = y0 + i * row + 7, h = row - 14; const w = (pct / max) * (R - L); g += `<text x="0" y="${y + h / 2 + 4}" font-size="12" fill="${css('--ink')}">${f}</text>`; g += `<text x="76" y="${y + h / 2 + 4}" font-size="11" fill="${css('--mute')}">${k}</text>`; g += `<path d="M${L},${y} h${w - 4} q4,0 4,4 v${h - 8} q0,4 -4,4 h${-(w - 4)} z" fill="${css('--accent')}"/>`; g += `<text x="${L + w + 8}" y="${y + h / 2 + 4}" font-size="12" fill="${css('--ink')}">−${pct.toFixed(1)}%</text>`; g += `<rect x="0" y="${y0 + i * row}" width="760" height="${row}" fill="transparent" data-i="${i}"/>`; document.querySelector('#tbl tbody').insertAdjacentHTML('beforeend', `<tr><td>${f}</td><td>${k}</td><td>${a}</td><td>${b}</td><td>${pct.toFixed(2)}%</td></tr>`); }); s.innerHTML = g; s.addEventListener('mousemove', e => { const i = e.target.dataset && e.target.dataset.i; if (i === undefined) { t.style.display = 'none'; return; } const [f, k, a, b] = D[i], r = s.parentNode.getBoundingClientRect(); t.textContent = `${f} (${k}): ${a.toLocaleString()} → ${b.toLocaleString()} bytes`; t.style.display = 'block'; t.style.left = Math.min(e.clientX - r.left + 12, r.width - 280) + 'px'; t.style.top = (e.clientY - r.top - 30) + 'px'; }); s.addEventListener('mouseleave', () => t.style.display = 'none');</script>
Database tables gain 7.6% and the executable 3.9% (measured with 96 cells per layer; the default is now 128, which takes about another 0.5% off). The likely reason: binary formats are full of fields whose meaning depends on position and type, things a byte-context model can't name and a recurrent state can track. Prose gains less, because context models are already very good at text. On the 3 MB Dickens test it's still a solid 1.3%.
The obvious one is speed. augur was never fast, and the LSTM roughly halves its speed on non-media data: about 10 KB/s now. That's fine for archival and cold storage and miserable for anything interactive, so there's a --fast flag that skips the LSTM.
Wider networks where they pay. The default is 128 cells per layer; --lstm 192 takes another 0.2–0.9% off at twice the time. The right width probably depends on the data, so augur could pick it per file.
The rest of the word model. paq8px still has text tricks augur doesn't: word classes, stemming, more indirect contexts.
Images and executables, where paq8px is still 10–25% ahead (the executable gap predates the LSTM, which took about 4% off it).