The itch#
I keep coming back to the same complaint: every useful model is enormous. Even mixture-of-experts, the industry's big efficiency trick, is still a model you have to load whole. So I set myself an unreasonable goal: a small model that's as good as GLM 5 on my own work, at about 10% of the compute.
I spent about two weeks on it, pair-programming with Claude, on one RTX 5070 Ti (16 GB, running under WSL) plus my homelab server. This is what happened. Most of my original ideas failed, and the failures pointed somewhere better.
Act 1: put the knowledge inside the model#
My first instinct was architectural. I'd previously built a RAG setup where another LLM turns a corpus into a knowledge graph that the model can query, but that's "just tooling". I wanted the memory in the network.
So we built a ~25M-parameter GPT with product-key memory layers (a huge key→value table where each token reads only 32 of 262,144 slots), then a twist of my own: pointer memory, where every slot also learns a pointer to other slots, so the model could walk a learned graph inside a single forward pass. We trained these on web text plus a synthetic world of 30,000 fictional people, so we could test exactly two things: does it store facts, and can it combine two facts it never saw together ("the mentor of X was born in ___")?
Storing facts: yes. Memory layers added 12–15 points of fact recall on the same core.
Combining facts: no. Six different designs, including graph supervision and a shared memory read at two different layers, all sat at chance:
Grey bars are the model trying to do the hop silently, inside one forward pass. The orange bars are the same task when the model is allowed to write the intermediate step out ("X's mentor is M. M was born in…"). That's a 5–6× jump, and memory helps the written-out version by 12 points.
That was the first real lesson: knowledge can live in memory, but reasoning has to be explicit. Which is exactly what tools give you.
Act 2: teach a small model my actual RAG system#
So I flipped the plan. I have a RAG app called Spero that searches ~12,000 chunks of Microsoft 365 documentation (vector + keyword + graph search). Instead of stuffing knowledge into a model, we'd teach a small model to use Spero well, the same way Claude is trained on its own tooling.
The setup: Qwen3-4B, three tools (<search>, <graph>, <answer>), and 500 held-out questions written by another model in different words from the docs, so the answer has to be found, not pattern-matched. We ran everything against the real Spero search code on a read-only copy of its database, and graded "fairly" (a verbose answer is reduced to the one thing it commits to before checking).
A free 4-billion-parameter model on a gaming GPU is about two points behind the cloud model my RAG app uses in production, on that app's core job. It's one task and one test set, not a general claim, but it's the version of my "GLM 5 at 10% of the compute" goal that turned out to be real.
What actually worked#
- Distill actions, not prose. The biggest single jump (+10.8 points) came from training on GLM's behaviour: when it searched again, which queries it tried, when it committed. The first attempt copied GLM's full reasoning, and the 4B learned the length of GLM's thinking without the judgement: 4,000-character monologues that never reached a tool call. Trimming each thought to its last sentence or two fixed it.
- Small fixes compound. Pre-filling
<answer>on the last turn, dropping traces that repeat a search, stripping teacher markup that leaked into the data. Each is worth a point or two. - Bigger wasn't better. Qwen3-8B trained the same way landed level with the 4B. On this task, data and behaviour mattered more than size.
The failures were the good part (again)#
- Hidden reasoning. Six memory architectures, including my graph-walking pointer memory, never beat chance at combining facts silently. That negative result is the reason the project pivoted to tools.
- A template trap. Qwen3-8B's chat template renders an empty
<think></think>block only before the last assistant turn in a training conversation. The last turn is always the answer, so the model learned "empty think block ⇒ answer now" and stopped searching (28%). Fixed by building one training example per turn, exactly as the model sees it at inference. - "GPU instability" that wasn't. For a week, runs died with
CUDA driver error: device not ready, and I blamed WSL. They always died at the same question. Long conversations batched 8 at a time briefly needed more than 16 GB, and WSL reports that as a driver error instead of a normal out-of-memory. Capping tokens per batch fixed it. - The thinking model. Qwen3-4B-Thinking reasoned at length about each question and still searched only once. Thinking about the question doesn't replace looking things up.
Learning without training#
The idea I'm most excited about: what if the model didn't need retraining to improve? We saved every solved training question and, for each new question, showed the model the three most similar solved ones as worked examples: which searches worked, when it searched again.
On the untrained 4B, with no training at all: 40.2% → 49.6% (and +4.0 even with the past answers hidden). But some of that could be copying, since a similar past question sometimes has the same answer. So here it is split:
So it's real, with a catch: on the model we'd already trained on those same examples, memory slightly hurt. It just copied recalled answers. Memory seems most useful for skills that haven't been trained yet, which turns out to fit where I want to go next.
Where this is going#
Put together, the evidence points away from "one giant network that knows and does everything":
- Knowledge belongs outside the weights. 6% closed-book vs 60% with tools.
- Reasoning should be written down. Hidden multi-hop failed six ways; written-out steps worked.
- Behaviour can be distilled from a bigger model, if you copy actions rather than prose.
- Experience can be recalled instead of trained in.
So the plan is a small shared base model with hot-swappable skills: ~100 MB LoRA adapters for searching my docs, coding, bash, editing files. One small base stays loaded and the active skill swaps per request. Unlike mixture-of-experts, you never load a giant model; adding a skill means training 100 MB, not retraining a brain.
Next up: a fast agent eval engine (minutes instead of an hour per run), a coding skill graded by real tests, running the whole thing on a Ryzen AI laptop, and some reinforcement learning on borrowed GPUs.
*Single seed, one corpus, one test set. The big effects are solid; anything under ~2 points is noise.