The itch#
Every LLM I use has the same shape: it sits there, completely inert, until I send it a prompt. Even "agents" are just a loop someone wrote around the model that keeps prompting it. The drive to do anything always comes from outside.
So I wanted to try the opposite. Not a bigger model, but a different kind of thing: an AI that learns from scratch, has a real memory, keeps running whether or not anyone talks to it, and decides for itself what to do with its time. My first thought was "we'll just build a brain!", which is obviously a lot of work, and we don't fully understand the brain anyway. But we understand a surprising amount of how it does things. So the plan became: borrow the brain's principles, not its wiring, and invent where neuroscience runs out.
One rule shaped everything:
Hardcode drives and mechanisms, never content or triggers.
Evolution gave us hunger and sleepiness, not a script. So the system is allowed built-in urges (surprise matters, hunger hurts, being awake makes you tired), but nothing ever tells it what to think or when to act. Each tick, the world hands it a symbol or silence, and that's it.
I built this over a few sessions pair-programming with Claude. It's ~1,500 lines of Python and NumPy, with no backprop and no training phase: every synapse learns locally, on every tick.
The architecture#
Sparse columns. Every input (a letter, a glimpse of an image, an action) activates 8 of 256 "columns". The input decides which columns fire; context decides which cells inside them fire. The same letter gets a different cell-level code depending on what came before it. That's how context is held (loosely after cortical minicolumns).
Two memory systems, like the brain's Complementary Learning Systems:
- A hippocampus: exact, one-shot, no interference. It remembers an episode the first time, but only recognizes a context it has seen exactly.
- A cortex: slow and generalizing. It looks only at recent context (as overlapping chunks, plus "time cells" that fire for "this symbol, k steps ago"), so it recognizes a familiar phrase wherever it shows up.
Sleep. Episodes are stored as the sequence of hippocampal cells that fired. During sleep they're replayed, shuffled and interleaved, into the cortex. Sleep pressure builds from synapses grown while awake (the synaptic homeostasis hypothesis) and from time awake (adenosine), and the system falls asleep on its own when the world goes quiet.
A body. Hunger, a few motor actions, curiosity, and an eye that saccades around an image. It plans by imagining: running "if I do X I'll see Y" through its own cortex.
What it can do#
| Milestone | Result |
|---|---|
| Learn a text stream | 99.8% next-character accuracy after one read (best possible: 99.8%) |
| Learn story A, then B | Remembers A at 99.8% after learning B. A small gradient-trained net drops 84.5% → 74.1% |
| Recognize phrases out of context | 0.6% with exact memory alone → 97% after sleep teaches the cortex |
| Name handwritten digits it has never seen | 94% after seeing 1,000 digits with their spoken names (chance: 10%), no labels used as targets |
| Find food | Learns where food is, that it alternates between two rooms, and walks there in ~5 actions (the optimum) vs ~50 at random |
| Always-on life | Reads, daydreams in the gaps, puts itself to sleep, gets woken by input. All self-initiated |
The naming result is the one I like most. Experience is just "glimpses of a digit, then its name spelled out", like a kid hearing a word while looking at a thing. Its mistakes look human, too: nine → "eight", eight → "six".
The failures were the good part#
Almost every working piece started as a failure, and the failures were more interesting than the wins:
- Contexts merging. When a column is surprised, all its cells fire, and choosing the next cell from that blend collapsed different contexts into one. Accuracy sat at 76%. The fix (follow only the cells that carry the specific context, and require a real match before reusing one) took it to 99.8%.
- Memories hijacking each other. Learning story B after A, B got captured by A's well-rehearsed synapses. The fix was very brain-like: one-shot encoding at full strength, plus "reuse a cell only if 6 of 8 context signals agree".
- Confabulation. I gave it the ability to recognize where it is in something familiar. With too little evidence required, it "recognized" a memory after only 4 matching letters, confidently predicted that memory's continuation (often the wrong story), and stopped learning entirely. Accuracy fell from 98% to 87%. That looks a lot like false familiarity in people. It was fixed by requiring 8 letters and having exact memory stay quiet at forks.
- Delusion. On guided walks, eating only ever happened where there was food, so it learned "eat → food" unconditionally. Hungry in the wrong room, it kept eating nothing, forever.
- I had to retract a result. Early on, wandering appeared to help it find food a lot. That was a bug: imagined meals in its daydreams were being counted as real ones.
Does daydreaming do anything?#
This was the question I cared about most, because it's the whole point of an always-on mind. For a long time the honest answer was no.
I tested the textbook theory, that wandering rehearses important unresolved memories. It does do that: without sleep, daydreaming alone recovers about half of what sleep provides. But with sleep available it adds nothing, even when 76% of its daydreams were on exactly the right memories. Sleep replays everything, faster.
So what can wandering do that sleep can't? Sleep only replays what actually happened. A wandering mind can imagine past the end of a memory. I led the system around the left half of a corridor on some walks and the right half (where the food is) on others, never all the way through. While resting it recalled a walk, kept imagining past the end, and stored routes like b>c>d>e>f>g>h>I!+, a path it was never led along. Rats do something similar: their hippocampus "preplays" routes to places they've never been.
| Mind (plans only 3 steps ahead while acting) | Actions to food (best: 9) |
|---|---|
| Daydreams with insight | 13.5 |
| Daydreams without insight | 22.4 |
| Doesn't daydream | 21.1 |
| Plans 12 steps ahead at every action | 12.9 |
A mind that thinks 3 steps ahead matched one that thinks 12 steps ahead at every step, because it worked the route out while idle.
The catch: insight without reality-checking was worse than no insight at all. Its imagination guessed that walking left from the first room leads somewhere (it had never tried; there's a wall), stored that route, and spent the whole test walking into the wall. The fix was to treat an imagined route as a hypothesis. When following it, it knows what it should see next, and if reality disagrees, the route is thrown out. Imagination needs prediction error to stay honest.
What's still unsolved#
- Smart looking doesn't beat random looking yet. The eye needed a sense of where it's pointing plus "feature at a location" detectors (the binding problem) before moving eyes could even match a fixed scan. Curious and purposeful looking still land about where random looking does.
- Habits vs. planning. Remembered routes make it learn much faster early, but slower to adapt when the world changes. When should a mind trust habit over deliberation?
- Finding "stories" in a continuous stream. It keeps its place well now, but it segments the stream into units it remembers, not into stories in our sense.
The thing I keep coming back to is that the parts that worked best were the parts nature had already tested: two memory systems, sleep, recognizing where you are, knowing what you saw where, and imagination checked against reality. It's a tiny system, but it has started doing things on its own. That was the whole point.