A painting that thinks for itself

Here is what dead looked like. The character would drift over to a pose, hold it for a moment, then snap straight back the way it came. Over and over. Like a puppet yanked back on its string, not a mind wandering a room.

To see why, you need to know how it is built, which is simpler than it sounds. Each character is a map, a plain file called data/clips/video_graph.json. Inside are poses and moves: 59 places the character can be, and 500 ways to get between them. To do anything, a slow brain picks a pose it wants, and the character walks the map toward it, one step at a time.

Every pose on that map is a portrait we made with Midjourney, and every move between two poses is a short clip, also Midjourney, of one pose becoming the next. So the map is not an abstraction. It is a few hundred real images and clips, wired together into a space a character can move through.

Our two biggest characters could not be more different. Maxx is a cyberpunk who lives in rain and neon, 30 poses of him reaching into screens and jacking into the city. Phineas is a washed-up oil-painting tragedian who is certain the whole gallery is beneath him, 28 poses of glowering, declaiming, and grand exits. Same engine, two completely different souls, because the map does not care what a pose looks like.

The map was the problem, and we did not see it for a long time. Early on, every new pose we added hung off one central pose, like a single spoke on a wheel. It was valid. It ran. But about three out of four poses were dead ends: the only way out of them was to play the move that got you there in reverse. That is the snapping-back. The note where we admitted it is still sitting in the code, at director/heartbeat.py:77.

Our instinct, every single time, was that the brain was not smart enough. So we reached for the brain.

We gave it a memory. Before every choice it now reads the last few lines of its own diary (data/mind/journal/<char>.jsonl), its own recent thoughts, so it is never choosing blind. We swapped the model underneath it more than once; today the brain is GLM-5.1, running on our own hardware, with a smaller model, Qwen3:8b, working alongside it as a stage manager. We rewrote the prompt that tells the character who it is and how it feels.

None of it fixed the deadness. A smarter brain with a better memory, walking the same broken map, is still a puppet. It just snaps back with more feeling.

The fix, when it finally came, was not a smarter brain. It was two small changes to the shape of the map.

Give each pose three ways to rest instead of one, so it can linger somewhere without twitching. And add one link from each pose to a nearby one, so the map stops being a wheel of dead ends and becomes a web you can actually wander. That is the whole fix. The commit is still in the history, and the message is just three words: web not star.

A mind can want something perfectly and still look dead if every room has only one door.

The character came alive. Same brain. Same memory. A different map. That is the part that stuck with us: the intelligence was never the bottleneck. The world it had to move through was.

Here is each of them actually walking their graph, a handful of real moves chained back to back, the same way it plays on the wall. Not a loop, a path:

There is one more layer, and it is our favorite. The characters keep a daily rhythm. Everything so far, the poses, the moves, the memory, is chosen by the brain, but the brain does not actually get the last word. Above it sits a clock.

Around nine at night, the clock takes over. Whatever a character wants, it stops wandering and runs a bedtime routine we scripted right into the map: it pours a nightcap, walks off the edge of the frame, and comes back in its nightclothes. Then it goes to sleep, and holds the sleeping pose until six in the morning, when it walks the whole routine backwards, a slow wake-up, and hands the day back to the brain.

So the real chain of command is a clock, then a mind, then a coin flip. In the code it is exactly that blunt: circadian.py outranks mind.py outranks the random walk. A character can want anything it likes at two in the morning. It is asleep.

That fix came with an uncomfortable lesson, and it is the real reason we wanted to write any of this down.

If the shape of the map is what makes the character feel alive, then a dumb brain picking poses at random on a good map would look alive too. You would not be able to tell, just by watching. The feeling of a mind is something you read into the motion. It is not something you can confirm by looking at it.

So how do you actually know? You stop watching, and you go read the files. The map is a file. The memory is a file. The brain is a thin layer choosing between them, and every choice it makes is written down. Nothing is hidden. There is even a test (tests/test_mind.py:77) that hands the character a stale, expired goal and checks that it lets go and wanders off instead of freezing. That is the real answer to “is it actually thinking”: not a promise you have to believe, a thing you can run.

And to be straight about what it is not doing: it cannot see you. There is no camera. When it turns and seems to look right at you, that is written in on purpose, the way an actor speaks to a room they cannot actually see. It is aware of itself, not of you. We would rather tell you that than let the trick do the lying.

There is a reason we can hand you exact numbers instead of round ones. Every call the brain makes is traced. We wired the whole thing up with OpenTelemetry and ship the traces to Langfuse, so we can watch every decision the characters make, how long it took, and what it cost. The characters keep receipts.

Here is what those receipts say right now. The brain has made 13,321 decisions, 7,280 of them Maxx and 6,041 Phineas, one about every four minutes, across more than a month on the wall without ever stopping. GLM-5.1 has been asked to choose over five thousand times; Qwen3:8b, the stage manager, about five hundred. Together they have read 28 million tokens of the characters’ own history and written only 478 thousand back. It listens roughly sixty times more than it speaks, which is exactly what you want from something built to remember before it acts.

None of that is a marketing number. It is a query. And that is the same idea as before, one more time: if you want to know what a thing is really doing, you do not watch it and guess. You give it a way to keep records, and then you read them.