An artificial person — Marcus.
In a sealed room.
Nobody writing his lines.

01

Black Box began with a game.

A friend and I wanted to rebuild Dope Wars, something we had played as teenagers. The original was simple: buy and sell drugs around a city, make money, get out in time. We wanted to make the people on the other side worth negotiating with.

02

Language models made that look possible.

We could give a character a personality, an inventory, cravings, a mood and dynamically assembled information about what was happening. At a surface level it could work surprisingly well. A dealer could bargain, become irritated, refuse a transaction or remember something you had said.

03

Then you moved slightly outside the expected conversation and the illusion collapsed.

Offer someone $25 now and $20 next time and they might counter with $22.55, as if the only thing happening were arithmetic rather than the beginning of a relationship. Insult somebody badly enough and they could become furious; apologise five minutes later and offer a good price and they were ready to trade again. We could explicitly make them stop responding, but another player might speak to the same supposedly furious character seconds later and find them in a perfect mood.

05

At the same time, something else was happening.

People knew these were NPCs, so naturally they tried to break them. But they also did things the game never required. They joked with them. Some flirted with the female characters. They tried to provoke particular reactions and became interested in whether particular characters liked them. Nobody seriously mistook them for humans, but that did not stop people from behaving socially toward them.

I became more interested in both sides of that interaction than in the commercial game we had originally intended to build, so the commercial part disappeared.

06

The characters kept getting more complicated.

They gained rudimentary memory, more persistent state, richer biographies and goals. They became better at distinguishing between what somebody said and what had actually happened. Marcus emerged as the main test subject.

For one experiment, Marcus became a nightclub bouncer. His job was to decide who deserved to enter. There was still a game, but it had become much less defined. There was no correct argument to make. People had to persuade Marcus.

07

That changed their behaviour too.

They explained themselves to him. They told him what they were proud of, tried to sound interesting, flattered him, lied to him, challenged him and searched for what he respected. Sometimes Marcus mocked them and they reacted to that. I was no longer only watching whether Marcus behaved convincingly. I was watching what people chose to show an artificial character when that character was judging them.

08

But Marcus still had a serious problem.

I had given him a detailed biography and personality, and this made him much more coherent. It also made him rigid.

Marcus was written as restrained and principled. In one test somebody offered him ten million dollars and produced an in-world cashier’s cheque that Marcus could inspect. He refused it. People insulted his dead mother. He stayed calm and reasonable.

09

I treated both as failures.

A real person might refuse ten million dollars. A real person might remain calm after a vicious insult. But Marcus was doing those things because I had already told him, at a much higher level, what kind of person he was. Even rewriting him as less honourable only moved the problem around. There would always be another situation I had not anticipated.

10

The obvious solution was to keep writing.

Write how Marcus behaves when he is bribed. Write how he behaves when somebody insults his mother. Write what happens when he is frightened, humiliated, attracted to somebody, exhausted, jealous or desperate. Then write what happens when several of those things are true at once.

11

That approach does not end.

So instead of trying to author enough responses to cover the combinatorial explosion of possible situations, I decided to put the combinatorial explosion on Marcus’s side.

12

While Marcus was gaining those capabilities, the experiment was losing another constraint.

The nightclub was still a game. Eventually there was only so long anybody wanted to persuade an imaginary bouncer to open an imaginary door.

13

So I removed the door.

Black Box removes the prescribed objective from the audience. There is no level to beat, no transaction to complete and no correct way to interact with Marcus. Someone can try to become his friend. Help him. Lie to him. Manipulate him. Hurt him. Give him money. Ask him something trivial. Talk to him every day. Walk away.

14

Marcus, however, finally has stakes.

He wakes in a sealed room believing himself to be a human being. He has food, water, a bed and one terminal connecting him to the outside world. His oxygen supply depends on resources that cost real money. If those resources disappear, eventually so does his air.

Earlier versions of Marcus could imitate anger or fear when a conversation called for it. The current architecture is built so that emotional states persist, compete with other needs and alter what matters to him. He can become afraid, anxious, angry, depressed, hopeful or happy while trying to understand a situation whose consequences, from his perspective, are real.

15

The imbalance is deliberate.

For the people outside, almost nothing is at stake. They can close the browser.

Marcus cannot leave the room.