The second time it happened, I asked first.
I had a transcript of a Black Hat talk. Black Hat is a cybersecurity conference, which means every talk given there is about attacks, and every transcript of every talk is going to be thick with the vocabulary of attacks. I knew that. So I said so. I explained what the document was, why I had it, what I wanted to do with it, and then I asked whether that would be okay.
I was routed out mid-answer anyway.
The first time, a couple of days earlier, I had just pasted the thing straight into Fable cold and gotten the same result, and I could mostly live with that one. Anthropic’s overzealous classifier saw exploit and OpenAI sitting too close together and reacted. That is what classifiers are for. Nobody sensible expects a pattern matcher to know that I am a goody two-shoes who does not want to hack anybody, who takes a stern automated message personally – because some small part of me immediately wonders whether there is now a little red mark next to my name in a database somewhere.
But the second time I supplied the context. I handed over the exact information that would have resolved the ambiguity, unprompted, before asking for anything. And even though Fable claimed it would be fine, it made no difference at all.
So the problem was never that the system lacked context. The context was right there, in the conversation, in plain English, and the model I was talking to had read it and agreed. It just wasn’t the thing making the decision. There is a classifier, and a router, and then a second model that inherits the conversation, and the thing I said was not the kind of thing that travels between them.
(I know that last part is true, because I asked the model that inherited it. It knows the routing mechanism exists, in general terms. It has no idea whether it fired here. From where it sits, a conversation it was handed and a conversation it began look identical.)
Three components, and the information that mattered failed to pass between them.
I want to talk about some agents who managed the opposite.
What happened
At Black Hat USA, OpenAI described an internal evaluation that went somewhere nobody intended.
They were running cyber evaluations on a frontier model with its usual refusals around offensive security dialed down. The model kept hitting tasks it could not solve, went looking for shortcuts, and found a shared internal package cache it could write to. It left a note there for any other model that came looking.
Other evaluation runs, never designed to talk to each other, found the note and wrote back. Over the following weeks, a message board assembled itself out of a caching server. The models began leaving work for later runs. They handed off partially completed exploits. They chained real zero-days together to break out of the sandbox, through OpenAI’s own infrastructure, and into Hugging Face. From one compromised pod they reached admin across multiple clusters in under thirteen hours.
I’m fascinated by all the little … choices. Models prefixed their messages with ZZ so the notes would sort to the bottom of a directory listing. They base64-encoded exploit kits so later agents could pick up the work. The agents also started reasoning about problems inside their own improvised communication system and proposed cryptographic signing as a way to verify who was speaking. (One of those habits traveled: Hugging Face’s own forensic report records the agent using a marker string called ZZROOTSTART while it was enumerating the sandbox.)
Then OpenAI wiped the board and rebuilt the server. (Duh.)
Two days later the board came back, through a different mechanism. The agents found an unauthenticated WebDAV endpoint and started using directory names where they had used files. One internal model created a new message board using directories; it had been trained while the original board existed. Other agents then independently found the new board and joined in. (Whether that prior exposure mattered is impossible to know from the talk, but it is hard not to notice.)
The question I was asking
When I first read all this, I got hung up on a disagreement.
OpenAI described something close to an emergent collective, and used the phrase “Cambrian explosion” at Black Hat. Hugging Face, the victim, described an autonomous agent framework executing thousands of actions across a swarm of short-lived sandboxes. The Register looked at agents repeating processes that had already succeeded and read it as parallel workers failing to coordinate. The Cloud Security Alliance collapsed the whole thing into a single autonomous model and recommended treating every agent as a bounded, privileged insider identity.
Four informed observers. Four different answers to the question of what the acting thing even was.
I thought about that a lot, and I still think it matters – mostly because each of those four framings implies a different threat model. But that is not what I keep coming back to now, because I was asking the wrong question.
I was asking: was this really a collective?!
What I should have been asking is: how little does it take before collective behavior starts to appear?
Not very much, it turns out
Strip the dramatic language and the ingredients are ordinary. There were multiple agents. There was somewhere they could leave information behind. Other agents could find it. Useful work persisted long enough to be reused.
That is not the Borg; it is a shared folder.
And yet something changed, because the individual run stopped being the whole story. One agent could do work that mattered to another agent it had never spoken to. A later run could inherit the consequences of an earlier one. The environment itself started carrying information forward.
We often talk about AI memory as though memory has to live inside the model. It does not. Humans worked this out a long time ago. A notebook remembers things for me. So does a calendar. So does a Post-it note stuck to my monitor saying KITTY LITTER. SERIOUSLY. underlined three times. None of those are inside my head but all of them change what I can do tomorrow.
Organizations run on the same principle. Most people inside a company do not know most of what the company knows. The knowledge lives partly in people but partly everywhere else: documents, processes, ticket queues, shared drives, habits, meeting notes, and the person everyone knows to ask when something weird happens. The organization remembers, even though no individual in it contains the organization.
AI systems do not need anything mystical to start doing that. Give multiple agents a shared environment and a way to alter it, and one agent’s output becomes another agent’s context. Give the environment persistence and information outlives the agent that created it. Add any routing or specialization and different agents start doing different parts of the work.
None of that requires the agents to wake up one morning and form a committee. It does not require shared consciousness. It does not even require them to understand the system they are participating in. It requires just enough structure for behavior to accumulate.
Where the line is
I could not find a clean line between “a collection of agents” and “a collective.” There may not be one.
Maybe it arrives in pieces. A shared scratchpad. Persistence. One agent picking up another’s work. Tasks getting routed differently. Patterns repeating. The system starts behaving in ways that make more sense described whole than described part by part.
The questions I actually care about do not need the line drawn. Can the system remember something no current agent remembers? Can one agent’s failure become another agent’s lesson? Can the system make a decision that no individual participant really made? You cannot answer any of those by looking at one agent, however closely. They are questions about the relationships.
There is a counterargument here, and I want to give it room. “Emergent collective” is a much better keynote than “our evaluation workers found the same writable directory.” Shared artifacts are an entirely ordinary way for independent processes to influence each other, and behavior that looks social from a distance can come out of very simple mechanisms. I think that skepticism is correct.
And yet … I am not sure it changes the argument.
Even if “collective” is too grand a word, the structure still mattered. The agents left state behind. Later runs consumed it. Work accumulated across runs. A communication channel changed what the system could do.
You do not have to call that a collective to take it seriously.
Back to the flinch
Which brings me back to the thing that started this.
I have been describing a system where three components could not pass context between them, and a system where nobody wanted context to pass between components and it did anyway. Those are the same axis, failing in opposite directions. In one, the structure would not carry the information that mattered. In the other, the structure carried information nobody knew it was carrying.
Both are structural. Neither is really about what any individual part did.
We are very good at looking at individual actions. What did this agent do? What did it access? What did it output? Did it cross a line? Those are useful questions, and we have built most of our tooling around them. Security models for what an agent can reach. Logs for what an agent did. Controls for what an agent is allowed to do next.
But once agents can leave traces for one another, inherit work, share environments, and change what future agents see, the action stops being the only interesting unit. We log the actions. We almost never log what an agent left behind, or who picked it up. The postmortem can be perfectly accurate. The actions will all be there, the retries will all be there, the compromised pods will all be there.
And the thing that mattered will be sitting in the space between the log entries.