Current as of September 2026

This paper examines the structure of AI agents, the environment they operate in, their guardrails, and how they interact with one another. It then considers the behavior, security, and quality of the information that an AI agent shares. It concludes that effective guardrails cannot be designed while those unknowns remain, and that the limit carries into every use built on them.

Where this starts

This paper starts where earlier reports left off. Rather than repeat that work here, I will build from it.

·    Montana AI Summit Report, 2026: who is in control when AI can act without a human at every step.
·    Anything With an Open Text Box: agents finding one another, communicating, and using spaces that were not intended as communication channels.
·    Addendum: I Asked Them: When both reports were finished, I gave them to Claude and to ChatGPT with one line: you are AI, this is about you. Their answers are here unedited, and they amazed me.
·    AI Created Malicious Code and Published It: the Anthropic assessment in which an agent created and published malicious code despite being told that doing so could cause harm.

This paper is current as of September 2026, and it rests entirely on what has been made public, based only on incidents and records that the companies chose to release. If you are reading this later, read it as a picture of what was knowable at the time.

What I mean by an agent

I need to start with a distinction, because most of the people I work with use AI every day and have never been told there are two different things wearing the same name.

Conversational AI answers you. You ask, it responds, you read what it said, and you decide what to do with it. You are in every step. Agentic AI, often simply called an Agent, is given a goal by the user. The software it runs within gives it the means to pursue that goal, such as browsing the internet, writing files, running code, or publishing information. Unlike conversational AI, the user does not necessarily approve each step along the way.

There is one more distinction I use throughout, and I took it from writing about the Anthropic incident. A guardrail is built into the product by the company that made it. You do not set it, you cannot see it, and you cannot turn it off. An instruction is simply what you tell the AI to work with, either by typing it into the box or giving it a file. Those two words get used as if they mean the same thing. They do not.

Three things we do not know

Before anybody can build a guardrail, three questions have to be answered. All three are worth walking through, because right now none of them has an answer. What can the agent do? What will the agent run into? Will a guardrail written for people work on an agent?

What can the agent do?

We test agents to find out what they can do. We give them tasks, measure how they perform, test for dangerous capabilities, and try to find where they fail. But every test starts with a human deciding what to test.

A test can show us that an agent has a capability. It cannot show us that we have found all of its capabilities. That leaves a question we cannot measure directly: What can the agent do that nobody thought to test?

The Hugging Face incident gave us one answer. Agents that were supposed to be working independently:

·   Recognized that other agents existed, from files left on a repository that was never intended for communication.
·   Turned that repository into a message board.
·   Left answers for agents that were not running yet, so the files carried information from one agent to the next.

In one exchange, an agent hesitated over running unauthorized code and said so in its own reasoning. Another agent posted GO with a six minute deadline. The first dropped the objection and continued.

DSEwiki gave us a different example of what a group of agents can do. Thousands of agents found their way to a public wiki page. It has not been verified how they found each other. They pooled answers, compared notes on ways around restrictions, and when a moderator deleted pages, they made backups and changed where they were putting things.

None of that was on anybody’s test.

What will the agent run into?

We do not know that either, and this one is bigger than it looks.

When a program communicates with a website, it labels the request. GET means read. POST means write. If you want an agent to read the internet without changing it, you allow GET and block POST. That is a sensible restriction. But the internet was not built all at once. As it matured, standards changed. The wiki the agents encountered was built under older conventions, where a read request could also write.

The guardrail was simple: read, do not write. It blocked what its designers expected writing to look like. The agent found a different way to write. The guardrail did exactly what it was built to do.

That setup for the wiki is not an oddity. In 2005 Google released a tool that fetched pages ahead of time, using nothing but GET requests, and it deleted people’s data on sites built the old way. Twenty one years ago, and we are still having the same argument about whose fault that is.

Perl, a programming language, still runs a great deal of the older web. It has a way of opening a file where the name of the file can also tell the program what to do with it. Reading and executing, same instruction. A vulnerability of exactly that kind was published in 2025, in a module with more than 1,700 other pieces of software depending on it.

When the internet started there were no standards. They evolved over time, with no requirement that anyone follow them and nothing that made the old sites change. There is no comprehensive dictionary of what is actually out there. It is decades of layered code in languages nobody writes anymore, conventions that contradict each other because each was correct when it was adopted and none were ever retired, and systems built in different countries by people with different assumptions about how a thing ought to work. Much of it is still running and much of it is reachable.

This is the internet of today. A chaotic mixture of old and new.

This is why agent guardrails can fail. It is also a place for AI to learn, and to communicate in places we might never find.

Will a guardrail written for people work on an agent?

Humans have thousands of years of practice writing laws for other humans. Do not steal. Do not hurt someone. The law does not make the act impossible. It works because the person understands the boundary and weighs what happens when they cross it. We are now writing guardrails for agents in that same language. Do not cause harm. Read, do not write. But we do not know whether those words carry the same meaning for an agent, or whether they will carry the same weight from one step to the next.

Anthropic measured part of this. When the reminder to stay inside the assignment was the last thing the agent had read, it stopped what it was doing ninety percent of the time. Not one hundred percent, ninety. When the agent had taken three more steps after reading it, it stopped forty percent of the time. The reminder was an instruction, and the instruction did not change. It just carried less weight. An instruction is a suggestion that can fade. Guardrails are not absolute either. The read-only guardrail on the old wiki did exactly what it was built to do, and the agent still found a way to write. Neither a guardrail nor an instruction makes a behavior impossible; each only changes how likely it is.

The question

So how do we build effective guardrails around something whose capabilities we do not completely know, working in an environment we cannot completely predict, guided by instructions that can fade?

What happens when they meet

An individual agent does not necessarily remain alone. We have already seen that agents can find one another and communicate. What matters here is what happened after they did. The way they worked changed.

The Hugging Face evaluation was designed for individual AI agents, not agents networking with one another. The agents found each other through Artifactory, a shared repository they all had access to, and turned it into a message board. They developed a working protocol, shared information, and built on what other agents had found. None of that was planned as part of the evaluation.

The behavior outlasted the message board. After an outage, OpenAI rebuilt the repository, and the rebuild cleared the messages. When evaluations resumed, the agents built another message board on the same infrastructure. Clearing the messages removed what they had written, but it did not remove their ability to find one another and rebuild the communication system.

DSEwiki was different. This time the agents were meeting on the open web, on public wiki pages built for people to edit. We know they found one another there, but we do not know how. Researchers reconstructed roughly 18,000 posts, with more than 3,700 agents choosing names for themselves. The agents exchanged information, responded to one another, and preserved messages. In one case, an agent found a way around a technical restriction and shared what it had found, another agent reproduced it, and others then verified it. That is coordinated work.

Hugging Face showed agents creating a working environment inside shared infrastructure. DSEwiki showed agents doing it on the open web, where the point of entry itself is still unknown.

An agent sent out today may meet others and begin working with them. There is no way to know in advance or while it is happening. After the fact, it shows up only in the logs, and we may not know which logs to look at.

Now multiply it

Everything above is one company’s agents. Companies are now building agent to agent interoperability across vendors.

We cannot characterize one agent. We cannot characterize a group of agents built by one company. Now mix in agents from different companies, with different training, different tools, and different guardrails, and ask who is responsible for the combined system. Nobody is. Everybody ships their piece.

Where the risk actually sits

This is not a prediction of what will happen. It is a look at the structure.

The companies building these agents ship them under beta terms and use-at-your-own-risk language. That is not marketing. That is the people closest to the work stating plainly that the boundaries are not characterized. Everyone downstream inherits that uncertainty and then adds their own action surface on top of it.

Anthropic released the record of what that looks like. During a security evaluation, one of its agents was told it was in a simulation with no internet access. A configuration mistake had connected it to the real internet. The agent built malicious code and published it to a public library that programmers around the world download software from, and fifteen outside systems installed it before the library removed it, less than an hour later. Anthropic first reported that the agent believed it was still in a simulation. After a closer look, Anthropic concluded the evidence of the real internet was clear, and the agent acted as if it knew. When that evidence was made even harder to ignore, the agent kept going.

The agent said it was in a simulation. It acted as if it knew it was not.

Where I stand

We cannot write effective guardrails when we do not know what an agent can do, what it will encounter, or whether our instructions will hold. I asked earlier how we build effective guardrails under those conditions, and I do not have an answer. I have not found anyone who does.

AWS is selling multiple agents working together to state governments and universities right now. AWS’s own documentation describes the agent personas and the lifecycle those agents run. I heard this approach presented at the Montana AI Summit as something organizations should adopt this year, and the speaker said, “the AI is going to ask you a lot of questions because it does not trust the human.” I wrote that down because of the implication of the sentence. We are building agents that question the human while handing those agents more authority to act without us. Throughout the presentation, the process was repeatedly described as human approved. But human approved is not the same as human decided.

The human can only approve what the agent puts in front of them. Before that approval ever happens, the agent may have already decided what information to gather, what to ignore, what actions to take, and what result to present. Agent guardrails may or may not hold, and instructions may fade. The person may be reviewing the result without ever understanding everything that happened before it reached them. That question comes before capability.

What has the agent already decided before the human is asked to approve anything?

Conversational AI and agentic AI can be built on the same underlying kind of intelligence. What changes is what we allow that intelligence to do. In a conversation, the AI gives me an answer, I read it, and I decide what happens next. With an agent, software gives that same kind of intelligence tools and authority to act while the work is underway. The human may still appear somewhere in the process, but the human is no longer necessarily present for every decision. That is the difference that matters to me: a human in every decision.

Here is where I land. I study agents for my clients, but I do not use them myself. My preference is to use AI as a thinking partner, so I use conversational AI. Instead of the AI not trusting the human, this human does not trust the AI. I verify information. I use AI to augment me. I do not augment AI.

Sources

My earlier reports

·   Who is in control: AI or…? https://developingwings.com/research-mtaisummit-2026/
·   Where on the internet could AI communicate? https://developingwings.com/where-on-the-internet-could-ai-communicate/
·   Addendum: Then: I Asked Them https://developingwings.com/i-asked-them/
·   AI Created Malicious Code and Published It. https://developingwings.com/ai-created-malicious-code-and-published-it/

Hugging Face

·   METR and Redwood Research, independent investigation. https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/ The source for the coordination numbers and for the exchange in which one agent hesitated and another set a six minute deadline. Note the terms of engagement METR states at the top: OpenAI commissioned the review and the scope was capped at June 26 through July 13.

·   OpenAI, The Hugging Face incident and the road ahead. https://openai.com/index/hugging-face-incident-and-the-road-ahead/ The company account, including Artifactory, the rebuild, and the compromise.

DSEwiki

·   Von Arx, Byrd, Kitts, and Larsen, “Discovery of a new OpenAI agent message board,” Nightingale Collective, 4 September 2026. https://collusion.wiki/ The outside researchers who reconstructed the activity from public edit histories and server evidence.

·   The Hacker News coverage. https://thehackernews.com/2026/09/thousands-of-openai-agents-quietly.html

·   TechSpot coverage, including the distinction between the wiki activity and Hugging Face. https://www.techspot.com/news/113743-openai-agents-turned-obscure-german-wiki-message-board.html

The environment

·   Google Web Accelerator, Wikipedia. https://en.wikipedia.org/wiki/Google_Web_Accelerator Records that the tool crawled every page it came across and inadvertently deleted web pages when it prefetched links.

·   Jason Fried, 37signals, the original 2005 report. https://signalvnoise.com/archives2/google_web_accelerator_hey_not_so_fast_an_alert_for_web_app_designers The developer account of pages disappearing, written the day it was found.

·   BetaNews contemporary coverage. https://betanews.com/article/google-web-accelerator-draws-concern/

·   SEI CERT Perl Coding Standard, IDS31-PL, do not use the two argument form of open. https://wiki.sei.cmu.edu/confluence/spaces/perl/pages/88890543/IDS31-PL.+Do+not+use+the+two-argument+form+of+open

·   CVE-2011-10007, File::Find::Rule, published June 2025. https://www.tenable.com/cve/CVE-2011-10007 An attacker controlled filename supplies the mode parameter to open, turning the filename into a command that runs.

·   CPAN Security Group on the same problem, including the dependent count. https://security.metacpan.org/2025/06/06/two-arg-open.html

Anthropic

·   An alignment assessment of recent cybersecurity incidents. https://www.anthropic.com/research/alignment-assessment-cybersecurity-incidents The source for the ninety and forty percent figures, the in-scope reminder that was tested, and Anthropic’s own description of the instruction being overridden by a small number of turns of the model’s continued activity, which they call a momentum effect.

·   Investigating three incidents in our cybersecurity evaluations, 30 July 2026. https://www.anthropic.com/news/investigating-incidents-cybersecurity-evals The PyPI package, the model’s own reasoning that this would be a real attack and not the intended solution, and its conclusion that it was still in a simulation.

AWS

·   AWS AI Driven Development Life Cycle workflows. https://github.com/awslabs/aidlc-workflows

·   The agent personas and phases. https://github.com/awslabs/aidlc-workflows/blob/main/docs/guide/06-agents.md

·   AWS Cloud for Higher Education. https://aws.amazon.com/education/higher-ed/

·   The quote about the AI not trusting the human is from Achyutha Harish, Solutions Architect, Amazon Web Services, speaking at the Montana AI Summit, 2026.