What Systems Can't Know About Themselves
The limits of AI agents, and the one call only a human can make
Imagine an apartment. You know the layout: where the kitchen is, which window faces north, why the radiator makes that noise in January. The frame is tight. The information set is bounded. A well-designed agent operating within that frame can do genuinely useful things.
Now step back. The apartment sits inside a building. The building sits in a block. The block is part of a neighbourhood, the neighbourhood part of a city, the city part of a country. The frame has expanded, but not linearly. Exponentially. Each zoom level introduces not just more data but different categories of data, different rules, different relevant history - and some of those rules contradict the ones that applied at the level below. The agent that managed your apartment perfectly now has no idea what to do.
This is a structural property of frames.
The Frame Is Not Just a Technical Limit
The reason I choose this metaphor is that it describes something more fundamental than context window length. Yes, token limits are real. Yes, an agent’s working memory has a ceiling. But even if those technical constraints disappeared overnight, the frame problem would remain.
Because the frame problem is not about information volume. It is about the impossibility of evaluating a frame from inside it.
This is the same argument Gödel made about formal systems, which I presented in a recent essay on the limits of knowledge. A system cannot prove its own completeness using its own axioms. To evaluate whether a frame is the right frame, you have to step outside it. Agents cannot do that. They can accumulate knowledge and apply judgment within a predefined frame. But true judgment - the kind that decides which frame to use, when to abandon it, and whether the metric you are optimising for is even the right metric - requires a vantage point that is not inside the system being evaluated. That capacity is not algorithmic.
So when people suggest that AI will eventually handle every decision, every context, every strategic move, they are proposing that the Gödel problem will be solved. I find that doubtful, not as a pessimist, but as someone who takes the argument seriously.
Why Zooming Out Breaks Things
The conventional assumption is that more context is better. Feed the agent more background, more history, more nuance, and it performs better. There is a version of this that holds, within a frame. But it reverses the moment you cross a frame boundary.
When you zoom from apartment to building, you do not add one layer of context. You introduce a new class of constraints and a new set of actors whose priorities sometimes contradict the apartment-level rules. The agent has no mechanism for arbitrating between conflicting frame-level inputs. It tries to resolve the conflict using the rules it has. Those rules were written for a different scale.
The practical consequence: the more a fully automated system scales up in scope, the more it becomes an echo chamber of its own initial assumptions. It engages with what it was trained to expect. Inputs that fit the mould get processed. Inputs that don’t fit the mould get either forced into the nearest available category or excluded entirely.
This is the hidden cost of full automation. An agent operating in a narrow frame fails visibly, and you can correct it. An agent operating in a wide frame fails silently, by quietly excluding everything its frame cannot accommodate. The output looks coherent. The gaps are not gaps - they are absences. You would not know to look for them.
The Echo Chamber Is Not a Bug
The instinct is to treat this as something that better training will fix. I am not sure it will.
An agent can only engage with what is inside its wheelhouse. That wheelhouse was defined by whoever built the system and by whatever data shaped it. Authentic human engagement - the kind that produces unexpected surface area, where you end up in a conversation you did not plan for that genuinely changes your thinking - is structurally incompatible with a fully closed loop.
The randomness of genuine human impulse is not noise. It is the mechanism by which the frame gets updated. The experience of waking up and finding yourself unexpectedly in an argument with people you have never met about the philosophy of mathematics is not inefficiency (this happened to me yesterday). It is the system working. It creates new context that the agent, operating within its predefined scope, would never have surfaced.
The fully automated version of that person produces content that stays within expected themes, responds to expected inputs, and misses every oblique angle that would have stretched the frame. The output is higher in volume and lower in information.
What Stays Human
Taste and judgment get invoked constantly in discussions about AI, usually as a placeholder for the category of things we haven’t figured out how to automate yet. I think this underestimates what they actually are.
Taste and judgment are the skills capable of evaluating themselves. That is why they compound in a way that pure knowledge accumulation does not. An agent can accumulate facts about what content performs well. It cannot evaluate whether the frame in which performance is measured is the right frame. It cannot decide that the metric itself is wrong and that a different metric would produce better outcomes. Those are frame-level decisions.
An agent can also be wrong about something within a frame and correct itself with more data. It cannot be wrong about the frame and correct itself, because the tools it uses for correction are inside the frame it would need to evaluate. This is not a software version problem. It is a structural feature of what it means to operate within a frame at all.
The implication for anyone building with agents is fairly direct: the human in the loop is the mechanism that evaluates and updates the frame. Remove that function entirely and the system optimises intelligently within a box it can never question.
Design Follows from This
If the frame problem is structural rather than technical, it changes what good system design looks like.
The first question is what to zoom into. The tighter the frame, the more stable the context, the more effective the agent. This is not a limitation to work around - it is a feature to exploit deliberately. An agent that is brilliant within a very tight scope is more useful than an agent that is mediocre across a wide one.
The second question is where the frame boundary is. At what point does the agent’s confidence break down? At what point does the relevant context expand beyond what the system was designed for? That boundary is where human judgment enters. The design has to make that entry point explicit rather than hoping the agent handles it gracefully. It won’t.
The third question is what feeds the frame. If the human is the mechanism for updating the frame, then the quality and variety of human inputs matters significantly. A system that filters all inputs through an automated layer before they reach the human is degrading the frame-update function, not improving it. The agent handles the work within the frame. The human updates the frame. Those are distinct jobs, and conflating them is how you end up with a system that runs efficiently toward the wrong destination.
The best version of building with agents is this: commit to the close range, respect the frame boundary, and preserve the function that evaluates the frame itself. Everything after that is implementation detail.
If this piece added something to your week, please consider subscribing :)