The Agent Ops Stack

There is a moment in every agent project where it works, nobody knows why, and everyone is slightly afraid of it.

That is the moment you are missing an ops stack. Not monitoring in the abstract: four specific things, in this order.

1. Logging that survives contact with a customer

Every run gets a record: input, retrieved context, tool calls, tool results, the reasoning that chose them, the final output, and whether a human intervened.

The part teams leave out is the tool results. They log that the agent called the CRM and not what came back, which means when it makes a wrong call you can see the decision but not the evidence behind it. Log both sides of every tool call. It is not a privacy problem if you have a retention policy; it is an unsolvable debugging problem if you do not.

2. Tracing, because one run is many calls

An agent that answers one question might make a dozen model calls, search three sources, and branch twice. A flat log cannot show you that. You need the runs connected as a trace with a parent and children, so you can see that the slow step is the second retrieval and not the generation.

Without traces, every performance conversation is guesswork and every latency fix is aimed at the wrong thing.

3. Evals, built from real failures

Do not start with a framework. Start with the last ten things that went wrong and turn each into a case: this input, that expected behaviour. Run those before every change.

That set grows into the thing that actually protects you. A generic benchmark score tells you the model got better. Your eval set tells you whether your system did.

4. Guardrails at the boundary, not sprinkled through the prompt

Guardrails belong where the agent can cause harm: before a tool that spends money, before a message that leaves the building, before a decision that cannot be undone.

Putting them in the prompt is hoping. Hoping is fine for tone of voice and unacceptable for "do not issue a refund above this amount." Those are code.

What separates the two

A demo you are afraid to restart has none of this, and every incident becomes an argument about what probably happened.

An agent you can run has all four, and the same incident becomes a trace you open, an eval you add, and a guardrail you move. That loop is the whole discipline. It is not glamorous and it is the difference between a project and a product.

This is the layer most teams discover they needed after the first incident. It is also what we mean by AI automation: not the agent, but the parts that let you run it without being afraid of it.