Most of the "agents" running in CS orgs right now aren't agents.
They're automated tasks that happen to run on an LLM.
I get why. Every vendor is selling agents. Every board deck has a slide. Your team is under pressure to show something running, so they build something that runs — a workflow that drafts the QBR summary, flags the at-risk accounts, writes the renewal email. It demos beautifully.
Then you check on it three weeks later.
Here's the test I've started using. An agent earns the name by learning while it works. You give it feedback on Tuesday's task, and Thursday's task comes back better. It captures memories. It connects to your systems. It runs without you standing over it.
No feedback loop? You've built an automation that burns tokens.
And those go stale fast. The report format drifts, the source data changes, and you're back in the tool fixing the thing you just built.
I call it agent babysitting.
(naming it made me feel slightly better about how much of it I've done.)
The static thing never gets more valuable. The learning thing compounds. That's the whole difference.
My friend Justin Chappell describes three stages of agent deployment. Every agent should move through them in order.
Stage one: human in the loop. Nothing consequential runs without you verifying it first. The agent drafts, you correct, it learns. Reviewing every output feels like it defeats the purpose — it doesn't. You're training, not checking. Every correction is one you'll never make again. You graduate when the corrections dry up.
Stage two: human on the loop. It executes on its own. You audit — not every run, just samples, the way a good manager reviews a rep's calls. You're looking for drift: the edge case it hasn't seen, the confident answer that's confidently wrong. You graduate when the audits stop surprising you.
Stage three: autonomous. You've proven it. You've audited it. You let it run — and move your attention to the next thing.
Stage three is where the magic happens. Most agents never arrive.
Not because the models aren't ready. Because someone built a task with no way to learn — and a task can't be promoted. Nobody's training it. They're just babysitting it.
Here's the exercise to run with your team this week. List every "agent" in your stack. Two questions each:
When it gets something wrong, where does the correction go? If the answer is "someone edits the prompt and hopes," it's a task.
What stage is it in — and what evidence moves it to the next one?
If nothing on the list has a path to stage three, you don't have an agent strategy. You have a backlog of future babysitting.
Build things that learn. Everything else is maintenance.
