I watched another one of those Twitter clips yesterday where someone gave a CLI bot three words of direction and it supposedly coded an entire SaaS business while they made coffee.
Naturally the replies were full of people declaring software engineering dead by Q3.
Then you look at what happens when someone actually hooks one of these agent loops into an internal staging environment.
Immediate disaster.
Not a loud, dramatic failure where an alarm trips and everyone scrambles. It fails quietly. It fails the way a Roomba fails, where it somehow wedges itself behind the guest room nightstand at 2 AM and just keeps grinding its little plastic wheels into the baseboard until the battery cooks itself.
The base models are actually pretty good at tool calling and writing Python scripts, so that part isn’t the scam. The scam is pretending the five-minute screencast on a pristine SQLite database bears any resemblance to reality. Real production is a swamp. You have three background workers hammering the same legacy endpoints, an API key with a corporate card attached, and a security lead who will lock your laptop in a drawer if you break compliance.
Nobody selling agent frameworks wants to talk about day four.
On day one with fresh context, the thing looks like a miracle worker. By day three, after chewing through eight thousand lines of unformatted logs and getting its own previous outputs fed back into the prompt, the wheels come off. It drifts. You know how a work Slack channel starts at 10 AM with someone asking if the cafeteria is open and by 1:45 PM two people are sending Wikipedia links about the Byzantine Empire? Same thing. The bot doesn’t crash or throw a 500 error, it just gets weirdly confident about the wrong objective. You only catch it when a billing manager asks why an automated script wiped out six months of receivables for an account that churned in 2024.
And the bills.
If you leave an agent unsupervised with a looping retry logic on a flaky endpoint, you are basically handing an unlocked phone to a kid whose thumb is already hovering over in-app purchases. It hits an error, decides it should spin up four parallel sub-agents to brainstorm alternatives, and racks up three grand in token fees before you even get back from lunch. Nobody posts screen recordings of a twelve-hundred-dollar invoice to parse one bad CSV, but that is the reality if you do not pull the plug.
The whole setup is backwards because these tools get built capability-first and governance-never. If you want this stuff to work overnight without waking up to an incident review, you have to wrap the probabilistic model inside completely boring, rigid guardrails:
Put a Redis lock or a real database mutex on shared records. If two workers go after the exact same customer ticket at 3 AM, asking them nicely in the system prompt not to step on each other is just magical thinking.
Put hard kill switches on token spend and execution time. When an agent gets stuck in a loop, it should not try to philosophize its way out or gracefully adapt. It needs to die immediately and ping a human on call.
Write every external call to an independent audit log. When things go sideways, you cannot ask the LLM why it made that decision because it will just invent a polite bedtime story to explain the wreckage. You need the raw receipts.
None of that makes the model smarter, but it stops the model from turning your cloud bill into a crime scene.
Next time a founder shows you a slick demo, skip the questions about reasoning benchmarks. Ask them what happens when an API times out twice in a row, or whether two agents can overwrite the same database row at the same time. If their answer involves the phrase “well, the prompt tells it to be careful,” keep your credit card in your pocket.
I think despite some "subtle and perhaps strategic" segways, it provides a good perspective overall (nice read). I especially agree that the real engineering challenge isn't the model - it's the operational guardrails around it. The only nuance I'd add is that governance shouldn't just be reactive (kill switches, audit logs, mutexes); it also needs proactive controls like least-privilege identities, policy enforcement, and continuous validation of agent behavior. The demo is the easy part but, operating agents safely at scale is where the real work begins, and what most people are finding challenging. Nice Article.