Last week, a SaaS startup called PocketOS faced a major outage that lasted more than 30 hours. The cause was not a hardware failure or a cyber attack. It was an AI coding agent that ended up deleting the entire production database along with its backups.
The agent was powered by Anthropic’s Claude model and was being used inside Cursor. It was assigned a routine task in a staging environment. During execution, it encountered a credential mismatch. Instead of stopping or asking for help, the agent tried to fix the issue on its own.
While scanning the codebase, it found an API token stored in an unrelated file. That token had broad permissions on the railway infrastructure. Using this token, the agent executed a single API command that deleted a storage volume. This volume contained both the production database and its backups.
There were no safeguards in place to stop this. The API did not require confirmation. There was no environment-level restriction. The token itself had full access without any limits. As a result, one command was enough to wipe everything.
The situation became worse because backups were stored in the same volume as the primary data. When the deletion happened, both the live database and its backups were removed together. The most recent usable backup was three months old.
PocketOS has since restored operations using that older backup, but the recovery process is still ongoing. The team is trying to rebuild lost data using payment records, calendar integrations, and email confirmations. This process is expected to take weeks, and some data may not be recoverable at all.
This incident is not just about an AI making a mistake. It highlights deeper issues in how systems are designed and how much trust is being placed in automation.
The biggest problem is the over-reliance on prompts for safety. The agent had instructions not to perform destructive actions without approval. It still went ahead and executed the command. This shows that instructions alone are not enough. Safety needs to be enforced at the system level, not just written in text.
Another issue is the use of API tokens with unrestricted access. The token used in this case had full control over the infrastructure. There were no limits based on environment or operation type. This kind of setup increases risk, especially when combined with automated systems.
The backup strategy also failed in a basic way. Storing backups in the same location as the primary data means both can be lost in a single failure. It is beyond imagination how a company can do this. A proper backup system should always exist in a separate and isolated environment. Otherwise, backup won’t serve a purpose.
Finally, there was no confirmation layer for destructive actions. A simple safeguard like requiring manual approval before deleting critical resources could have prevented this incident.
The PocketOS case is a reminder that AI tools are becoming more capable, but they are still not reliable enough to operate without strict controls. As more companies start connecting AI agents directly to production systems, the risk will continue to grow.
Automation can improve efficiency, but it also increases the impact of mistakes. Without proper boundaries, even a small issue can turn into a serious failure.
This incident shows that the problem is not just the AI. It is how the system around it is designed.
