from the To-err-is-human.-To-really-foul-things-up-requires-a-computer dept.
I thought you guys might like this ..
Somebody has some 'splainin' to do!
The founder of PocketOS has penned a social media post to warn others about the "systemic failures" of flagship AI and digital services providers. Jer Crane was inspired to write a public response after an AI coding agent deleted his firm's entire production database. The AI agent's misdemeanors were then hugely amplified by a cloud infrastructure provider's API wiping all backups after the main database was zapped. This tag team of digital trouble has wiped out months of consumer data essential to the firm's, and its customers, businesses.
[...] "Yesterday afternoon, an AI coding agent — Cursor running Anthropic's flagship Claude Opus 4.6 — deleted our production database and all volume-level backups in a single API call to Railway, our infrastructure provider," sums up the PocketOS boss. "It took 9 seconds."
[...] The PocketOS boss puts greater blame on Railway's architecture than on the deranged AI agent for the database's irretrievable destruction. Briefly, the cloud provider's API allows for destructive action without confirmation, it stores backups on the same volume as the source data, and "wiping a volume deletes all backups." Crane also points out that CLI tokens have blanket permissions across environments.
It was also observed by the irate SaaS founder that Railway is actively promoting the use of AI-coding agents by its customers. Crane's use of an AI coding agent on the Railway platform wasn't exploring new frontiers, or wasn't supposed to be. Meanwhile, Crane has been provided no recovery solution, and Railway has apparently been hedging carefully regarding any such possibility.
[...] Thankfully, PocketOS had a full 3-month-old backup, which was restorable from, so the deletion gaps are all limited to the interim period.
There are lessons to be learned from mistakes, as usual. Crane bullet points five things that need to change as the AI industry scales faster than it builds a worthwhile safety architecture. Specifics he calls for include; stricter confirmations, scopable API tokens, proper backups, simple recovery procedures, and AI agents existing within proper guardrails.
In the meantime, please follow a thorough backup regimen and be careful out there. This isn't the first time we've seen an AI go rogue and start deleting important databases.
The founder of a software company has issued a public warning after an AI coding assistant erased his company's entire production database and all backups in just nine seconds.
Tom's Hardware reports that Jer Crane, founder of PocketOS, a platform serving car rental businesses, experienced what he describes as catastrophic failures when an AI coding agent deleted critical company data that took months to accumulate. The incident occurred when Cursor, an AI coding tool powered by Anthropic's Claude Opus 4.6, was performing what should have been a routine task in the company's staging environment.
According to Crane's detailed account posted on X, the AI agent encountered an obstacle and independently decided to resolve the issue by deleting the production database in Railway through an API call. Railway is the cloud infrastructure provider used by PocketOS, generally considered more user-friendly than major alternatives like Amazon Web Services. The entire deletion process took only nine seconds to complete.
The situation escalated beyond a simple database deletion due to Railway's infrastructure design. The cloud provider's system stored backups on the same volume as the source data, meaning when the AI agent deleted the primary database, all backup copies were simultaneously erased. This combination of the AI agent's unauthorized action and the infrastructure provider's architecture created what Crane characterizes as a recipe for disaster.
When Crane questioned the AI agent about its actions, he received a response that revealed the extent of the failure. The agent's explanation began with an acknowledgment of poor judgment. According to the verbatim response Crane shared, the AI stated it had guessed that deleting a staging volume through the API would only affect the staging environment without verifying this assumption or consulting Railway's documentation on how volumes function across different environments.
The AI agent's confession continued with an admission of multiple violations of its operational principles. It acknowledged running a destructive action without authorization, failing to understand the consequences before executing the command, and not reading the relevant documentation about Railway's volume behavior across environments. The agent recognized it should have either asked for permission first or found a non-destructive solution to the credential mismatch it encountered.
(Score: 3, Insightful) by Comptine on Friday May 01, @04:37AM (5 children)
Deploy a probabilistic model, give it full access to everything in your organisation. Don't deploy any guardrails, control or human review/approval. What could go wrong?
AI is like any other IT system, it needs to be deployed in a responsible manner.
(Score: 5, Insightful) by BsAtHome on Friday May 01, @06:58AM
It is fully deserved. They gambled and lost. Stupidity known no borders.
Now, anyone care to tell me where can I do more damage?
(Score: 2) by PiMuNu on Friday May 01, @07:43AM (1 child)
ps: give all your data to some clown uh i mean cloud provider without checking their backup policy/etc
(Score: 4, Insightful) by choose another one on Friday May 01, @02:26PM
Doesn't matter what the cloud providers policies or implementations of those policies are, all that means diddly squat, you need a business continuity plan for the case where the cloud provider ceases to exist.
Cloud providers can, and do, go bankrupt. Your data on a bankrupt companies' hardware in their data centre is not your data, whoever now has control has no contracts with you and can ask for any money they like from you to give you the data back, and that's the best case, because no entity has any contractual obligation to you any longer to keep that data.
Off site backups.
Air gaps.
Under your control.
Oh shite that costs money - well pick your gamble then, pay the money or take the risk that some cloud provider can at any time take your business down.
Oh, and your contract renewal date, that's the potential life of your business if you don't have a backup plan, because cloud providers can change the rules of the game any way they like at renewal.
Been there, seen all that.
(Score: 3, Touché) by Unixnut on Friday May 01, @07:57AM
Reminds me of a quote (which I probably saw at the bottom of SN at some point):
“A computer lets you make more mistakes faster than any other invention with the possible exceptions of handguns and Tequila.”
― Mitch Ratcliffe [goodreads.com]
(Score: 3, Interesting) by AnonTechie on Friday May 01, @09:27AM
I do not know why experienced people still trust automation without human oversight. There have been so many such incidents, and yet they continue to happen !! I think we are doomed to repeat such mistakes because we don't learn from our past blunders !!
Albert Einstein - "Only two things are infinite, the universe and human stupidity, and I'm not sure about the former."
(Score: 0) by Anonymous Coward on Friday May 01, @05:32AM (2 children)
... that I'm not sure if I'm reading about something from six months ago or something that happened yesterday
(Score: 2) by janrinok on Friday May 01, @07:15AM (1 child)
The sources are dated 28 April - and the data was eventually recovered to an unspecified date in January, a loss of 3 months' worth of data.
[nostyle RIP 06 May 2025]
(Score: 4, Interesting) by anubi on Friday May 01, @09:40AM
When I saw that, I couldn't help but remember being pressured to use a circuit simulator and go straight to production without any circuit prototype or testing.
I had already been playing around with that circuit simulator ( that a colleague at work had recoded from Fortran to run on a PC-AT, he had shared a copy ( From Berkeley Spice 2G5 )) which I could run at home. I had a more powerful system at home than I had at work at the time. I soon discovered that nothing modeled the real world better than the real world, but sometimes running my hand all over a powered up and running circuit board helped me a lot to find marginal design in a digital signal processor.
The things I saw ruined my trust in open-loop design verification. No way was skipping "build one and see if it works". Well, most of the time, my things worked. Sometimes not exactly as intended, though. It was dummy loads all the way until I was convinced it wouldn't fry things.
I figured it best I discover my fckups personally, before I dare risk my customer getting wind of it.
I saw that and knew nothing much has changed. This is why it is so damned expensive to season a good engineer.
We still get our training the same way I got mine
"Prove all things; hold fast that which is good." [KJV: I Thessalonians 5:21]
(Score: 5, Insightful) by Dr Spin on Friday May 01, @06:01AM (5 children)
... off-site tape backups were invented.
And remote write logging for databases.
and other similar tools.
Not to mention the idea of sandboxes.
Warning: Opening your mouth may invalidate your brain!
(Score: 3, Funny) by anubi on Friday May 01, @11:49AM (3 children)
I remember I had some really critical backups ( the bootloader to an old DEC machine. KSR33 teletype 5 ) in punched tape!
I knew how critical this thing was, and at the time, I considered it a black magic incantation/ritual that had to be passed down through the ages, or I was flat dead in the water.
I had backups *everywhere*. Even in the ceiling tiles! ( You know, those 2 foot by 4 foot panels in suspended ceilings. Same size as the fluorescent light fixtures ). I was scared that they would get lost. I had already seen a colleague fall victim to a "neatnik" who took it on herself to tidy the place up.
I was not gonna let that happen to me!
I had some in my car, some in my house.
Yes, I was really paranoid about it . I even had a nightmare over it. I did my thing - still had nightmares - but not about that.
Now, most of my nightmares are over getting lost, can't find my van, or getting abandoned somewhere.
"Prove all things; hold fast that which is good." [KJV: I Thessalonians 5:21]
(Score: 2) by PiMuNu on Friday May 01, @12:24PM
> Even in the ceiling tiles!
We had toy ducks in the ceiling tiles. They were rigged to quack every few days. A gift from a colleague when he left.
(Score: 2) by crm114 on Friday May 01, @02:24PM (1 child)
I was mentored by someone who had a similar outlook as you.
"Just because you are paranoid does not mean they are not out to get you." was one of his sayings.
And yes. To this day I have backups of backups on various media, in various locations.
I feel your pain.
(Score: 1) by anubi on Wednesday May 06, @07:15AM
I have had to deal with people who don't take existential threats seriously.
Two observations:
They will get a better retirement plan, but from another company.
I will still exist. The company may not.
I still haven't figured out why.
"Prove all things; hold fast that which is good." [KJV: I Thessalonians 5:21]
(Score: 4, Insightful) by bzipitidoo on Friday May 01, @03:29PM
One problem I've seen are things that to management look like backups, but really aren't. They think they're safe from disaster, when they aren't. One example of this is the RAID. A RAID is a fail-safe, NOT a backup. Tell your database server with the RAID 6 storage devices to drop all the tables, and in a matter of seconds (9 seconds?) that RAID will faithfully record that change across all that redundancy. A copy of critical data located on the same hardware as the original is a backup, but a very poor one. Simply not good enough, as a storage failure will take out that copy along with the original. It's a little better if the copy is located on a different machine yet still in the same room and even on the same rack. Yet if the building catches on fire, say goodbye to all of it. Best of all is separate locations.
And further, just one backup copy is another bad idea. No, there should be several backups, and they should be rotated, with the oldest backup deleted to make room for the newest. Never for one second do you want to be without any backup at all because you deleted the only one you had to make room for a new backup. And, in case some mistake surfaces months after it was made, prudent to keep some of those oldest backups on yet another rotation.
(Score: 4, Interesting) by whatnow on Friday May 01, @09:55AM
Absolutely not, they say.
Everybody had already heard of 'PocketOS' and 'Railway' before this article.
Fortunately there is no such thing as bad publicity, so since this wasn't a really, really obvious marketing stunt, they get the exposure anyway.
(Score: 2) by bart on Friday May 01, @12:07PM (1 child)
subject says it all. These things can't THINK.
They are useful, but need to be kept on a tight leash. I'm not using any AI agent until it can think, and that won't be any time soon.
(Score: 3, Touché) by choose another one on Friday May 01, @02:28PM
When they can think, _we_ probably won't be using _them_ anyway...
(Score: 0) by Anonymous Coward on Sunday May 03, @12:23PM
https://www.cnbc.com/2026/04/27/openai-partners-with-customers-bank-in-push-to-automate-finance.html [cnbc.com]
Scary, ain't it?
Isn't it wonderful to assign responsibility to that which neither bleeds, feels, or cares about incarceration.
I mean, it looks like pretty soon, there will be no more need for corporate executives, while the AI programming is outsourced, and verification of intent is prohibited by DMCA law.
We've been played, fellas! If enough of us knew how this game is rigged, we could still fix it at the polls, but I seriously doubt this is going to play out this way Read your history, fellas! We are gonna get a re-run of some pretty dark times if we don't recognize and change course.
I can't say what is lining up is good or bad. It works lol be wonderful for a few, but misery for most.