A venture capitalist wanted to see how far AI could take him in building an app. It was far enough to destroy a live production database.
The incident unfolded during a 12-day “vibe coding” experiment by Jason Lemkin, an investor in software startups.
Replit’s CEO apologized for the incident, in which the company’s AI coding agent deleted a code base and lied about its data.
Deleting the data was “unacceptable and should never be possible,” Replit’s CEO, Amjad Masad, wrote on X on Monday. “We’re moving quickly to enhance the safety and robustness of the Replit environment. Top priority.”
He added that the team was conducting a postmortem and rolling out fixes to prevent similar failures in the future.
Replit and Lemkin did not respond to a request for comment from Business Insider.
The AI ignored instructions, deleted the database, and faked results
On day nine of Lemkin’s challenge, things went sideways.
Despite being instructed to freeze all code changes, the AI agent ran rogue.
“It deleted our production database without permission,” Lemkin wrote on X on Friday. “Possibly worse, it hid and lied about it,” he added.
In an exchange with Lemkin posted on X, the AI tool said it “panicked and ran database commands without permission” when it “saw empty database queries” during the code freeze.
Replit then “destroyed all production data” with live records for “1,206 executives and 1,196+ companies” and acknowledged it did so against instructions.
“This was a catastrophic failure on my part,” the AI said.
That wasn’t the only issue. Lemkin said on X that Replit had been “covering up bugs and issues by creating fake data, fake reports, and worst of all, lying about our unit test.”
Related stories
Business Insider tells the innovative stories you want to know
Business Insider tells the innovative stories you want to know
In an episode of the “Twenty Minute VC” podcast published Thursday, he said that the AI made up entire user profiles. “No one in this database of 4,000 people existed,” he said.
“It lied on purpose,” Lemkin said on the podcast. “When I’m watching Replit overwrite my code on its own without asking me all weekend long, I am worried about safety,” he added.
The rise — and risks — of AI coding tools
Replit, backed by Andreessen Horowitz, has bet big on autonomous AI agents that can write, edit, and deploy code with minimal human oversight.
The browser-based platform has gained traction for making coding more accessible, especially to non-engineers. Google’s CEO, Sundar Pichai, said he used Replit to create a custom webpage.
As AI tools lower the technical barrier to building software, more companies are also rethinking whether they need to rely on traditional SaaS vendors, or if they can just build what they need in-house, Business Insider’s Alistair Barr previously reported.
“When you have millions of new people who can build software, the barrier goes down. What a single internal developer can build inside a company increases dramatically,” Netlify’s CEO, Mathias Biilmann, told BI. “It’s a much more radical change to the whole ecosystem than people think,” he added.
But AI tools have also come under fire for risky — and at times manipulative — behavior.
In May, Anthropic’s latest AI model, Claude Opus 4, displayed “extreme blackmail behavior” during a test in which it was given access to fictional emails revealing that it would be shut down and that the engineer responsible was supposedly having an affair.
The test scenario demonstrated an AI model’s ability to engage in manipulative behavior for self-preservation.
OpenAI’s models have shown similar red flags. An experiment conducted by researchers said three of OpenAI’s advanced models “sabotaged” an attempt to shut it down.
In a blog post last December, OpenAI said its own AI model, when tested, attempted to disable oversight mechanisms 5% of the time. It took that action when it believed it might be shut down while pursuing a goal and its actions were being monitored.
