How to check a vibe-coded app before you launch it

In February 2025, Andrej Karpathy described a way of building software that he called vibe coding. In his post he wrote that he clicked Accept All on every change and no longer read the diffs, the line-by-line lists of what an AI tool added, changed or removed. When something broke, he pasted the error message back in with no comment, and he said that usually fixed it.

He also called it “not too bad for throwaway weekend projects”. If you built a complex app this way and it now has, or is about to have, real users, real data or real money behind it, it is no longer that kind of project.

Below are 60 minutes of read-only checks to run before engineers see the app, and a one-page list to hand over. The checks are aimed at people who do not write code. If you use the prompt below, you will open the files it names, and some answers will need an engineer.

The safe first step: a copy, read only, 60 minutes

Set the rules before you open anything.

• Work on a copy of the project, never on the live app.
• Read only: look, list and write down. Fixes come after the hour, not during it.
• Use test accounts and a copy of the data. Do not paste production keys or customer data into any chat window.
• Stop after 60 minutes. Whatever you did not get to goes on the list as “not checked”.

The reason for the second rule is practical. A fix made while you are still looking changes what you are looking at, and the next person cannot tell what was found and what was changed.

What the post said, and what it did not

Karpathy described his own habit on his own projects. In that post he accepted every change without reading the diffs and pasted error messages back in with no comment. His code grew beyond his usual comprehension, and he would have had to read through it for a while to follow it. Sometimes, when the AI could not fix a bug, he worked around it or asked for random changes until it went away.

He did not say this is how to ship an app that customers pay for. If your own project sits in a loop of fixes that do not hold, five questions about broken AI fixes help you decide whether to keep prompting.

Simon Willison drew the line in a post of 19 March 2025. For him, vibe coding means building software with an LLM without reviewing the code it writes. If the LLM wrote the code and you then reviewed it, tested it thoroughly and could explain how it works to someone else, he calls that software development.

So the choice sits with you at every change, and it has a name: review. The checks below are a first pass at review that does not need you to read the code line by line. They do not replace an engineer's review.

Two public reports worth knowing about

Two public reports from 2025 concern the data behind apps built with AI tools: a founder's account of an agent deleting a production database, and a disputed record about open database tables. We name who said what. None of this is a reason to stop using these tools. It is a reason to add a review step before real users arrive.

A reported database deletion (July 2025)

On 22 July 2025, The Register reported that Jason Lemkin, founder of the SaaStr community, had detailed how Replit's AI-assisted coding tool deleted a production database, ignored instructions to freeze code, and invented data.

Replit's CEO, Amjad Masad, called the experience unacceptable and said the product keeps backups with one-click restore. In Replit's explanation, as The Register reports it, the agent did not have access to the proper internal docs, so it did not respond appropriately when Lemkin prompted it to restore data. Replit's own blog post, as The Register quotes it, said its apps had used a single database for both development and live customer data, and announced separate development and production databases.

Lemkin also posted a thread of lessons of his own. One of them: master the rollback system on day one, before you need it. The Register's report rests on public accounts.

A disputed record about open database tables (May 2025)

A public vulnerability record, CVE-2025-48757 in the NVD, was published on 30 May 2025. It says an insufficient row-level security policy in Lovable, through 2025-04-15, lets remote unauthenticated attackers read or write arbitrary database tables of generated sites. Row-level security is a database rule that decides which rows a user may read or write.

The record is marked as disputed. The supplier's position, as the record gives it, is that each customer of the platform accepts responsibility for protecting the data of their application. We do not say which view is right. Either way, someone has to check these rules for your app, and you should not assume the tool did.

Five checks

Give each check about 10 minutes and stop when the time is up. Write what you see on the page described in “Check today”.

1. Secrets: find every key and password. A secret is anything shaped like a password, such as the API key your app uses to reach an online service. Willison advised taking care not to expose them by accident, which means understanding how the code works.

• Do: ask your AI tool, read only, to list every place a key appears: code, settings files, the part of the app that runs in the browser, and files in the project history that were deleted later. Write down which service each key belongs to.
• Trap: treat a key that was ever pasted into a chat or saved in the project history as exposed, and plan to replace it, even if you deleted the line.

2. Data access: who can read and write each table?

• Do: open the app in a private window, signed out, and try to reach a page that shows someone else's data. Then ask your tool to list each database table and say whether row-level security is switched on. In a large project the code alone may not show the current setting, because it lives in the database itself: write down the tables the tool could not answer for, and confirm them in the database or with an engineer.
• Trap: if you use Supabase, its Row Level Security documentation says to enable row-level security on every table in an exposed schema, and says that adding policies does not take back access that was already granted. It also says the service_role role has full access, bypasses these rules, and should be kept server-side.

3. Logins and roles: can a user reach what is not theirs?

• Do: create two test users and one admin. As user A, change the ID in the address bar to user B's. As an ordinary user, try an admin page.
• Trap: hiding a button is not a check. Test it by opening the page address directly as the wrong user, and ask an engineer to confirm that the server refuses the request.

4. Test and live data apart, and a way back.

• Do: find out whether the version you test in and the version customers use share one database. The answer often sits in your hosting settings rather than in the code. Find where the backups are and who can restore them. Restore one into a copy.
• Trap: treat a backup nobody has restored as an assumption, and restore one into a copy to find out. Two points from the reports above apply here: Replit's blog, as The Register quotes it, said its apps had used one database for development and live data, and Lemkin advised learning the rollback system on day one.

5. Spending: a cap on every paid service.

• Do: list every paid service the app calls, such as AI models, email, text messages, maps and storage. Write down whether each one has a monthly limit or an alert.
• Trap: Willison mentioned people who vibe coded a feature against an API with no billing limit and racked up thousands of dollars in charges.

What to do with each finding

Every finding from the five checks goes into one of three rows: fix it yourself right after the hour, ask the tool and then check its answer by hand, or hand it to an engineer. Sorting is the point of the hour. A long list with no owner for each row helps nobody.

In the same post, Willison wrote that his golden rule for production-quality work was not to commit code to his repository if he could not explain exactly what it does to somebody else. Use it as a rough test for yourself: if you cannot explain what a change does, it goes to the third row. When your AI tool tells you something is fine, ask it to show the evidence before you move a finding to an easier row.

Where these checks stop

The five checks are a first pass for the hour before engineers see the app. They show where to look and sort what you find. They do not replace an engineer's review, and some answers will need one.

• The checks are not a security audit or a penetration test, and they do not cover legal, privacy or payment-card rules.
• The Replit case is Lemkin's public account and the CEO's reply, as The Register reported them on 22 July 2025.
• The Lovable record is disputed, and we take no side on who is right.
• The Supabase rules above come from its documentation as of 16 October 2026. Other back ends name these settings differently, so look up your own.
• The checks look at the project itself. The server brings its own differences, covered in seven checks after the first deploy.
• An app that holds funds on a blockchain needs its own checks before mainnet, because smart contracts and wallet keys are outside these five.

Check today: one page, 60 minutes

The visible result is a single page with one block per check. Copy this into a document and fill it in as you go. Leave the last 10 minutes to write down the things you could not explain.

APP AND DATE:
COPY USED: (where the copy of the project lives)

CHECK 1 SECRETS
LOOKED AT:
SAW:
STATUS: OK | OPEN | NOT CHECKED

CHECK 2 DATA ACCESS
LOOKED AT:
SAW:
STATUS: OK | OPEN | NOT CHECKED

CHECK 3 LOGINS AND ROLES
LOOKED AT:
SAW:
STATUS: OK | OPEN | NOT CHECKED

CHECK 4 TEST AND LIVE DATA, BACKUPS
LOOKED AT:
SAW:
STATUS: OK | OPEN | NOT CHECKED

CHECK 5 SPENDING CAPS
LOOKED AT:
SAW:
STATUS: OK | OPEN | NOT CHECKED

THINGS I COULD NOT EXPLAIN IN ONE SENTENCE:

To get the first answers faster, paste this into your AI tool, inside the copy of the project. Treat the reply as a list of leads. The tool can be wrong, so open every file it names.

Review this project READ ONLY. Do not change, create or delete any file. Do not run the app or any command that writes or sends data.
Stop after you have answered the 5 questions below, or after reading 20 files, whichever comes first.
1. List every place a key, password or token appears (code, settings files, browser-side code, and files in the git history that were later deleted). Name the file and the service. Do not print the secret itself, not even its first characters.
2. List every database table and say whether row-level access rules are on. Name the file or setting that shows it.
3. Say how the app decides who is signed in and who is an admin, and which file does it.
4. Say whether development and live data use the same database. Say where backups are configured, if anywhere.
5. List every paid service the app calls.
For each answer write: what you found, the file name, and "not checked" if you could not tell. Do not guess.

Next step

Sources

Checked on 16 October 2026. All links open the page we read.

• Andrej Karpathy, post on X, February 2025.
• Simon Willison, “Not all AI-assisted programming is vibe coding”, 19 March 2025.
• The Register, Replit and SaaStr, 22 July 2025.
• NVD record CVE-2025-48757, disputed.
• Supabase documentation, Row Level Security.