How to check a vibe-coded app before you launch it

You built the app by clicking Accept All, the button in an AI coding tool that applies every change without you reading the code. Now real users, data or money are about to go through it. Before customers or engineers see it, spend 60 minutes on a copy of the project, read only: you look and write down, you change nothing. Five checks fill the hour: keys, who can read your data, logins, test versus live data with backups, and spending. Each finding then goes into one of three rows, and you keep a one-page list to hand over.

Andrej Karpathy described this way of building in a February 2025 post and called it vibe coding. He wrote that he clicked Accept All on every change and no longer read the diffs, the line-by-line lists of what an AI tool added, changed or removed. When something broke, he pasted the error message back in with no comment, and he said that usually fixed it. He also called it “not too bad for throwaway weekend projects”. If you built a complex app this way and it now has, or is about to have, real users, real data or real money behind it, it is no longer that kind of project.

The checks are aimed at people who do not write code. If you use the prompt below, you will open the files it names, and some answers will need an engineer.

How do you start without breaking anything?

Set four rules before you open anything.

• Work on a copy of the project, never on the live app.
• Read only: look, list and write down. Fixes come after the hour, not during it.
• Use test accounts and a copy of the data. Do not paste production keys or customer data into any chat window.
• Stop after 60 minutes. Whatever you did not get to goes on the list as “not checked”.

The second rule is practical. A fix made while you are still looking changes what you are looking at, and the next person cannot tell what was found and what was changed.

Which five checks, and how can each one fool you?

Give each check about 10 minutes and stop when the time is up. Write what you see on the one-page list at the end.

Check 1: are keys or passwords saved in the project?

A secret is anything shaped like a password, such as the API key your app uses to reach an online service. In a post of 19 March 2025, Simon Willison advised taking care not to expose them by accident, which means understanding how the code works. Ask your AI tool, read only, to list every place a key appears: code, settings files, the part of the app that runs in the browser, and files in the project history that were deleted later. Write down which service each key belongs to.

Adding a file to .gitignore after you saved it does not take the key out. On 29 October 2026 we tried it on a scratch repository with git 2.50.1, the tool that saves every version of a project. We committed a .env settings file with a demo key, then added .env to .gitignore, the list of files git leaves alone, and committed that.

The key stays in the history until the history is rewritten. A second scratch repository the same day showed the rest. After git rm --cached .env and a commit, the file was still on disk, and git log still listed the commit that first added it. In one more repository, where .env was in .gitignore before the first commit, git ls-files and git log printed nothing for it.

So treat a key that was ever pasted into a chat or saved in the project history as exposed, and plan to replace it, even if you deleted the line.

We also ran a search on a demo project we made for the check, on a Mac, the same day. It looked for setting names starting with NEXT_PUBLIC_ or VITE_ and found NEXT_PUBLIC_STRIPE_SECRET_KEY: the name says secret key, and it is bound for the browser. That is the finding this check is for.

On that Mac the shell's grep was a function wrapping another tool, ugrep, and it skipped hidden files such as .env. The system grep, /usr/bin/grep, listed them. If a search finds no .env file you know is there, ask for /usr/bin/grep.

Check 2: who can read and write your data?

Open the app in a private window, signed out, and try to reach a page that shows someone else's data. Then add something as one test account and sign in as another. If the second account sees it, write that down.

Those tests only show what the pages let through. Next, ask your tool to list each database table and say whether row-level security is switched on. Row-level security is a database rule that decides which rows a user may read or write. In a large project the code alone may not show the current setting, because it lives in the database itself. Write down the tables the tool could not answer for, and confirm them in the database or with an engineer.

If you use Supabase, its Row Level Security documentation says to enable row-level security on every table in an exposed schema, and says that adding policies does not take back access that was already granted. It also says the service_role role has full access, bypasses these rules, and should be kept server-side.

A public record, CVE-2025-48757 in the NVD, was published on 30 May 2025. It says an insufficient row-level security policy in Lovable, through 2025-04-15, lets remote unauthenticated attackers read or write arbitrary database tables of generated sites. The record is marked as disputed. The supplier's position, as the record gives it, is that each customer of the platform accepts responsibility for protecting the data of their application.

We do not say which view is right. Either way, someone has to check these rules for your app, and you should not assume the tool did.

Check 3: can a user reach what is not theirs?

Create two test users and one admin. As user A, change the ID in the address bar to user B's. As an ordinary user, try an admin page.

Hiding a button is not a check. Test it by opening the page address directly as the wrong user, and ask an engineer to confirm that the server refuses the request.

Check 4: are test and live data apart, and is there a way back?

Find out whether the version you test in and the version customers use share one database. The answer often sits in your hosting settings rather than in the code. Find where the backups are and who can restore them, then restore one into a copy and look for your newest record. A backup nobody has restored is an assumption.

A public report shows why. On 22 July 2025, The Register reported that Jason Lemkin, founder of the SaaStr community, had detailed how Replit's AI-assisted coding tool deleted a production database, ignored instructions to freeze code, and invented data. The report rests on public accounts.

Replit's CEO, Amjad Masad, called the experience unacceptable and said the product keeps backups with one-click restore. In Replit's explanation, as The Register reports it, the agent did not have access to the proper internal docs, so it did not respond appropriately when Lemkin prompted it to restore data. Replit's own blog post, as The Register quotes it, said its apps had used a single database for both development and live customer data, and announced separate development and production databases. Lemkin also posted a thread of lessons of his own. One of them: master the rollback system on day one, before you need it.

So two public reports from 2025 concern the data behind apps built with AI tools: a founder's account of an agent deleting a production database, and the disputed record about open database tables in check 2. We name who said what. None of this is a reason to stop using these tools. It is a reason to add a review step before real users arrive.

Check 5: what is the most this app can spend?

List every paid service the app calls, such as AI models, email, text messages, maps and storage. Write down whether each one has a monthly limit or an alert, and who reads the alert. Willison mentioned people who vibe coded a feature against an API with no billing limit and racked up thousands of dollars in charges.

Tests count too. When an AI coding agent writes tests that call a paid AI model, every run is billed. On one of our projects the bill for one day of such tests was $1,000. Give the tests their own key with a spending limit, and agree the cost of a test run before it starts.

Which prompt does the digging for you?

To get the first answers faster, paste this into your AI tool, inside the copy of the project.

Review this project READ ONLY. Do not change, create or delete any file. Do not run the app or any command that writes or sends data.
Stop after you have answered the 5 questions below, or after reading 20 files, whichever comes first.
1. List every place a key, password or token appears (code, settings files, browser-side code, and files in the git history that were later deleted). Name the file and the service. Do not print the secret itself, not even its first characters.
2. List every database table and say whether row-level access rules are on. Name the file or setting that shows it.
3. Say how the app decides who is signed in and who is an admin, and which file does it.
4. Say whether development and live data use the same database. Say where backups are configured, if anywhere.
5. List every paid service the app calls, and what stops one visitor or one test run from running up a bill.
For each answer write: what you found, the file name, how sure you are (certain or likely), and "not checked" if you could not tell. Do not guess.

Treat the reply as a list of leads. The tool can be wrong, so open every file it names and repeat the private-window tests yourself. A tool that reports no secrets has searched, not proved.

How do you sort what you find?

Every finding goes into one of three rows: fix it yourself right after the hour, ask the tool and then check its answer by hand, or hand it to an engineer. Sorting is the point of the hour. A long list with no owner for each row helps nobody.

In the same post of 19 March 2025, Willison wrote that his golden rule for production-quality work was not to commit code to his repository if he could not explain exactly what it does to somebody else. Use it as a rough test for yourself: if you cannot explain what a change does, it goes to the third row. When your AI tool tells you something is fine, ask it to show the evidence before you move a finding to an easier row.

Keep the app closed to customers while a table is open to signed-out visitors or other customers, or while a key that was ever saved in the project is still in use.

What did Karpathy and Willison actually say?

Karpathy described his own habit on his own projects. His code grew beyond his usual comprehension, and he would have had to read through it for a while to follow it. Sometimes, when the AI could not fix a bug, he worked around it or asked for random changes until it went away. He did not say this is how to ship an app that customers pay for. If your project sits in a loop of fixes that do not hold, five questions about broken AI fixes help you decide whether to keep prompting.

Willison drew the line in that post of 19 March 2025. For him, vibe coding means building software with an LLM, the kind of AI model behind your coding tool, without reviewing the code it writes. If the LLM wrote the code and you then reviewed it, tested it thoroughly and could explain how it works to someone else, he calls that software development. So the choice sits with you at every change, and it has a name: review. The five checks are a first pass at review that does not need you to read the code line by line.

Where these checks stop

The checks are a first pass for the hour before engineers see the app. They show where to look and sort what you find. They do not replace an engineer's review, and some answers will need one.

• They are not a security audit or a penetration test, and they do not cover legal, privacy or payment-card rules.
• The Replit case is Lemkin's public account and the CEO's reply, as The Register reported them on 22 July 2025.
• The Lovable record is disputed, and we take no side on who is right.
• The Supabase rules above come from its documentation as of 16 October 2026. Other back ends name these settings differently, so look up your own.
• The checks look at the project itself. The server brings its own differences, covered in seven checks after the first deploy.
• An app that holds funds on a blockchain needs its own checks before mainnet, because smart contracts and wallet keys are outside these five.
• How far an app must go depends on who can open it, what data it holds and whether money moves; the four levels of launch checks put this review at level 3.

Check today: one page, 60 minutes

The visible result is a single page with one block per check. Copy this into a document and fill it in as you go. Leave the last 10 minutes to write down the things you could not explain.

APP AND DATE:
COPY USED: (where the copy of the project lives)

CHECK 1 SECRETS
LOOKED AT:
SAW:
STATUS: OK | OPEN | NOT CHECKED
ROW: FIX AFTER THE HOUR | ASK THE TOOL AND RE-CHECK | ENGINEER

CHECK 2 DATA ACCESS
LOOKED AT:
SAW:
STATUS: OK | OPEN | NOT CHECKED
ROW: FIX AFTER THE HOUR | ASK THE TOOL AND RE-CHECK | ENGINEER

CHECK 3 LOGINS AND ROLES
LOOKED AT:
SAW:
STATUS: OK | OPEN | NOT CHECKED
ROW: FIX AFTER THE HOUR | ASK THE TOOL AND RE-CHECK | ENGINEER

CHECK 4 TEST AND LIVE DATA, BACKUPS
LOOKED AT:
SAW:
STATUS: OK | OPEN | NOT CHECKED
ROW: FIX AFTER THE HOUR | ASK THE TOOL AND RE-CHECK | ENGINEER

CHECK 5 SPENDING CAPS
LOOKED AT:
SAW:
STATUS: OK | OPEN | NOT CHECKED
ROW: FIX AFTER THE HOUR | ASK THE TOOL AND RE-CHECK | ENGINEER

THINGS I COULD NOT EXPLAIN IN ONE SENTENCE:

Next step

Sources

Checked on 16 October 2026. All links open the page we read. The git and search runs in check 1 are our own, from 29 October 2026.

• Andrej Karpathy, post on X, February 2025.
• Simon Willison, “Not all AI-assisted programming is vibe coding”, 19 March 2025.
• The Register, Replit and SaaStr, 22 July 2025.
• NVD record CVE-2025-48757, disputed.
• Supabase documentation, Row Level Security.