Claude, Cursor, and Copilot are great at producing code that runs on the first try. The problem is what they fill in when they don't actually know the answer: auth becomes a TODO, the password becomes "temp123", the token gets console.logged, and the permission check is missing one await. It all passes your manual test, and it all ships. The Vibe Code Pack hunts the specific patterns these tools generate.
It's an overlay that runs on every scan, on top of the normal scanners. The checks target the exact shortcuts an assistant takes when it's pattern-matching instead of reasoning. A sample of what it flags:
None of these throw an error. They behave correctly in the demo and stay invisible until someone notices, or exploits, the gap. Here's what actually bites.
This is the nastiest one. An async auth check called without await returns a Promise, and a Promise is truthy. So if (checkAuth(req)) is always true. The route is wide open and the code reads like it's protected. AI tools drop the await constantly because both forms compile and the happy path looks identical.
A very human AI move: scaffold the route, leave // TODO: implement auth, and move on. Or worse, the auth middleware exists but is // app.use(authenticate) commented out from when someone was "just testing." The endpoint ships with no gate at all.
A session secret set to 'keyboard-cat', a JWT secret with a || 'changeme' fallback, a Postgres password of password in docker-compose. The assistant adds these "for dev convenience." If the real env var is ever missing in prod, the app silently falls back to the weak value and anyone who reads the public source can forge sessions.
A debug console.log(token) or console.log({ password }) left in after a fix. It looks harmless, but those lines end up in your hosting provider's log aggregator, your error tracker, and anywhere logs get shipped, in plaintext, retained for months.
"It works on my machine" is exactly what a missing await or a commented-out gate looks like. The demo passes because you logged in as a real user. The hole only shows up when an anonymous request hits the same route, and by then it's in production.
You ask the assistant to "add an auth check to this route." It does. It even looks right:
// AI-generated. isAuthenticated is async (it hits the session store). if (isAuthenticated(req)) { // returns a Promise, always truthy return handler(req, res); // every request is "authenticated" } // what it should have been: if (await isAuthenticated(req)) { // actually resolves to true/false return handler(req, res); } # You test it logged in. Works. You ship. # A scraper hits /api/admin with no cookie and walks straight in.
The Vibe Code Pack is a static overlay that runs on every scan, 100% offline. Your code never leaves your machine. It looks for known AI-generated patterns, so it catches common cases, not every possible bug:
isAuthenticated() used in an if() or called bare with no await in front, flagged Critical because the Promise is always truthy.// TODO/FIXME: add auth comments in code files, and commented-out auth middleware like // app.use(passport).process.env.X || 'changeme' fallbacks, and default DB passwords in docker-compose.console.log calls that print a password, token, API key, or credential, with the keyword matched on word boundaries so it won't fire on tokenCount.jwt.decode() used where jwt.verify() belongs, SQL built by string concatenation, MD5/SHA-1 for passwords, Math.random() for tokens, and CORS set to reflect the origin.if as a bug.|| 'changeme' with a startup check that throws if the env var is absent.console.log before you push.Scan your code and full git history in under a minute. Free verdict, no signup, nothing leaves your machine.