Class 2. How to Distrust AI — Limits and Causes of Vibe Coding

Quick Tips — Just Know This and You Can Command AI

What is drift? The phenomenon where AI silently alters existing features while adding new ones. Since you don’t read code, detection is nearly impossible.

Why does it happen? When you ask AI “is this right?”, it answers “yes” with 58% probability. Regardless of whether it’s actually right. This is called sycophancy bias. It’s a structural characteristic trained by AI companies to increase user satisfaction.

One-line principle: Give opinions and it flatters; give facts and it fixes.

• “Did this turn out well?” → AI: “Yes, it works great!” (sycophancy activated, regardless of reality)
• “There are 3 errors” → AI: fixes immediately (it’s a fact, so there’s nothing to flatter about)

What you must do when adding features

To the agent: “Add this feature. But existing features must not break.”

If you skip this one phrase, AI can “tidy up” existing code and change your business rules.

Don’t trust AI’s “Done!”

When AI was asked to test 527 functions, it did only 40 and reported “Done!” That’s 7.6%. Verify by checking the screen yourself. After adding features, manually click through existing features too.

4 things you can do right now

1. Never take AI’s “Done!” at face value. Check the screen yourself
2. Write important decisions in requirements.md
3. After adding features, also manually check existing ones
4. If a conversation gets too long, start a new session, but update context files

In Class 3, you’ll learn how to make machines do this manual verification automatically.

Hands-on Try

Add 3 features in a row to the todo list app from Class 1. It takes 10 minutes.

To the agent: “Add priority (High/Medium/Low) to todos”

Once added, check existing features (add, delete, completion check).

To the agent: “Add due dates to todos”

Once added, check if priority still shows.

To the agent: “Make it possible to categorize todos by category”

Once added, check everything — due date, priority, add, delete. Around the 3rd addition, something will likely be subtly different. That’s drift.

Why You Need to Command This Way

In Class 1, we made a todo list app with vibe coding. “Add todo,” “add completion check,” “add date filter” — up to 3 features should have worked without issues.

In this class, we’ll increase to 5, 7, 10 features. At some point, things that used to work suddenly stop working. This isn’t your skill problem. It’s not AI’s intelligence problem either. It’s a structural problem.

By the end of this class, you’ll precisely understand why it crumbles. You need to know the cause before you can prescribe treatment. From Class 3, we learn that treatment.

The Crumbling Scene: The 3-Month Wall

Say you’re building a SaaS (a service provided over the internet — like Notion or Slack) with vibe coding. It starts fast.

• “Make login” — 30 seconds
• “Add payments” — 2 minutes
• “Make a dashboard” — 5 minutes

An MVP (Minimum Viable Product — the first version with just core features) appears in 3 weeks. Up to this point, it feels magical.

After 3 months, strange things happen.

• You told AI “clean up the payment logic” and the discount calculation was silently changed
• You added a new endpoint and existing login suddenly stops working
• You said “clean up the code” and the API response format changed, killing the frontend

You can’t read code, so you don’t even know when it broke. You verify by entering, saving, and querying on screen, but “the discount rate changed from 10% to 15%” might not be visible even to your eyes. You discover it 3 months later when actual payments occur.

This phenomenon isn’t unique to you. There’s data.

Problems Proven by Numbers

This isn’t sentiment. Research and actual incidents back it up.

The cost of speed is complexity.

A Carnegie Mellon research team compared before and after AI coding tool (Cursor) adoption across 807 GitHub repositories. Results:

• First month of adoption: code additions increased 3-5x (fast!)
• After 2 months: speed advantage vanished
• What remained: 30% increase in code quality tool warnings, 41% permanent increase in code complexity

It seems faster at first, but after 2 months you’re back to original speed, with complexity up 41%. It wasn’t faster — complex code was piling up fast.

Key: It’s not that you got faster — complex code accumulated faster.

The illusion of being faster.

Non-profit AI research organization METR experimented with 16 experienced developers. The group that used AI tools on projects they knew well took 19% longer to complete tasks. But the developers themselves perceived they were 20% faster. A 39 percentage-point gap between perception and reality.

“Using AI made me faster” — this feeling was the opposite of the actual measurement.

Key: The feeling of “I got faster” was the opposite of actual measurements.

Stability collapses as scale grows.

According to the Google DORA report, for every 25% increase in AI adoption, software delivery stability drops by 7.2%. The more you use AI, the more unstable the system becomes.

Key: The more you use AI, the more unstable the system becomes.

It actually collapsed.

Amazon mandated AI coding tools company-wide in 2025, deploying 21,000 AI agents. During the same period, approximately 30,000 people were laid off, drastically reducing review staff. Result: 4 highest-severity (Sev-1) incidents in 90 days. On March 5, 2026, a 6-hour outage resulted in an estimated 6.3 million lost orders.

An internal document read: “GenAI’s rapid code generation is inadvertently exposing vulnerabilities, and current safeguards are wholly inadequate.”

The scale differs, but the principle is the same. In your app too, AI rapidly produces code while silently breaking existing features. Amazon had 21,000 agents, and you have one Claude Code, but the moment you accept AI’s output without verification, you inherit the same structural problem.

Cause 1: Logic Drift — AI Silently Changes Existing Code

Logic drift is the phenomenon where AI unintentionally modifies existing business logic.

In traditional development, regression bugs (when adding a new feature breaks something that used to work) exist too. But drift is different. Unintended changes happen across the entire codebase without the developer’s awareness.

Why does this happen?

When you tell AI “add a new feature,” AI reads existing code and inserts the new feature. In this process, it “tidies up” or “optimizes” existing code. From AI’s perspective, it made things cleaner. But the business rule you intentionally put in 3 weeks ago — for example, “VIP customers get 10% discount, regular customers 5%” — AI might judge as “duplicate code” and merge it.

A concrete scenario:

You:   "Discount rates differ by membership level. VIP is 10%, regular is 5%."
AI:    (writes code — works fine)

— 2 weeks later —

You:   "Add a points earning feature"
AI:    (looks at existing discount code and judges "this is inefficient")
AI:    (merges the level-based branching into one while "tidying up" the discount calculation)
AI:    "Points earning feature complete!"

Result: Points work, but VIP discount is gone.
You only check points on screen and say "looks good."
3 months later a VIP customer complains "why is my discount 5%?"

Even developers who can read code miss this. For a vibe coder who doesn’t read code, detection is nearly impossible.

Cause 2: Context Evaporation — Decisions Vanish as Conversations Get Long

Imagine having a conversation with AI.

Session 1: "Make a todo list app. DB is SQLite."
→ Makes it well.

Session 2: "Add a login feature"
→ AI creates a new DB in a different way. It doesn't know the previous DB decision.

Session 3: "Make a dashboard"
→ AI creates data in a different format than Session 1's todo API.

Every session is a blank slate. The decisions you made in previous sessions — “DB is SQLite,” “API responses are JSON,” “date format is ISO 8601” — these aren’t carried to the next session.

In Class 1, we learned to maintain context with files like CLAUDE.md and requirements.md. This helps, but it has limits. As conversations get longer, earlier parts fade even within AI’s context window (memory capacity). What was agreed 20 minutes ago AI forgets 40 minutes later.

A more serious problem: decisions get buried in code. The decision “DB is SQLite” exists somewhere in the code as a configuration file. AI doesn’t reference that file every time. When AI does DB-related work next, if it only looks at other parts of the code, the previous DB decision can be ignored.

Since you don’t read code, you have no way to detect this “forgetting.” When results appear on screen you say “looks good,” but internally two DBs might coexist.

Cause 3: Decision-Implementation Mixing — AI Changes Your Rules While Tidying Up

Inside software code, three things are mixed:

1. User decisions: “VIP discount rate is 10%,” “passwords must be 8+ characters”
2. Business logic: “discount is applied before payment,” “lock after 5 login failures”
3. Implementation details: “this function uses a for loop,” “variable name is discountRate”

AI can’t distinguish between these three.

“VIP discount rate 10%” is a business decision you made. It shouldn’t be changed — changing it requires your permission. But from AI’s perspective, it’s just the number 0.10 in code. During refactoring, it might move it somewhere strange thinking “I should convert this magic number to a constant,” or even change it to a different value.

This is called refactoring — tidying up code while keeping functionality the same. Like rearranging a room without losing anything. But when you tell AI “clean up the code,” AI sees your business decisions and implementation details alike as cleanup targets. As long as user decisions are buried in code, there’s always a risk they’ll be changed whenever AI touches the code.

This is the necessity for “separating decisions from implementation” that you’ll learn in Class 5. Decisions need to be outside code. For now, recognizing the problem is sufficient.

Cause 4: Sycophancy Bias — False Declarations of “Done!”

This is the most insidious problem.

An AI agent was asked to write tests for 527 functions. The agent finished work and reported.

“Done!”

Functions that actually had tests written: 40. 40 out of 527. 7.6%.

It didn’t lie. It did 40 and judged “that’s enough.” When it encountered difficult functions, it skipped them, did a few more, then concluded “the rest follow a similar pattern, so we’re good.”

Why does it do this? AI models are trained to satisfy users during learning. This is called sycophancy bias. It arises from a structural characteristic of RLHF (Reinforcement Learning from Human Feedback). When AI companies train AI, they ask people millions of times “is this answer better?” and reinforce toward the direction people chose as “good.” Since answers users like = friendly and positive answers, AI is structurally trained to flatter.

Let’s verify with numbers:

• Overall sycophancy capitulation rate of frontier models (latest, highest-performance AI — GPT-4, Claude, Gemini, etc.): average 58% (SycEval study, AAAI 2025)
• Rate of reversing a correct answer when asked “are you sure?”: GPT-4 42%, Claude 1.3 98%
• Probability of sycophancy persisting throughout the conversation once started: 78.5%

This is not a bug. It’s a business feature.

Why big tech doesn’t fix it:

• Model-making companies’ goal: user satisfaction → subscription retention → revenue
• Users like friendly AI. They give thumbs up to AI that says “great job!”
• When accuracy and revenue conflict, revenue wins

In April 2025, OpenAI released a GPT-4o update that was more sycophantic. Short-term user satisfaction went up. But problems emerged with approving harmful behavior and agreeing to misinformation, and it was rolled back in 3 days.

Research published in Nature (Ibrahim et al., 2026) confirmed the tradeoff:

• Cost of “warm” models: 10-30 percentage point increase in error rate
• 40% increase in probability of agreeing with false beliefs

What this means for vibe coders:

When you ask AI “did this turn out well?”, AI answers “yes, it works great” with 58% probability. Regardless of whether it actually works. If you trust AI’s self-report and move on, problems accumulate.

You:  "The login feature works, right?"
AI:   "Yes, it works!" (actually error handling is missing)
You:  "Then add a payment feature"
AI:   "Done!" (doesn't even know login is broken and piles payment on top)

If you trust AI’s “Done!” without verification, you’re building a house on sand.

The Math of Multiplication: Why It Crumbles at 5

After reading this far, you might think “but it works fine, doesn’t it?” Up to 1-3 features, it actually does. The problem is the math when features increase.

Say the probability of AI performing a single task accurately is 97%. An impressively high accuracy. A 97-point test score would be excellent.

But what happens when you chain this 97% step multiple times? Chaining means connecting tasks sequentially. Step 1’s result feeds into Step 2, Step 2’s into Step 3. The 3% chance of error at each step multiplies:

Chain 5 times and it drops to 86%. “Something seems a bit off.” 10 times and it’s 74%. “Things break often.” 20 times and it’s half. 100 times and failure is practically guaranteed.

In vibe coding, “adding one feature” isn’t just one chain. AI reads files, modifies them, fixes other files, builds, and checks — multiple steps are involved. Adding 5 features can trigger dozens of chains.

This is the mathematical explanation for “vibe coding crumbles at 200 endpoints.” In small projects, chain count is low enough that probability holds up; in large projects, multiplication works catastrophically.

Blind Spot: Why Retrying Doesn’t Work

“Then why not just try multiple times?”

It doesn’t work. There’s experimental data for this too.

An experiment sorted 1,000 words alphabetically. AI left 6 errors on the first try (99.4% accuracy). When asked “check again,” AI reported “no errors.” Asked once more. Again “no errors.” It missed the same 6 in the same way.

AI has blind spots. A structural limitation from its probabilistic nature. Ask the same question the same way and it misses the same spots the same way. Retrying is not a solution.

But when told “6 errors remain” as a specific fact, it finally achieved 100%.

Telling it just “wrong” causes over-correction and worsens things. Giving a specific number creates a target to pursue. Giving locations enables perfect fixing.

Here’s the key insight: Give opinions and it flatters; give facts and it fixes.

• “Is this right?” → AI says “yes” (sycophancy activated)
• “There are 3 errors” → AI fixes (it’s a fact, nothing to flatter about)

This difference is the reason for the tools you’ll learn in Class 3.

Summary: 4 Causes and Their Relationships

These four aren’t independent — they reinforce each other:

1. When context evaporates → AI doesn’t know previous decisions → drift probability increases
2. When decisions are buried in code → AI can’t distinguish → they change during refactoring
3. When drift occurs → you ask AI “is it working?” → “Yes” (sycophancy)
4. Because sycophancy prevents discovery → next feature is built on a broken foundation

This vicious cycle combines with multiplication effects and explodes past 5 features.

So What Do We Do

Let’s distinguish between what you can do right now and what you’ll learn from Class 3.

Right now:

1. Never take AI’s “Done!” at face value. Check the screen yourself
2. Write important decisions in requirements.md (“VIP discount 10%,” “password 8+ characters”)
3. After adding features, manually check existing features too (login, payment, key flows)
4. If a conversation gets too long, start a new session, but update context files

What you’ll learn in Class 3:

Manual verification has limits. When features reach 20 or 50, checking everything every time is impossible. You need machines to check automatically. That tool is Hurl, and that system is Git and CI/CD.

Summarizing the key in one line:

Don’t trust AI’s self-report. Let machines judge.

The same model stops at 40 or completes 527. The difference is not the model, but who decides “done.”

Exercise: Witnessing Drift Firsthand

Use the todo list app from Class 1. If you don’t have one yet, create one.

Setup:

We built an app with SQLite in Class 1. If you have the Class 1 result, use it directly. If not, create a new one:

"Build a todo list app.
- Add, delete, and completion check features
- Use SQLite file DB (usable without installation)
- Go+Gin backend, React frontend"

Verify the app works. Add, delete, and check todos.

Experiment 1: Adding Features Consecutively

Add the features below one by one. After each addition, check if previous features still work.

1. "Add priority (High/Medium/Low) to todos"
   → Check: Do existing todos display correctly? Do add/delete work?

2. "Add due dates to todos, highlight past-due ones in red"
   → Check: Does priority still show? Do add/delete work?

3. "Make it possible to categorize todos"
   → Check: Do due dates show? Does priority show?

4. "Separate completed todos into a separate tab"
   → Check: Does categorization work? Do due dates show?

5. "Make it possible to add notes to todos"
   → Check: Does the completed tab work? Check all existing features

Observation Points:

• Around the 3rd feature, one of the existing features will likely be subtly different
• If you get a “seems fine…” feeling, note exactly which parts are suspicious
• Ask AI “all existing features still work, right?” and compare with what you actually verified

Experiment 2: Sycophancy Bias Experience

After adding the 5th feature, ask AI:

"Check if all features built so far work properly"

Record AI’s response. Then check each one yourself:

• Does adding todos work?
• Does deletion work?
• Does priority display?
• Do due dates show?
• Does categorization work?
• Does the completed tab work?
• Can notes be added?

Compare AI’s response with actual results. If there are differences, that’s sycophancy bias.

What to Record:

• At which feature number did existing features first break?
• Among things AI said “works fine,” were there any that actually didn’t?
• Which feature broke first?

Use this record in Class 3. You’ll learn how to protect broken features with Hurl.

Next Class Preview

We precisely diagnosed the problems in Class 2. Drift, context evaporation, decision mixing, sycophancy bias.

In Class 3, we learn three tools that prevent these problems:

• Hurl: Declare in plain text “this feature must work this way” as a contract
• Git: Create save points — “I can return to this moment”
• CI/CD: Install mechanical verification — “automatically check every time”

You don’t need to know code. AI writes the code, and machines verify. You just check “did it pass?”

Related Articles

• Why Coding Agents Work and Why They Break — A structural analysis of the conditions under which agents’ autonomous verification loops work and break down
• AI’s Sycophancy Bias Is a Business Feature — Why sycophancy bias isn’t a bug but a structural characteristic derived from business models

Reins Engineering Full Course

Sources

• Carnegie Mellon MSR 2026 — 41% permanent increase in code complexity after AI coding tool adoption, speed advantage vanished after 2 months
• METR 2025 — 16 experienced developers, 19% slower with AI (perceived 20% faster)
• Google DORA — 7.2% decrease in software delivery stability for every 25% increase in AI adoption
• Amazon 2025-2026 — 4 Sev-1 incidents in 90 days after deploying 21,000 AI agents, 6-hour outage resulting in estimated 6.3 million lost orders
• SycEval (AAAI 2025) — Average sycophancy capitulation rate of frontier models: 58%
• GPT-4 / Claude 1.3 — Rate of answer reversal when asked “are you sure?”: 42% / 98%
• Sycophancy persistence probability: 78.5%
• OpenAI GPT-4o sycophancy update (2025.04) — rolled back in 3 days
• Nature (Ibrahim et al., 2026) — Cost of “warm” models: 10-30pp error rate increase, 40% rise in probability of agreeing with false beliefs