The Bug AI Can't Fix: Where Does AI Reliability End in Software Development?

Today's AI models can write a functioning piece of code, but when it comes to bugs rooted in tangled, undocumented decisions spread across a system, they repeatedly fail even after several attempts. The reason is simple: the AI has no context for why the code was built that particular way, and if nobody on your team has that context either, nobody will find the bug. The more code you produce without anyone actually understanding it, the higher the risk of hitting a bug that even the latest model can't handle.

What happens when even the latest AI model can't fix a bug?

Developer and blogger Florian Herrengt describes, in an essay on the disappearing middle class of software engineering, a situation he offers as a textbook example of what can happen on a team. Users report a bug, the team tries to fix it for the fourth time, this time handing the task to an AI, and even the latest model can't find the issue. The team lead goes to the person who worked on that feature and hears only "actually, I'm not sure, let me ask Claude." The two of them then read a long AI response, unable to judge whether any of it is true, while the model sounds perfectly confident. The team tries a more thorough review, hits a daily usage limit, and pushes the fix to the next day.

This is exactly the core of the problem: the more tangled a system becomes, and the fewer people understand why it's built the way it is, the less even the best AI model can help. As a quote captured by Simon Willison puts it, a project becomes so knotted that no one on the team can even begin to understand what's actually going on inside it.

Why can AI write code but not understand why it's there?

An AI agent is a tool you assign a task to, and it proposes and carries out code changes on its own, so you don't have to write every line yourself. It can propose an architecture, then in the next round choose a different one, apologise, and change it again, depending on how you prompt it. Design decisions end up buried in a long chat conversation instead of in the head of a person who could explain them.

In practice, this means that decisions about why a data model is built a certain way, or why a new database table was added, often now sit buried in an AI conversation history rather than with a specific person. When a bug related to that decision later surfaces, the AI is tasked with finding the root cause, but it lacks the context of the person who originally made that decision. So it repeats the same attempts over and over, because it's examining the symptom, not the actual cause.

When does the speed of AI-driven development turn into a risk for your business?

The author compares this to buying a luxury car on credit: you don't see the debt, you only see a car that looks great. Technical debt is the umbrella term for shortcuts and compromises in code that save time now but become more expensive to fix later. An AI agent can add a new table or column to a database in a matter of minutes. But once live data starts being stored in them, you can't just delete them — you need to plan a migration, meaning a safe transition of data to the new structure without downtime or loss, handle what happens if it fails, and make sure no orphaned relationships are left between tables.

While you're dealing with that, the team keeps generating more code, more layers, more decisions. By the time you've untangled one bad decision, five more may well have piled up. For a business owner or manager, this carries the same risk as any technical debt, just at a faster pace: your people's time, the cost of fixing it, and in the worst case, a product that stops working exactly when it matters most.

AI vs. an experienced developer: who can handle what?

• Write a new feature from a clear spec - AI model: Reliable, follows the spec. Experienced developer: Reliable, weighs the broader impact.
• Find a bug in simple, isolated code - AI model: Usually reliable. Experienced developer: Reliable.
• Find a bug in a tangled system with unclear decision history - AI model: May fail even after repeated attempts. Experienced developer: Can trace the root cause, even if it takes longer.
• Explain why a decision was made the way it was - AI model: Lacks context of the original request. Experienced developer: Made the decision or reviewed it.
• Safely change a database structure on a live system - AI model: Proposes a change but can't assess all operational risks. Experienced developer: Plans the migration and manages downtime risk.

How do you know this risk applies to your business?

This applies to you if:

• most of your code over the past few months was produced by an AI agent and nobody actually took the time for a proper review
• when you ask why the system works the way it does, the answer points to an AI conversation
• the same bug keeps coming back even after several "fixes" and nobody knows exactly why
• new team members have no way to understand the architecture, because it isn't documented anywhere other than in people's heads or chat history
• you're planning to scale or sell the product, but nobody can estimate how much work any major change would take

How do you fix a bug and code that AI can't handle?

Fixing a bug that AI repeatedly fails to solve usually doesn't start with another prompt, but with an audit of the code's current state. An experienced developer goes through the codebase, maps how the individual parts connect, and identifies places where decisions were made without a clear rationale or got lost in AI conversation history. Only then can you judge whether a targeted fix is enough, or whether an entire part of the system needs rewriting because it's so tangled that a partial fix would only make it worse.

If the problem reaches down into the data itself, it's handled as a separate migration project, with a plan for what happens if something fails and how to keep the product running in the meantime. For businesses that need to connect multiple systems together, or that lack their own custom-built application, the same approach applies beyond fixing a single bug: first understand what the system needs to do and why, and only then start building. AI remains a useful tool in this process, but under the oversight of someone who can judge whether a proposed solution makes sense in the context of the whole system, not just one conversation.

Frequently Asked Questions

Why can't AI fix a bug it has already tried to fix several times? Because it lacks the context for why the code was built the way it is. It examines the symptom, not the cause, and if the decision that caused the bug was never documented anywhere outside chat history, the model keeps circling the same loop.

Does this mean AI isn't suitable for software development? No. AI reliably writes new code from a clear spec and finds bugs in simple, well-structured code. The problem arises with tangled systems with an unclear decision history, where human understanding of the whole is missing.

How do we know our code is "running away" without oversight? Typical warning signs include large pull requests, meaning proposed code changes, that nobody properly reviewed, recurring bugs, and getting an explanation of the architecture only as a link to an AI conversation.

Do we still need experienced developers if AI writes most of the code? Yes, for anything more complex than an isolated function. Someone needs to understand why the system works the way it does and be able to judge whether a proposed solution creates a problem that will only surface later.

How much does it cost to fix code neglected due to uncontrolled AI use? Only an audit of the specific code can give an exact answer, since it depends on the scope and how tangled the system has become. Generally speaking, though, the longer the problem is put off, the more work the fix requires.

How do you recognise technical debt that came from AI-generated code? You recognise it when nobody on the team can explain, without hesitation, why the system is built the way it is, and why a particular database table, service, or technology was added.

When does it make sense to let AI modify code without detailed human review? For small, isolated changes with a clearly measurable outcome. For anything touching data structure, architecture, or multiple interconnected parts of the system, the output should always be reviewed by someone who understands the whole picture.

How long does it take to bring neglected code back under control? It depends on the scope and on how many decisions need to be traced back and understood. Only an audit of the current state can give a concrete estimate; generally, the sooner you start, the less time the problem has had to grow.

Summary

AI models today can write code faster than ever, but they repeatedly fail on bugs rooted in tangled, undocumented decisions spread across a system, because they lack the context that only people have. The more code you produce without anyone actually understanding it, the greater the likelihood you'll hit a bug that AI can't solve even after several attempts. AI reliability in software development ends where the need to understand the whole system and its decision history begins — not where writing another line of code does. The solution isn't to stop using AI, but to have people on hand who can judge when the output is sound and when it needs intervention. Without that, the speed of AI-driven development will sooner or later turn into a bill someone has to pay.

If you suspect your code has drifted into a similar situation, or you're dealing with a bug that AI repeatedly can't solve, we're happy to walk through your current state with you, no strings attached, and tell you exactly where the problem lies.