From technical to ethical, four reasons we shouldn’t hand all code responsibility to a single AI model
There’s an old adage in superhero lore: With great power comes great responsibility. In software development, AI has handed us the power of a thousand developers — and we’ve rushed to give AI code generators the keys to the spaceship. The power is obvious. The responsibility is where both humans and machines are falling short.
With the release of Claude Code Security this weekend, the technical world faces a dilemma: go all-in on foundational AI development, or pause to consider the guardrails and best practices we must learn to operate with.
As appealing as it is, we must resist the temptation to outsource judgment, architecture, and validation to a single model. Here’s why:
1. AI Doesn’t Eliminate Human Errors, it Replicates Them
AI doesn’t invent fundamentally new code patterns. It reproduces the most common ones it has seen before. That means it scales not only productivity, but also existing weaknesses in software engineering practice.
This often leads to a regression from decades of hard-earned best practices – as documented in OX’s recent “Army of Juniors” Report. The 10 anti-patterns we identified include:
- “Vanilla-style” code: AI frequently ignores battle-tested, community-audited libraries and instead generates from-scratch implementations — reinventing the wheel, except this wheel still contains old cracks.
- Security theater in testing: Models can generate high coverage metrics (90%+) by writing shallow tests that prove code runs, not that it’s secure.
Security isn’t just about spotting vulnerabilities. It’s about architecture — how a Dart frontend talks to a Go backend through middleware, how data flows from input to database to third-party API and back.
That architectural understanding is precisely where current models struggle. Today’s models are trained primarily on publicly available code and technical content. But real architectural knowledge often lives in undocumented experience and internal design decisions that models never see. Without that context, you may get code that looks solid but fails quickly in production.
In short: an AI can flag a vulnerability in a single repository. It cannot reliably tell you whether that issue is actually exploitable in a complex and unique environment.
2. The Scale Problem: How Much Will this Cost?
AI code generators work by reading. That distinction matters.
Purpose-built security tools don’t read your entire codebase line by line like a bedtime story — they run targeted analyses against specific patterns and signatures. Ask a general-purpose model to audit an enterprise repository and you’re burning astronomical tokens on noise that still isn’t prioritized by exploitability.
Now scale that to 1,000 repositories.
The math breaks. The signal degrades. Instead of actionable findings, you get floods of false positives and low-value alerts at high-costs.
When you force a general model to behave like a specialized scanner at enterprise scale, the output quality collapses.
3. The Echo Chamber Problem: Concentrating Too Much Power in One Model
The industry’s default response to AI velocity has been simple: give the same AI more jobs. Write the code. Scan the code. Fix the code.
But consolidating that much responsibility into a single system creates a structural risk — not from attackers, but from trust architecture.
This is the echo chamber problem.
When the same system writes, evaluates, filters, and patches, you don’t have independent validation. You have self-approval.
And self-approval is not security.
This was also demonstrated in our recent Lovely App! Don’t Look Inside research, where AI app builders ran internal security scans on their own generated code — and cleared it — even though it contained basic, exploitable XSS vulnerabilities that any attacker could trigger in minutes.
Security has always relied on layered controls and adversarial thinking. Separate builders from breakers. Separate implementation from verification.Collapse those layers into one model, and you collapse the governance structure that made modern software survivable.
4. The Logical Loophole: Why Does it Write Vulnerable Code if it Knows Better?
There’s one question every security leader should ask:
If an AI can detect a vulnerability during scanning, why did it generate that vulnerability in the first place?
Either it didn’t understand secure implementation while writing, or it only recognized the flaw during review. Neither answer inspires confidence in fully delegating responsibility.
Security demands consistency. A system that alternates between introducing risk and identifying it isn’t demonstrating reliability; it’s demonstrating variability.
And variability is the opposite of assurance.
Conclusion: on Behalf of Accountability
AI is not the enemy of security. Inconsistency, unverifiable ‘truth,’ and lack of accountability are.
Tools like Claude Code and Cursor are remarkable accelerators. They can dramatically increase development velocity and reduce friction. But velocity is not security.
With great power comes responsibility — and that responsibility means resisting the temptation to outsource judgment, architecture, and validation to a single model. AI can assist secure development. It cannot replace the layered systems, independent verification, and human accountability.
