The Architecture Problem Behind AI Hallucinations That Compromise Security

The Architecture Problem Behind AI Hallucinations That Compromise Security

Nearly one in five AI-generated code samples recommends a software package that exists nowhere in any registry, according to a USENIX Security 2025 study. Attackers are registering those names before developers do.

Here is how it works. A developer using an AI coding assistant accepts a suggestion to install a software library. The assistant named a library that doesn’t exist. Attackers monitor AI outputs for these invented names and register them on public registries before the developer can. They fill the package with malware. The developer installs what they believe is a harmless utility and executes the attacker’s code instead. Security researchers call the practice slopsquatting. A USENIX Security 2025 study of 576,000 AI-generated code samples confirmed the pattern at scale.

  The names are one problem, the code is the next. Veracode’s 2025 GenAI Code Security Report tested more than 100 large language models across 80 coding tasks and found that 45% of the resulting code introduced a known security flaw.  

In rinf.tech’s own engagements, standard prompting alone rarely holds hallucination rates below 7%. Better models keep shipping, but hallucination rates have barely moved. The problem starts before the model runs, with the prompt.

What the Prompt Can’t See

Andrej Karpathy named the pattern in February 2025: prompt loosely, hope for the best, clean up the mess afterward. He called it vibe coding, and it has become the industry’s default. 

Prompt engineering, fine-tuning, and retrieval-augmented generation all hit the same ceiling, context window limits, because a prompt can show a model a snippet of code or a database schema, but it lacks the capacity to hold the wider system:  how the application deploys and where its data comes from. It has no view of the business rules governing either. The model generates code that works in isolation and breaks once it goes live, blind to constraints it was never shown.

A VentureBeat analysis of spec-driven development argues that agentic coding at enterprise scale demands a structured specification the agent can reason against throughout the build. Lacking that structure, architectural decisions, business rules, and downstream dependencies scatter across chat windows and private conversations, vanishing the moment the session closes. Six months later, the system has amnesia about its own design. Natural language as the primary software interface introduces a flaw that persists regardless of model size.

Spec-as-Source

There is another way. A structured, machine-readable specification becomes the permanent source of truth, and code is treated as disposable output, regenerated on demand whenever the spec changes. This is what rinf.delivery implements across four distinct phases.

All information first passes through a recursive context engine, based on MIT research, that stores documents as variables in a sandboxed environment rather than stuffing them into the prompt window. For existing systems, the codebase is indexed as a queryable database and the specification is reconstructed from live system structure. The output is a formal behavior contract governed by a single rule: the pipeline excludes anything that isn’t in the spec. Hallucination risk drops before the model ever receives a query, because the system avoids guessing at facts that already exist.

Typed signatures then define input and output schema once, and generation follows that schema across any model or provider. Streaming assertions validate output as it streams. The moment a field violates the contract, the stream cancels and corrects before tokens are spent on invalid output. The planner agent lacks writing tools and cannot modify files during planning. The guardrails, rather than the prompt, are the architecture.

An autoresearch loop then generates competing implementations and runs them against edge cases. Tree search, parallel agents in isolated git worktrees, and automated gates discard experiments that fail regression tests, even when their score beats the current best. Engagement data shows the loop finds simpler implementations that hold performance with less code. Most agents get worse the longer they run. This one improves.

Finally, engineers review architecture, security, and judgment calls beyond the pipeline’s scope. Every artifact carries a full traceability chain; requirement, generated variant, benchmark score, gate result, selection, human sign-off. Microsoft’s ASSERT framework turns the same Gherkin contract into executable test cases, producing scored results with labels, rationales, policy citations, and failure patterns. The report is a gate rather than a suggestion.

Off-the-shelf agent frameworks, fine-tuned models used without additional guardrails, and basic retrieval-augmented generation share the same weakness: nothing stops a plausible-sounding but wrong output from shipping. The four-phase approach closes that gap at each stage rather than catching it after the fact.

Proof Anchor

The approach has been validated at enterprise scale. An enterprise vendor engaged rinf.tech to re-engineer two modules of a platform that had been running for more than 15 years. The work proceeded largely from written specifications rather than direct access to the live system. Both modules closed on schedule.

Combined test coverage exceeded 90%. Every business requirement was traceable to a specification and a test that verified it, rather than to a person’s memory of how the system was supposed to behave. The full codebase and test suite transferred to the client with complete ownership, and by the end, the client’s own engineers could run the same workflow without rinf.tech.

The stack is cloud-agnostic and model-agnostic, running on sovereign, EU-managed GPU clusters through rinference to support compliance obligations under the EU AI Act.

Three Questions for the Engineering Leader

Before the next AI-assisted release ships, three questions are worth answering. How much of your team’s week goes to fixing AI output that already shipped, versus specifying what it should do before it ships? If an engineer left tomorrow, would the reasoning behind your system’s behavior survive in a document, or only in a chat history that is already gone? Could you produce a full requirement-to-sign-off trace for your last AI-generated change today, if a regulator asked for it?

The companies that treat the spec as the asset worth protecting will know what their software costs. The rest will find out.

The post The Architecture Problem Behind AI Hallucinations That Compromise Security appeared first on rinf.tech.