The Architecture Problem Behind AI Hallucinations That Compromise Security

The Architecture Problem Behind AI Hallucinations That Compromise Security

Nearly one in five AI-generated code samples recommends a software package that exists nowhere in any registry, according to a USENIX Security 2025 study. Attackers are registering those names before developers do.

Here is how it works. A developer using an AI coding assistant accepts a suggestion to install a software library. The assistant named a library that doesn’t exist. Attackers monitor AI outputs for these invented names and register them on public registries before the developer can. They fill the package with malware. The developer installs what they believe is a harmless utility and executes the attacker’s code instead. Security researchers call the practice slopsquatting. A USENIX Security 2025 study of 576,000 AI-generated code samples confirmed the pattern at scale.

  The names are one problem, the code is the next. Veracode’s 2025 GenAI Code Security Report tested more than 100 large language models across 80 coding tasks and found that 45% of the resulting code introduced a known security flaw.  

In rinf.tech’s own engagements, standard prompting alone rarely holds hallucination rates below 7%. Better models keep shipping, but hallucination rates have barely moved. The problem starts before the model runs, with the prompt.

What the Prompt Can’t See

Andrej Karpathy named the pattern in February 2025: prompt loosely, hope for the best, clean up the mess afterward. He called it vibe coding, and it has become the industry’s default. 

Prompt engineering, fine-tuning, and retrieval-augmented generation all hit the same ceiling, context window limits, because a prompt can show a model a snippet of code or a database schema, but it lacks the capacity to hold the wider system:  how the application deploys and where its data comes from. It has no view of the business rules governing either. The model generates code that works in isolation and breaks once it goes live, blind to constraints it was never shown.

A VentureBeat analysis of spec-driven development argues that agentic coding at enterprise scale demands a structured specification the agent can reason against throughout the build. Lacking that structure, architectural decisions, business rules, and downstream dependencies scatter across chat windows and private conversations, vanishing the moment the session closes. Six months later, the system has amnesia about its own design. Natural language as the primary software interface introduces a flaw that persists regardless of model size.

Spec-as-Source

There is another way. A structured, machine-readable specification becomes the permanent source of truth, and code is treated as disposable output, regenerated on demand whenever the spec changes. This is what rinf.delivery implements across four distinct phases.

All information first passes through a recursive context engine, based on MIT research, that stores documents as variables in a sandboxed environment rather than stuffing them into the prompt window. For existing systems, the codebase is indexed as a queryable database and the specification is reconstructed from live system structure. The output is a formal behavior contract governed by a single rule: the pipeline excludes anything that isn’t in the spec. Hallucination risk drops before the model ever receives a query, because the system avoids guessing at facts that already exist.

Typed signatures then define input and output schema once, and generation follows that schema across any model or provider. Streaming assertions validate output as it streams. The moment a field violates the contract, the stream cancels and corrects before tokens are spent on invalid output. The planner agent lacks writing tools and cannot modify files during planning. The guardrails, rather than the prompt, are the architecture.

An autoresearch loop then generates competing implementations and runs them against edge cases. Tree search, parallel agents in isolated git worktrees, and automated gates discard experiments that fail regression tests, even when their score beats the current best. Engagement data shows the loop finds simpler implementations that hold performance with less code. Most agents get worse the longer they run. This one improves.

Finally, engineers review architecture, security, and judgment calls beyond the pipeline’s scope. Every artifact carries a full traceability chain; requirement, generated variant, benchmark score, gate result, selection, human sign-off. Microsoft’s ASSERT framework turns the same Gherkin contract into executable test cases, producing scored results with labels, rationales, policy citations, and failure patterns. The report is a gate rather than a suggestion.

Off-the-shelf agent frameworks, fine-tuned models used without additional guardrails, and basic retrieval-augmented generation share the same weakness: nothing stops a plausible-sounding but wrong output from shipping. The four-phase approach closes that gap at each stage rather than catching it after the fact.

Proof Anchor

The approach has been validated at enterprise scale. An enterprise vendor engaged rinf.tech to re-engineer two modules of a platform that had been running for more than 15 years. The work proceeded largely from written specifications rather than direct access to the live system. Both modules closed on schedule.

Combined test coverage exceeded 90%. Every business requirement was traceable to a specification and a test that verified it, rather than to a person’s memory of how the system was supposed to behave. The full codebase and test suite transferred to the client with complete ownership, and by the end, the client’s own engineers could run the same workflow without rinf.tech.

The stack is cloud-agnostic and model-agnostic, running on sovereign, EU-managed GPU clusters through rinference to support compliance obligations under the EU AI Act.

Three Questions for the Engineering Leader

Before the next AI-assisted release ships, three questions are worth answering. How much of your team’s week goes to fixing AI output that already shipped, versus specifying what it should do before it ships? If an engineer left tomorrow, would the reasoning behind your system’s behavior survive in a document, or only in a chat history that is already gone? Could you produce a full requirement-to-sign-off trace for your last AI-generated change today, if a regulator asked for it?

The companies that treat the spec as the asset worth protecting will know what their software costs. The rest will find out.

The post The Architecture Problem Behind AI Hallucinations That Compromise Security appeared first on rinf.tech.

Why Better Prompts Won’t Fix AI-Generated Code

Silicon Valley sold AI coding tools as a superpower for developers. A Stanford study found they may be making programmers worse while convincing them they’ve never been better. The fault lies with the prompt, a throwaway instruction that forgets the whole system the moment the chat closes.

In the study, researchers gave 47 developers the same task to write code with and without AI assistance. Those using AI produced more security vulnerabilities than those working alone and were more confident their code was secure. Only 67% of the assisted group produced correct solutions, compared to 79% of the control group. The code looked right. It built. It passed CI. It deployed. And it broke in ways nobody caught, because output that looks plausible gets trusted.

Prompting is how most teams talk to AI code generation today. It is also where the whole thing falls apart. Every prompt runs up against the context window, the hard limit on how much a model can hold in view. Past that limit, the model sees only the snippet in front of it while the wider system slips out of view. A specification, on the other hand, lives outside that ceiling, which is exactly why it holds where the prompt caves.

The difference comes down to what each one can hold. It can show the coding assistant a single piece of code, a database snippet, or a few examples. But it cannot hold the bigger picture: how the application is deployed, where the data comes from, how results are reported, or the business rule requiring extra approval for claims over €50,000. The AI generates code that works in isolation and falls apart once it goes live, blind to rules it was never shown.

VentureBeat put it another way, prompting can build the pipeline but can’t explain it six months later, the business rules and architecture behind the code vanish the moment the chat closes. That same blind spot shows up four ways.

No Grounding in What Exists

The most visible symptom is package hallucination. When researchers tested 16 large language models across more than half a million code samples, one in five suggested packages didn’t exist. In total, the models invented more than 200,000 package names. Open-source models hallucinated more than commercial ones, with some exceeding 33%.

The same fake names come back again and again, 58% recurred within ten prompts. A mistake that repeats can be predicted, and a mistake that can be predicted can be exploited. An attacker publishes malicious code under a hallucinated name, then waits, sooner or later, someone’s model will recommend it. Researchers call the technique slopsquatting.

No Cross-Generation Coherence

Each generation is a fresh start, with no memory of the patterns it used in the previous generation, the conventions the team agreed on, or the architectural decisions embedded in the codebase. What results is a collection of individually plausible fragments that drift apart over time.

An analysis of 211 million changed lines of code across repositories owned by major technology companies and enterprises found that the percentage of lines affected by refactoring
dropped from 25% in 2021 to below 10% in 2024. Over the same period, copy and pasted lines rose from 8.3% to 12.3%, and copy-and-paste exceeded moved code for the first time in the dataset history. Code reuse declined as generation volume increased.

Say a team generates a dozen functions across a sprint. Three use early returns for error handling. Four use try/catch blocks. Two use result objects. Three mix patterns within the same function. Code compiles. Tests pass. Yet the codebase has become illegible. The illegibility stems from a deeper cause: no two generations share the same mental model. Review becomes the bottleneck: four thousand lines generated in an afternoon take forty hours to review properly. The ratio is unsustainable.

No System Behavior Representation

The model generates code that’s syntactically valid but carries no sense of how it behaves under load, with real data, in production. No way to reason about what happens when the claims queue backs up, when an upstream data feed goes stale, when a regulatory rule changes mid-quarter.

The Stanford study put numbers on it, developers using assistants were significantly more likely to use trivial ciphers and skip authenticity checks. The code looked right, compiled, and ran, yet it was wrong in ways nothing flagged. Participants trusted it because the surface was convincing.

The costliest version shows up in something like KYC risk scoring. A generated function handles the happy path and the null case, but misses the edge where a customer’s jurisdiction changes between document submission and verification, because that constraint lives in a compliance document the model never saw, in a workflow it was never told about. The output is plausible enough to survive review and wrong enough to surface as an audit exposure months later.

No Audit Trail or Governance

When a person writes a function by hand, it leaves a traceable trail: a commit message, a pull request, a reviewer, a decision record. When a model generates a function, there is a prompt and unless a team deliberately captures it, it is ephemeral, unversioned, and untraceable.
Six months later, when an auditor asks why the system applies a specific rule to claims under a specific threshold, there is no answer. That prompt is gone. The reasoning behind it is gone. A system is running on generated code that nobody can explain.

The EU AI Act requires that standalone high-risk AI systems technically allow for the automatic recording of events (logs) over the lifetime of the system, with strict enforcement taking effect in December 2027. This is mandatory. Regulated industries (banking, insurance, energy) cannot operate systems they cannot audit. AI-generated code without a verifiable record of why each decision was made is unauditable by definition.

A new engineer joins the team to discover that half the codebase was generated: code that nobody wrote, and nobody fully understands. Refactoring is dangerous because the side effects are unknown. The single place to understand what the system should do, if it existed, would be a specification. Without one, every change is a bet. As security practitioners noted after the first AI-orchestrated espionage campaign, control belongs at the architecture boundary, enforced by systems, not by prompts.

What the Alternative Looks Like

This structural shift turns a chaotic generation process into a closed loop engineering pipeline. Automated validation routes each generated output through testing and integration pipelines before any human engineer reviews the work. Architectural patterns remain consistent across delivery cycles, rather than drifting over time as individual prompts to diverge. The specification acts as the permanent reference point, with every generation of events mapped directly to formal requirements and compliance boundaries.

Engineering teams across the industry have already begun to shift focus from temporary prompt engineering to persistent context engineering, a transition that analyst firms like Gartner identify as an operational necessity for governing automated workflows. This shift treats the machine-interpretable specification as the central artifact, ensuring a fixed boundary between engineering intent and production execution.

For regulated enterprise environments, a formal machine-readable specification represents the most robust implementation of this principle. It establishes a fixed boundary between intent and execution, and provides the layer at which operational control can be maintained.

Better prompts address the surface of the problem. Additional manual review addresses individual symptoms. A sustainable long term approach requires an architectural shift in how generated code is treated.

Most organisations currently operate under the assumption that code generated by artificial intelligence should be reviewed, validated, and then maintained as a permanent asset, in the same way human written code has always been managed.

rinf.tech operates on an inverted model. The structured machine-readable specification, one that tools can parse and generate code from directly, becomes the single permanent source of truth. Code is treated as disposable output that is regenerated on demand. The code itself is no longer the unit of validation, the specification is.

This is not an incremental improvement on existing development methodologies. It is a structural adjustment to the fundamental assumptions that underpin software delivery. For organisations operating under governance and cost constraints this adjustment will become unavoidable as AI coding assistants become standard across engineering teams.

What you keep after the chat closes

The companies that scale AI will be the ones that treat the specification as the real asset and the generated code as throwaway. The models are only getting better, so the edge is no longer the code, it is having a written system the machine builds against and auditors can read. Teams that skip that step are shipping software today that they will pay to untangle for years.

The post Why Better Prompts Won’t Fix AI-Generated Code appeared first on rinf.tech.