Across all the AI deployments I’ve reviewed in a range of industries, one pattern stands out: Organizations are looking for risk in the wrong place. Their focus tends to settle on the model itself, including its training data, safety controls and behavior. But in practice, that’s rarely where things break down.
Many of the highest-impact vulnerabilities emerge in the surrounding system — how prompts are constructed, how external data is sourced and trusted, what the model is allowed to access and under which identities it operates. This is where AI systems are most often exposed: in the seams between components, where governance is usually the weakest.
The deeper issue is the gap between expectation and reality. Organizations anticipate one kind of failure, but production systems fail in entirely different ways. Risk emerges in that mismatch — in assumptions that do not survive scale, autonomy or integration.
The same misconceptions about securing AI appear again and again. It’s worth examining the five that matter most.
Misconception #1: If the model is secure, the system is secure
This is the most common mistake I see, and it’s an easy one to make. The model feels new, so teams focus on guardrails and testing, then assume the job is done.
But that’s not how these systems fail. The model is only one component of a much larger infrastructure, and most failures happen where it connects to data, tools and other systems. Focusing on the model alone is like installing a high-security door in a building with no walls.
How to fix it: To secure an AI system, think of the full environment as the attack surface. Map the entire data flow — from inputs to retrieval and memory, and from tools to outputs — and govern prompts, agents, vector stores, identities and connectors as owned assets, with clear control points, accountability and policy, as you would any other critical system.
Misconception #2: Prompt injection is just another input problem
Security teams with web backgrounds often reach for familiar tools when they see a new problem. When it comes to securing AI, that instinct can lead them in the wrong direction.
Prompt injection looks like Structured Query Language (SQL) injection — where systems treat malicious input as a command — but behaves very differently. Traditional software can enforce a clear separation between instructions and data. Large language models cannot reliably do that.. They process instructions and data as the same stream of text and make probabilistic judgments about which is which.
The UK’s National Cyber Security Centre (NCSC) is clear on this: Prompt injection is structurally different from SQL injection and needs to be approached differently.
How to fix it: Filters and detectors help, but they do not solve the problem on their own. The most effective safeguards are architectural. Constrain tool access, enforce least privilege, isolate untrusted content and validate tool calls and parameters deterministically. Require explicit approval for sensitive actions, sandbox execution and monitor aggressively. These measures reduce both the likelihood and impact of compromise, but they do not eliminate risk. If the residual risk remains unacceptable, the use case is not well suited to a large language model.
Misconception #3: AI outputs are just text — they don’t create real risk
Early AI deployments rewarded autonomy. That framing has been carried into production environments where it doesn’t belong.
AI outputs might look like harmless text but rarely stay that way. The moment they are passed to another system, they can lead to real actions — sending emails, querying databases, executing code or deleting a record. In that context, a successful prompt injection inherits everything the system can do.
That’s where the risk becomes real: The system’s capabilities become the attacker’s capabilities.
The Open Web Application Security Project identifies excessive agency as one of the most serious risks in agentic AI, while the NCSC notes this is precisely where prompt injection stops being a nuisance and starts being a breach.
How to fix it: It’s straightforward: Limit what the system can do, apply least-privilege access and treat model output as untrusted until it passes deterministic validation at the execution boundary. That will not make a compromised agent harmless, but it will materially reduce the blast radius.
Misconception #4: Using external data makes AI more reliable and more secure
Retrieval-augmented generation (RAG), where models pull in external data, improves accuracy but doesn’t make systems more secure. Research published by USENIX shows that corrupting a small number of knowledge-base entries is enough to reliably manipulate RAG outputs at scale.
Every data source you connect becomes a potential entry point. If that data is untrusted, outdated or manipulated, it can influence the model’s output in ways that are difficult to detect.
How to fix it: This is both a model problem and a data and supply chain problem. Treat external sources as dependencies that need governance. Apply controls for provenance, validation, write access, ingestion scanning, versioning, source separation and change management.
Misconception #5: Managed AI means the provider handles security
People often mistake managed services for outsourced security. In reality, responsibility is shared, but customer security responsibilities remain substantial.
The provider secures the service itself. You’re responsible for everything around it: What data goes in, who has access, what the model is allowed to do and how outputs are used.
How to fix it: Be explicit about what you own, clearly map the shared responsibility and don’t assume that it’s secure because it’s managed. Review the provider’s controls, then close the gaps in identity, data handling, configuration, monitoring and integration security yourself.
What every deployment should have in place
Most organizations I assess can list the AI they’ve deployed. Far fewer can tell me who owns it, what data it touches, what it’s capable of or what happens when it fails. That points to a governance problem.
The foundations aren’t especially complicated; they’re just unevenly applied.
At a minimum, you should have:
- A clear, executive-approved AI security posture and defined risk appetite, aligned to specific use cases and data types
- A complete inventory of your AI assets (models, prompts, agents, datasets, vector stores, connectors, service accounts and plugins), with named owners
- Threat models that define trust boundaries and enforce policy at predictable control points
- Strong integrity controls across your supply chain and data pipeline, including provenance, signing, scanning, lineage, versioning and, where appropriate, managed registries
- Least-privilege tool access for agents, with human-in-the-loop oversight for high-impact actions
- Validation layers on outputs before anything is executed, written or exposed to users
- Continuous evaluation and monitoring built into change management
- Incident runbooks that are tested in practice, including containment, safe-disable and rollback scenarios
Put these in place and you’re applying the same engineering discipline expected of any critical system.
The bottom line on AI security
AI security goes beyond securing only systems or models. Recognize that early, and you’ll be ahead of those who only learn it after something breaks.
This isn’t a one-off exercise. AI systems are always evolving, and security has to keep up. That means ongoing testing, including red teaming — deliberately trying to break the system to understand its weaknesses.
And if you can’t clearly account for your exposure across models, integrations, data pipelines and agents, that uncertainty is itself part of the risk.
Book your NTT DATA AI Security Risk Assessment today and get a clear picture of where risk is located throughout your AI stack.
