That’s a Great Question - AI Behaves Differently, How Should My Security Approach Change?

Unfortunately, even after years of championing a “build security in” mindset, many security professionals will tell you that it can still feel like security gets bolted on once a system is already live. We know the cost of addressing issues late in the cycle, which is why so many teams embraced DevSecOps and began weaving security into design and build from the outset. Yet when new technology arrives and the pressure to move fast is high, it is common for teams to rush ahead with promising ideas and innovations with a plan to tighten security after an initiative is already working.

Which brings us to today’s great question that I was asked after a recent talk: Why is it dangerous to treat AI systems like traditional software? And how do we explain that difference to non-technical leaders who must govern AI but do not have hours to unpack the technical details?

The short answer is that generative AI is a different kind of system. It does not behave like traditional, deterministic code. Large language models (LLMs) work on probabilities rather than fixed rules, which means the same prompt can produce different answers at different times. LLM behavior and responses can shift based on how a question is phrased, what information the system retrieves, or changes made to the way the model has been configured or fine tuned. Even when the core LLM has not changed, new inputs can push it into different patterns it learned during training, which can cause the system to respond differently than expected. For example, a customer support model that normally provides safe, compliant answers might inadvertently reveal sensitive internal details or offer incorrect guidance because a routine inquiry causes it to generate the wrong kind of response.

Traditional vulnerability scanners and rules based monitoring tools were never designed to evaluate these kinds of risks. Without AI aware threat modeling, testing, and runtime controls, organizations can end up with blind spots that remain hidden until they show up in financial, operational, or reputational impacts.

To explain this clearly to senior leaders, I find it helpful to start with how AI models work and then build toward what that means for enterprise risk.

AI Isn’t Written, It’s Trained

Software engineers build traditional applications by writing code line by line, and each function tends to be deterministic. For example, if the code is “when X happens, do Y,” then every time X occurs the system does Y. When you type an “A” on your keyboard in a word-processor, an “A” appears on screen. In traditional software security, you can locate a missing input-validation check, identify an access-control flaw, or fix a misconfiguration in an API gateway and then use testing to validate the fix worked.

Generative AI systems follow a different architecture. They are trained, not coded, meaning models learn from large datasets and tuned optimization routines (such as supervised learning, self-supervised learning, or reinforcement learning from human feedback) and then generate new content that is statistically plausible rather than strictly rule-based. Instead of a single guaranteed output, you get a spectrum of possible responses depending on the model’s learned patterns, prompt phrasing, context, and retrieval of additional information.

This variability is normal for AI and must be tested for. Because as NIST explains:

“AI systems … may be trained on data that can change over time, sometimes significantly and unexpectedly, affecting system functionality and trustworthiness in ways that are hard to understand.” (AI RMF 1.0, Executive Summary)

Because AI behavior depends on learned patterns and evolving data rather than static code rules, a system that appears secure in one environment or at one time may behave differently, and sometimes insecurely, in another. This is not simply a bug you can patch; it’s a property of how AI systems operate.

The Risk Surface Expands Beyond Traditional Boundaries

It can be helpful to explain the difference this way. Imagine the company is rolling out a customer support chatbot built the traditional way. The security risk is mostly limited to the software itself: the code the dev team wrote, the information the bot is allowed to use, and any systems it is intentionally connected to. If something goes wrong, it is usually due to a specific bug or misconfiguration that testing can find and fix.

A generative AI system’s behavior depends on many things outside the code. Including:

• Training data from outside the company may contain errors or bias that affect how the AI behaves
• Normal variations in how people phrase questions can unintentionally push the AI toward unsafe or misleading responses
• The more internal systems the AI can touch, the greater the risk if those connections are not tightly controlled
• Confident but incorrect responses can introduce compliance, operational, or brand exposure

Traditional software also operates within an ecosystem of libraries, data sources, and integrations. But with AI, the ecosystem becomes far more dynamic. The system’s behavior is shaped not only by the code and its dependencies, but also by constantly changing inputs, unpredictable interpretations, and learned patterns the company does not directly control, and in some cases the AI’s ability to take actions on its own when connected to internal tools or workflows. That means security teams must evaluate and protect the entire environment of data, models, instructions, permissions, and interactions that influence how the AI behaves and what it can do.

How to Explain This to Senior Leaders

When you sit down with the board, it can be helpful to summarize the above in three simple points.

1. AI systems are not deterministic. They do not behave like software, so old assumptions about predictability and test coverage no longer apply.
2. The risk surface is larger. We are not just securing code. We are securing data pipelines, foundation models, autonomy capabilities, and interactions with users and external systems.
3. Traditional security tools were not built for this. They cannot test an AI system’s reasoning, detect malicious prompt patterns, evaluate hallucination risk, or see when a model starts behaving differently after a deployment change. Which is why we need security tools that are AI ready.

When boards and executives understand these three points, they begin to see why AI security isn’t the same as traditional digital security. And why it’s so important to build security into the AI lifecycle.

If your teams are building or deploying AI today, a good next step is to map out where your current security controls rely on determinism. Any place where AI driven non-deterministic decision making plays a role will need AI-specific testing, controls, and monitoring. And if you are preparing to brief your board, consider framing AI risk not as a technical detail but as a governance responsibility tied directly to trust, resilience, and operational continuity. The more we treat AI as a dynamic system rather than a static product, the safer and more resilient our deployments will be.