The steering committee is five minutes from greenlighting a flagship AI feature. The slide on the screen shows a reassuring summary: “One-week AI red team, 27 jailbreaks found, all mitigated.” Screenshots of quirky prompt attacks get a few smiles. No one in the room asks the harder question: did anyone actually test how this thing behaves when a determined adversary leans on it for weeks, with real data, real fraud scenarios, and real incentives to break your business logic? This Wednesday “Headline” feature in Bare Metal Cyber Magazine is about that gap between the red team theater you can demo and the adversarial assurance you actually need in an AI-first world.
As organizations rush to embed artificial intelligence (AI) into customer channels, developer workflows, and internal decision-making, “we did a red team” has become the new comfort phrase. A burst of testing before launch, a handful of spectacular but narrow exploits, and a few new prompt filters can create the illusion of control. But AI-first systems are not just models and prompts. They are socio-technical stacks where identity, data access, plugins, and legacy controls all interact in ways your old testing regimes never had to consider. Attackers are already learning to live inside those interactions, not just inside the model.
For security and technology leaders, the next few years are going to hinge less on whether you “use AI safely” in the abstract and more on whether you build a credible adversarial testing function around the systems that matter. That means rethinking what red teaming looks like when the target is an evolving model pipeline rather than a static app, and pairing those red teams with “blue models” that help defend, simulate, and regression-test your AI stack at scale. The question is no longer “Did we run a red team exercise?” but “Do we have a living adversarial program that keeps pace with the AI we’re shipping?”
The New AI Attack Surface Isn’t The Demo
The easiest version of AI risk is the one you can screenshot: a large language model (LLM) persuaded to say something embarrassing, offensive, or simply “not on brand.” That is what most leaders see in red team readouts today. The reality is that the most damaging failures in AI-first systems rarely show up as a single shocking response. They emerge as quiet, systemic behaviors: models leaking sensitive context across sessions, copilots gradually normalizing risky workflows, or automated agents taking small but compounding actions in business systems that no one thought to constrain. The theater is the jailbreak. The real attack surface is everything around it.
In a typical AI rollout, the official risk story centers on prompt injection and content safety. Meanwhile, the architecture underneath is stitching together identity systems, retrieval-augmented generation (RAG) pipelines, plugins, and long-lived session state. A motivated adversary does not care that you blocked the specific prompt in last week’s demo. They care about whether they can use the model to infer business rules, enumerate internal resources, abuse plugins to move money or exfiltrate data, or subtly shape outputs that influence human decisions at scale. None of that is captured by a handful of screenshot-friendly tests.
The other blind spot is time. Most AI “red team” exercises are sprints measured in days, aimed at producing findings for a launch meeting. Attackers, by contrast, get to probe your AI system over months, across product updates, and often across tenants and regions. They can chain seemingly low-severity behaviors: a minor over-sharing of metadata here, a slightly over-permissive tool call there, a weak abuse detection threshold somewhere else. Leaders who only see a one-week snapshot of “we tried to break the chatbot” walk away with false confidence about systems that will face persistent, adaptive adversaries.
For an AI-first organization, the first mindset shift is simple but uncomfortable: the attack surface is not the demo environment, the scripted prompts, or even the base model alone. It is the living mesh of capabilities where the model mediates access to your data, your workflows, and your customers over time. Until your adversarial testing reflects that reality, the green checkmark on your AI red team slide is far more signal about theater than safety.
Red Teaming Systems, Not Just Models
When leaders hear “AI red teaming,” many still picture a model in isolation: a test harness feeding prompts to a single endpoint and rating the responses. That mental model was barely adequate for early chatbots and is completely wrong for today’s AI-first systems. The real target is a distributed architecture where models sit between users and a lattice of data sources, tools, and decision points. If your adversarial testing doesn’t reflect that architecture, you are effectively penetration testing a single microservice and calling it a review of the whole platform. The gap between those two views is where attackers thrive.
At a whiteboard level, most serious AI deployments now include several layers: base or fine-tuned models, orchestration logic, prompt templates, retrieval components, plugins or tools, data stores, logging and analytics, and the identity and policy layer tying it all together. Each layer has its own failure modes. A model with “safe” responses can still be used as a reasoning engine to discover misconfigurations in access control. A retrieval pipeline can be manipulated so that the model confidently surfaces outdated or sensitive information. A plugin that looks benign in isolation can become dangerous when called automatically in the context of a long-lived agent. Red teams that only probe the model miss the combinatorial risks that emerge from these interactions.
System-focused adversarial testing means constructing scenarios that cross these layers and mimic real attacker goals. Instead of “can we make the model say something offensive,” the questions shift to “can we induce the agent to call a payment API in an unintended way,” “can we cause cross-tenant data exposure through a RAG misconfiguration,” or “can we nudge the model into recommending a series of policy exceptions that weaken controls over time.” That requires red teams to understand the product, the business flows, and the underlying infrastructure as well as the model. It also means instrumenting the stack so that tests reveal where protections actually fail.
The payoff for leaders is a different kind of assurance. When you treat AI as a socio-technical system, red team findings start mapping directly to architectural decisions, access patterns, and governance gaps you can do something about. You are no longer reacting to clever prompt transcripts. You are seeing how an intelligent adversary would move through your AI stack, and where you must strengthen it before customers, partners, and regulators discover those weaknesses for you.
Building the AI Adversary Bench
Once you accept that AI-first systems are the new attack surface, the obvious next question is who is going to attack them for you before real adversaries do. Most organizations do not yet have a team that can credibly do this work. Traditional application security testers bring deep knowledge of web stacks, APIs, and abuse patterns, but may not be fluent in model behavior or data science. Machine learning specialists understand models and training data but may not think like attackers. Product managers grasp customer journeys but rarely frame them as adversarial scenarios. An effective AI red team is not a single superhero profile; it is a bench that spans these disciplines and can operate together.
Building that bench starts with reframing the mission. The job is not “find the funniest jailbreaks,” it is “discover the ways AI-first systems can be bent toward real attacker goals and quantify that risk.” That means hiring or upskilling people who are comfortable moving between abstractions: reading model eval reports one moment, tracing a tool call through the policy engine the next, and then sitting with product and legal to understand how a misuse case could manifest in the wild. It also means designing engagement models that avoid the classic trap where red teaming is a sporadic, compliance-driven event. For AI, adversarial testing needs to be embedded into model lifecycle management, with regular exercises tied to major versions, new tools, and shifts in data access.
Leaders also have to protect this function from being captured by optics. If you measure AI red teaming solely by the number of findings or the shock value of the examples, you will get exactly that. Instead, you want to reward depth and relevance: tests that map to key business processes, scenarios that stress cross-domain boundaries, and work that feeds directly into guardrail design, access reviews, and blue team playbooks. Some organizations are already experimenting with hybrid models where a centralized AI red team sets standards and frameworks, while embedded specialists sit close to high-risk product teams. Whatever you choose, the benchmark is simple: would this team find the issues that keep you up at night before someone else does?
Blue Models: Using AI to Defend AI
As AI-first systems scale, human-led adversarial testing alone cannot keep up. Models change, prompts evolve, new tools are wired in, and data access boundaries shift month to month. If every meaningful test requires a human red teamer in the loop, your assurance will always lag your release cadence. This is where the idea of “blue models” becomes essential: using AI on the defender side to simulate attacks, stress-test guardrails, and continuously exercise your AI stack, not just at launch but throughout its life. Instead of assuming the model is a static target, you start treating adversarial behavior as something you can generate, replay, and learn from at scale.
At a practical level, blue models show up in several places. One pattern is automated scenario generation: you feed a blue-side model descriptions of your AI features, high-level threat models, and historical red team findings, then have it propose new attack prompts, tool misuse sequences, or data exfiltration paths. Another pattern is behavioral monitoring, where models help classify and cluster real user interactions to highlight suspicious patterns that human analysts would miss, such as gradual probing of edge cases or repeated attempts to coax the system into policy-violating behavior. A third pattern is regression: when you update a model, change a prompt, or add a plugin, blue models can replay a library of known-bad and near-miss interactions to see what actually improved and what broke.
For leaders, the point is not to automate red teams out of existence. It is to build a feedback loop where human red teams discover the most meaningful failure modes, encode them into tests and adversarial templates, and then hand those to blue models that run continuously. That loop lets you treat AI risk more like you treat modern software quality: with a mix of exploratory testing, automated checks, and telemetry-driven improvement. It also creates a new class of metrics. Instead of just counting vulnerabilities, you can track how quickly blue models detect and surface new adversarial behaviors, how often guardrails catch them, and how those trends move over time. In an AI-first world, your defensive posture will depend as much on the intelligence of your blue models as on the cleverness of your red ones.
Governance, Metrics, and Not Killing Velocity
The moment you formalize AI red teaming and blue models, someone will ask the uncomfortable question: “Does this slow us down?” If the answer is yes, the function will be sidelined or reduced to a checkbox as soon as product pressure spikes. Governance for adversarial testing in an AI-first world has to be designed with velocity in mind. That means shifting from binary gates to risk-based guardrails, from ad hoc review boards to predictable triggers, and from vanity metrics to measures that actually reflect reduced exposure. Leaders who get this wrong either smother AI innovation under security process or greenlight systems with a thin layer of theater on top.
A durable pattern is to treat adversarial testing as part of your AI lifecycle rather than a one-off hurdle. High-risk features—those that touch money, safety, regulated data, or customer trust—get deeper and more frequent red and blue coverage. Lower-risk experiments may have lighter requirements, focused on content safety and basic misuse. Clear criteria for each tier let product teams know what to expect, while still giving security the authority to insist on serious testing where it matters. Crucially, governance needs to specify not just “we did a red team” but what evidence is required: summarized adversarial scenarios tied to business impacts, regression results from blue models, and a traceable set of mitigations or compensating controls. That is the packet you want in front of an executive sponsor or board committee.
Metrics are where many organizations default to what is easy to count rather than what is meaningful. Number of jailbreaks found, prompts tested, or models scanned tell you almost nothing about actual residual risk. Better indicators might include coverage of high-value flows, time-to-mitigate for serious adversarial findings, stability of defenses across model and prompt updates, and the rate at which blue models are catching issues before customers or external researchers do. When those metrics are visible to product and platform leaders, adversarial testing stops being an obstacle and becomes part of how teams prove they can ship AI features at speed without gambling the franchise.
Where Leaders Go From Here
At its heart, this topic is about refusing to confuse reassuring theater with real assurance in an AI-first world. The scene in that steering committee room, where a week of “AI red teaming” is treated as a stamp of safety, is going to replay across countless organizations over the next few years. What separates the resilient ones from the rest is whether those leaders are willing to ask what was actually tested, how the system behaves over time, and whether they have any intelligence on how an adaptive adversary would work through their AI stack. Greenlighting AI on the strength of a few jailbreak screenshots is a choice, not an inevitability.
When you internalize the argument of “Red Team, Blue Models,” the mental model shifts. AI is no longer a point feature you occasionally poke at; it is a socio-technical system that demands continuous adversarial attention. Red teams stop being stunt performers and become partners in stress-testing the architecture and business flows that matter. Blue models stop being a buzzword and become part of how you monitor, simulate, and regression-test your risk posture as models, prompts, and tools evolve. Governance stops being a ticket queue and becomes a living set of expectations about evidence, coverage, and response time that product teams can plan around. The result is not slower AI, but AI with a clearer envelope of acceptable risk.
For security and technology leaders, the next step is less about launching a new committee and more about changing the questions you bring into the room. Instead of “Did we run a red team?”, ask “What attacker goals did we simulate, what systems did we exercise, and how will blue models keep pressure on this surface after launch?” Ask your teams where AI is already mediating access to data, money, and trust without any structured adversarial testing around it. Then decide, explicitly, which of those systems you are willing to leave in that state. The organizations that do this work now will not be the ones explaining to regulators and boards, a year from today, why their AI stack failed in ways no one had ever thought to test.
