The first message drops into the on-call channel at 2:17 a.m. A monitoring alert has tipped into “major incident” territory, customer-facing services are glitching, and the first responder is already juggling three windows: the runbook, the Slack war room, and a growing list of people they “should probably loop in.” The incident command plan says this is simple: trigger the contact list, start the bridge, follow the playbook. In reality, half the names on the list are asleep, one key SME recently moved teams, and a director is already DM’ing side-channel updates to an executive who wants answers before you have facts. This is the gap between incident command on paper and incident command as humans actually experience it.
This Wednesday “Headline” feature in Bare Metal Cyber Magazine looks squarely at that gap. It focuses on incident command as a human system: the messy mix of trust, informal authority, communication habits, and fatigue that shows up when your organization is under pressure. The core idea is simple but uncomfortable for many leaders: your beautifully formatted escalation matrix is not the thing that saves you at three in the morning. The people, relationships, and unwritten rules are.
For security and technology leaders, this matters over the next few years because incidents are getting more complex, more cross-cutting, and more visible to boards and regulators. You are no longer managing a neat “security issue”; you are orchestrating a temporary, high-stakes network that spans product, operations, legal, communications, vendors, and executives. How you design that network, how you acknowledge the reality of shadow power and human limits, and how you practice together will determine whether your next major incident feels like a controlled response or an avoidable train wreck.
The Myth of the Perfect Phone Tree
Most organizations feel oddly reassured by their incident contact lists. There is comfort in seeing names, roles, and numbers neatly arranged in a spreadsheet or a runbook, with color-coding and escalation tiers that suggest order. Leaders ask “Do we have an escalation matrix?” and, once the answer is yes, mentally move incident command into the “handled” category. On paper, the logic is clean: detect, page, escalate, convene, decide. But that tidy sequence is built on assumptions about people being reachable, available, and correctly mapped to the work, and those assumptions break down quickly in the real world.
The first crack usually appears around currency. The contact list is only as accurate as your last reorg, vendor change, or offboarding. People move to new teams but stay in old distribution lists. Senior individual contributors become implicit “fixers” but never appear in any formal escalation path. A managed security provider quietly changes their on-call model, and suddenly your “twenty-four seven” number routes to a voicemail tree. Time zones and holidays add another layer: the matrix says “escalate to the regional lead,” but the regional lead is offline, the backup is unknown, and the person who actually responds is whoever happens to see the alert on their phone.
Leaders also overestimate how people behave when the alert hits. In theory, the first responder follows the documented path. In reality, they often start with their mental model of “who can actually fix this,” which blends past incidents, personal trust, and perceived responsiveness. That might mean pinging a favorite principal engineer before paging the official incident commander, or DM’ing a product manager who always “gets things moving” instead of going through Service Management. The contact list becomes a suggestion, not a map, and the real escalation flow is shaped by relationships rather than boxes on a chart.
The result is a subtle but important gap: you think you have a system; what you actually have is a collection of artifacts that loosely describe one possible system. The myth of the perfect phone tree is dangerous because it blinds leaders to the need for continuous validation. The right move is not to chase perfect lists but to treat them as hypotheses. You pressure-test them through drills, observe how people really coordinate, and then redesign the escalation model to reflect the network you actually have, not the one you wish you had.
Org Charts, Shadow Power, and Real Authority
In every major incident, there is the official incident commander and then there are the people everyone actually listens to. The official structure tends to mirror the org chart: managers at the top, team leads in the middle, individual contributors executing tasks. That structure is what gets documented in the incident management policy and what auditors see on paper. But under real pressure, the gravitational center often shifts toward senior engineers, veteran analysts, or key business operators whose authority comes from lived credibility, not title. When they speak, the room adjusts. When they disagree with the “official” plan, people hesitate to move.
This shadow power is not inherently bad. In fact, it is often what saves you when the incident is ambiguous and time is short. The problem is the mismatch between formal and informal authority. An incident commander who holds the role by rotation but lacks experience or trust can find themselves reduced to note-taker while others drive decisions. Conversely, a strong informal leader without a clear role may take over in ways that bypass risk, legal, or communications, creating downstream surprises for executives and regulators. Leaders are often surprised to discover that the person their process names as “decision maker” is actually treated as an administrator by the people doing the work.
You see the tension most clearly when the stakes cross organizational boundaries. A major security incident pulls in infrastructure, application teams, customer support, legal, communications, and external partners. Each group brings its own hierarchy and its own “go-to” people. If your incident command model assumes that a single role from the security organization can simply direct all of these groups, you are betting against years of established habit and status. The people who control deployments, the ones who understand critical customer workflows, the lawyer who knows your regulatory exposure by heart—these are power centers that need to be explicitly integrated into the command structure, not treated as optional guests.
For leaders, the practical move is to stop designing incident command around the nominal org chart and start designing around real influence. That means mapping who actually gets called in big moments, who people defer to on technical and business decisions, and where conflicts typically arise. Then you formalize those realities: give informal leaders explicit roles, pair less experienced incident commanders with seasoned “anchors,” and pre-negotiate how decisions will be made when technical urgency collides with business or legal risk. The goal is not to eliminate shadow power but to bring it into the light, so that in the next incident you are not discovering your true hierarchy in the middle of the storm.
Communication Overload and the Coordination Tax
Once an incident crosses a certain threshold, communication becomes its own parallel incident. The first war room fills up, someone spins up a “side channel” for just the core technical responders, executives ask for a separate bridge, and email threads bloom as external partners and customers demand updates. Each new channel seems reasonable in isolation. Together, they create a coordination tax that quietly drains attention from the one thing that actually matters in the early phase of a security event: reducing uncertainty and stopping the bleeding. The more rooms you have to “keep in sync,” the more your best people end up narrating instead of solving.
You can see this play out minute by minute. The incident commander tries to track workstreams in chat, answer questions from leaders who join late, and keep a running log in the ticketing system because “we’ll need this for the post-incident review.” A senior engineer spends half their time rewriting the same status for different audiences: one for the technical room, one for customer success, one for an executive who wants a simple “green, yellow, red” answer. Legal or communications may ask for carefully worded statements that slow down technical updates. None of these asks are unreasonable, but together they fragment focus and introduce delay, especially when every clarification requires handoffs across time zones and org boundaries.
The coordination tax is not just a productivity problem; it shapes decisions. When communication overhead is high, teams default to the path that is easiest to explain rather than the one that is technically optimal. Leaders may push for premature commitments—“we will be fully resolved by morning”—because they need something clean to tell the board, even if the underlying uncertainty is still high. Over the long term, teams internalize that incident response is as much about performance as it is about containment, which can make people less willing to surface ambiguity or bad news early. The leadership challenge is to simplify the communication surface during incidents, deliberately assign roles for upward and outward updates, and protect a core group whose attention is shielded from the noise. That choice will not show up in any single metric, but it is often the difference between a response that feels controlled and one that slowly unravels under the weight of its own messaging.
Designing Incident Command for When People Aren’t There
Every incident plan quietly assumes that the right people will be available. The rotation is staffed, the on-call calendar is full, and the primary and secondary contacts are neatly listed. Reality is less kind. People are on planes, in hospitals, at school events, deep in burnout, or halfway out the door to a new employer. Critical knowledge lives in a departing architect’s head or in a chat thread from three years ago. Vendors run “twenty-four seven” support that really means “best effort after business hours.” When you design incident command around ideal availability, you are designing for the one day a year when everything lines up, not the other 364.
A more honest approach starts from the assumption of partial staffing. You ask, in concrete terms: if this type of incident happens and our preferred subject-matter expert is not reachable for six hours, what can we still do safely? That pushes you toward patterns like progressive containment actions with clear risk bounds, pre-authorized “safe defaults” for traffic limiting or feature flags, and documented decision trees that a competent but less experienced engineer can follow. It also forces you to confront brittle single points of failure: the one database wizard, the lone identity engineer, the only communications lead trusted by executives. Wherever your plan relies on a specific name, you have fragility.
Designing for absence also means making rotations and redundancy a governance question, not an individual hero problem. Leaders can choose to fund paired incident commander models, where a more seasoned person acts as a coach and backup across time zones. They can normalize handovers mid-incident instead of assuming the same people will run a twelve-hour marathon. Vendor and partner contracts can be evaluated against real-world incident patterns, not just service level language. The organizations that handle incidents well over time are not the ones that always have the “right” person online; they are the ones that have made peace with human limits and built a command system that bends gracefully when people are missing, tired, or transitioning out.
Turning Drills into Muscle Memory, Not Theater
Most organizations can point to a calendar invite and say, “We do tabletops.” Too often, those exercises are scripted theater. The scenario is tidy, the injects are predictable, and everyone around the table knows there is no real stake. People speak in abstractions: “We’d engage vendor X,” “We’d page the on-call team,” “We’d loop in legal.” The exercise ends with everyone agreeing that “communication is key” and a few action items that quietly evaporate. What never gets tested is the messy human reality: who actually gets called first, which channels light up, where confusion appears, and how authority is contested when opinions diverge.
To build real muscle memory, drills have to follow the same communication paths and social dynamics as real incidents. That means running live-channel exercises where you use the actual war room tools, page the genuine on-call rotation, and watch how people route information. It means including non-technical stakeholders who will be part of the real response: customer support, legal, communications, and business owners. It means letting people feel genuine time pressure and ambiguity instead of handing them neat, linear clues. When you run exercises this way, you quickly discover things that no policy document will reveal: a dependency on a single person, a chat channel nobody monitors overnight, a habitual tendency to over-invite executives.
Leaders also need to treat after-action reviews as a design tool, not a blame arena. The most valuable part of a drill is not the scorecard; it is the narrative: who said what when, which decisions were hard, where people hesitated, and why. If the review fixates on technical details and ignores the human factors—trust, role clarity, psychological safety—you miss the real leverage points. Over time, a cadence of realistic drills and honest reviews creates a shared mental model of incident command across teams. People learn what “good” looks like in practice, not just in policy, and their reflexes begin to align even when the scenario is one you never rehearsed.
Metrics, Ownership, and the Long Game
If incident command is treated as a project, it will always feel half-built. Someone writes a policy, another team configures the tooling, a few people run a pilot, and then ownership diffuses into the background. The result is a system that decays quietly: runbooks drift out of sync with architecture, rotations become political compromises, and lessons from incidents never quite make it into how you fund or structure teams. Leaders sometimes try to fix this with more dashboards—mean time to detect, mean time to respond, number of major incidents—but those metrics often measure what is easy to count, not what actually makes your organization more resilient.
A better starting point is to decide, explicitly, who owns the incident command system as a product. That owner can sit in security, operations, or a shared reliability function, but they need a mandate that goes beyond tooling. Their job is to understand how incidents actually unfold across the organization, curate and evolve the playbooks, socialize norms for roles and behavior, and ensure that drills, rotations, and communication patterns are aligned. Crucially, they also need the authority to say no—to resist turning every executive preference into a hard rule that bloats the process and increases the coordination tax.
On the metrics side, the most useful signals tend to be qualitative before they are quantitative. Did responders feel they had clear authority and priorities? Were decisions documented well enough that someone could reconstruct the incident a month later without oral history? Did business and technical leaders come away with the same understanding of what happened and why? Over time, you can correlate these signals with more traditional measures—time to mitigate, time to contain, customer impact—but the point is to treat metrics as feedback on human systems, not just stopwatches. When leaders frame incident command as a long-term organizational capability with clear ownership, evolving design, and thoughtful measures, they stop chasing the illusion of perfect control and start building something more realistic and durable: a system that gets a little better, and a little more humane, every time it is used.
Where Leaders Go From Here
At its heart, this topic is about admitting that incident command is a human system first and a technical system second. The escalation matrices, on-call rotations, and finely tuned tools all sit on top of real people with uneven availability, informal influence, and emotional limits. When the alert hits at 2:17 a.m., what carries you is not the document itself but the web of trust, clarity, and habits you have built around it. Leaders who see that clearly stop treating incident response as a compliance checkbox and start treating it as a living capability that needs care, ownership, and iteration.
Tying back to that early-morning scene, the difference between chaos and controlled urgency is rarely a single heroic responder. It is whether the organization has acknowledged shadow power, simplified communications, designed for absence, and practiced together in ways that feel real. In those environments, the incident commander is not fighting the org chart, the tools, and the culture just to get basic decisions made. They are supported by a system that expects ambiguity, routes authority where it actually lives, and allows people to hand off work without guilt or drama.
When you internalize this view of incident command, your leadership lens shifts. You start asking which behaviors your current design rewards, where you are still betting on individuals instead of systems, and how you can reduce the coordination tax so responders can think. A simple next move is to bring a few questions to your team: whose authority really matters in our biggest incidents, where are we quietly relying on single people, and how could our next drill expose—not hide—the messy reality of how we work today. The answers will tell you where to start redesigning, before the next 2:17 a.m. test arrives.