This is the true story of how the Security War Room was born, at least for me…
In 2001, I was a relatively new Security Operations Center manager for IBM’s Managed Security Services Delivery (yay, MSSD!). On a Friday night relaxing at home, I was a few beers into my evening, when the phone rang. What came next was an instantaneous sobering moment. On the phone was my boss’s boss, and his first sentence went something like this, “Why am I calling you instead of you calling me?” This question coming from your boss’s superior is never how you want a conversation like this to start. Taken by surprise, I stammered out a lame response.
He then informed me that the Internet had “melted.” Oddly enough, he discovered the problem because his neighbor, whose Internet had evidently melted, had stopped by to ask him if he knew what was happening. Since I was his operational security manager, he expected me to know the answer right off the bat (and, of course, I didn’t). Perplexed and somewhat panicky, at that point, I called into our SOC to find out what was going on. To my surprise, they told me it was a quiet night with absolutely “no problems.” They said it was much quieter than usual. So quiet, in fact, that NO EVENTS were coming in at all! Given the volume of events they typically had to deal with, this unexpected but welcomed lack of activity felt like a badly needed break for them. And, no one thought to ask why.
The reality, this was the Code Red worm and the informal beginning of what I often refer to as the “worm years,” where the Internet “melted” on average every 3-4 months. Since we conducted security monitoring over the Internet, and the melted Internet wasn’t routing packets, we had nothing to monitor. So, when I called my boss’s boss back to give him an update, he requested that I report to work immediately and establish a War Room to manage the situation both for our internal security as well as for our customers.
My punishment for this failure turned out to be a very positive step in the right direction of security operations. I was subsequently given the responsibility of leading the After-Action Review (AAR). The sole purpose of this task force was to identify ways to prevent our organization from being blindsided again and, more importantly, recommend how best to respond to these critical situations in the future. As it turned out, we took a significant leap forward by assembling a War Room with our senior team to help manage the situation. In fact, that collaboration became an obvious recommendation that our company would ultimately formalize, document and even require rehearsals for these types of coordinated responses.
It was the job of those involved in the War Room to coordinate all aspects of the situation which included:
- Internal information gathering
- Internal communication to executives and employees
- Shared situational awareness across teams
- External vendor coordination to understand the appropriate remediation steps
- External communication to our customers, to the media, and our PR public team
We, fortunately, learned very quickly that the network and IT operations teams already had the call trees, process and procedures in place to assemble a remediation response rapidly. We only had to effectively connect with them and help with the subject matter. This was before ITIL was a common methodology.
On the Internet today, we are fighting the bad guys daily. Our SOC operational security teams are burdened, for the most part, with monitoring an overwhelming amount of security events and not situations. Given my experience, it seems to me that enterprise organizations would greatly benefit from formalizing the War Room as a “business as usual” function. By doing this, they could better manage critical situations, with the confidence of well-rehearsed coordination in a much more effective and timely manner.
Chris has over 30 years of experience in defensive information security; 14 years in the defense and intelligence community and 17 years in commercial industry. He has designed, built and managed global security operations centers and incident response teams for eight of the global fortune-50. As he often says, if you have complaints about today’s security operations model, you can partially blame him. It’s from his first-hand experience in learning the limitations of the man vs. data SecOps model that Chris leads product design and strategy for Respond Software.View all posts by Chris Calvert