The introduction of virtual analysts, artificial intelligence and other advanced technologies into the Security Operations Center (SOC) is changing how we should think about maturity models. AI is replacing traditional human tasks, and when those tasks are automated the code effectively becomes the procedure. Is that a -1 or a +10 for security operations? Let's discuss that.
To see the big picture here, we should review what a maturity model is and why we are using them for formal security operations. A maturity model is a process methodology that drives good documentation, repeatability, metrics and continuous improvement. The assumption being that these are a proxy for effectiveness and efficiency. The most common model used in Security Operations is a variant of the Carnegie Mellon, Capability Maturity Model for Integration (CMMI). Many process methods focus on defect management, this is even more evident in the CMMI since it originated in the software industry.
In the early 2000’s, we started using CMMI at IBM, Big Blue insisted that we couldn’t offer a commercial service that wasn’t on a maturity path and they had adopted CMMI across the entire company at that point. We had, at that time, what seemed like a never-ending series of failures in our security monitoring services, and for each failure a new “bandage” in the form of a process or procedure was applied. After a few years we had an enormous list of processes and procedures, each connected to the other in a PERT chart of SOC formality. Most of these “bandages” were intended to provide guidance and support to analysts as they conducted security monitoring and to prevent predictable failures, so we could offer a consistent and repeatable service across shifts and customers.
To understand this better, let’s look at the 5 levels of the CMMI model:
- Initial (ad hoc)
- Managed (can be repeated)
- Defined (is repeated)
- Measured (is appropriately measured)
- Self-optimizing (measurements leads to improvements)
This well-defined approach seemed to be perfect. It allowed us to take junior analysts and empower them to have a consistent level of service delivery. We could repeat ourselves across customers. We might not deliver the most effective results, but we could at least be reasonably consistent. As it turns out, people don’t like working in such structured roles because there’s little room for creativity or curiosity. Not surprisingly, this gave rise to the 18-24 month security analyst turn-over phenomenon. Many early analysts came from help desk positions and were escaping “call resolution” metrics in the first place.
Our application of SOC maturity morphed over the years from solving consistency problems into consistently repeating the wrong things because they could be easily measured. When failures happened, we were now in the habit of applying the same “bandages” over and over. Meanwhile, the bad guys had moved on to new and better attack techniques. I have seen security operations teams follow maturity guidelines right down a black hole, when for example, a minor SIEM content change can take months, not the few hours it should take.
According to the HPE Security Operations Maturity report, the industry median maturity score is 1.4, or slightly better than ad-hoc. I’m only aware of 2 SOCs in the world that are CMMI 3.0. So, while across the industry we are measuring our repeatability and hoping that it equates to effectiveness and efficiency, we are still highly immature, and this is reflected in the almost daily breaches being reported. You can also see this in the multi-year sine wave of SOC capability many organizations experience; it goes something like this:
- New SOC or SOC rebuild
- Delivery challenges
- Maturity program
- Difficulty articulating ROI
- Cost reductions
With a virtual analyst, your SOC can now leap to CMMI level 5 for what was traditionally a human-only task. An AI-based virtual analyst, like the Respond Analyst, conducts deep analysis in a consistent fashion and learns rationally from experience. This approach provides effective monitoring in real time and puts EVERY SINGLE security-relevant event under scrutiny. Not only that, you liberate your people from rigorous process control, and allow them to hunt for novel or persistent attackers using their creativity and curiosity.
This will tip the balance towards the defender and we need all the help we can get!
Chris has over 30 years of experience in defensive information security; 14 years in the defense and intelligence community and 17 years in commercial industry. He has designed, built and managed global security operations centers and incident response teams for eight of the global fortune-50. As he often says, if you have complaints about today’s security operations model, you can partially blame him. It’s from his first-hand experience in learning the limitations of the man vs. data SecOps model that Chris leads product design and strategy for Respond Software.View all posts by Chris Calvert