Early in my corporate career, I was promoted to the first SOC manager for IBM’s Managed Security Services. Prior to building that SOC, our security monitoring services were provided by the NOC.
This was back in 2000, the early days of security operations, before we’d figured it all out.
This SOC team was tasked with monitoring security events in real-time to determine which were important enough to be escalated to our customers for deeper investigation.
Within weeks, the team pointed out to me that they were completely overwhelmed by the volume of events. In fact, when I pushed back on the team they said, “You try it!” So, I did, and it turned out that 2,400 events per hour will make you crazy in less than 20 minutes (and they knew that). This led me to the question, “What is too much volume for a SOC analyst and how do you measure it?” This is when I began measuring Events Per Analyst Hour (EPAH).
Now that I have been measuring humans analyzing events for almost 20 years, the “green zone” for EPAH is between 75 and 150. And, those events can usually be collapsed into 12-15 different simultaneous potential attacks to analyze. An experienced analyst can usually handle around 150 events and a junior analyst will struggle with 75 events.
To demonstrate the challenge this limitation poses for Security Operations monitoring, let’s look at the hard numbers to bring this into focus:
- A mature SOC provides 24 x 7 x 365 coverage
- It takes a minimum of 10 FTE (really 12 with management) to cover that at 1 analyst per hour
- IDS/IPS alone can produce 100+M events in any given month
- Many other data sources are just as loud, so assume 300-500M events per month for many enterprise SOCs
- At 500M events per month, that equals 694,444 events per analyst hour
Clearly that scenario doesn’t work. So, if you implement an event funnel to reduce the above by 4,629x down to 150 EPAH, what did you miss? I’m not sure, how about you?
When the capacity of humans doing security monitoring is exceeded, it’s obvious. On the other hand, what was ignored and missed, that should have been spotted? Here’s the catch — you won’t know until it’s too late. While EPAH is a critical measure of what human analysts are capable of, it doesn’t tell you anything about what you ignored. This is the fundamental limitation of console monitoring.
Even if humans were perfectly effective at monitoring (see my previous article “How Human Factors Hurt the SOC”) catching capable attackers is nearly impossible due to the small amount of total possible attack traffic actually observed. This is borne out by the consistent truth that most attacks are caught when new stolen items show up in the darknet.
Chris has over 30 years of experience in defensive information security; 14 years in the defense and intelligence community and 17 years in commercial industry. He has designed, built and managed global security operations centers and incident response teams for eight of the global fortune-50. As he often says, if you have complaints about today’s security operations model, you can partially blame him. It’s from his first-hand experience in learning the limitations of the man vs. data SecOps model that Chris leads product design and strategy for Respond Software.View all posts by Chris Calvert