High Volume Cybersecurity Event Feeds

John Petropolous
by John Petropolous
category Perspective

Recently, I’ve found myself reminiscing about my early experiences as a security operations center (SOC) analyst and how much the industry has changed in the last 15 years. I can’t help but notice that despite all the technology we’ve thrown at the SOC, we really haven’t changed things that much. We’re still trying to ingest feeds at hundreds and thousands of gigabytes per day while expecting analysts to consistently and accurately sift through them, separating legitimate alerts from suspicious ones. We’ve invested time in processing events, correlating them and diminishing the fire hose. However, despite building logic funnels that reduce manual analysis, analysts still struggle with consistency and the fatigue caused by all these alerts.

Take Network IPS/IDS events as an example. At first, we find misconfigurations in our environments. Trying to fix them, often becomes such an insurmountable project that analysts quickly give up on correcting the misconfiguration in favor of ignoring them at the console. As the IPS/IDS continues to mature, analysts then begin to discover nuanced patterns but the technology does not provide enough evidence to thoroughly investigate and understand alerts. With limited information and capabilities, we analysts begin to believe that there is a problem with the signature that cannot be adjusted. Once again, the only conclusion for us battered analysts is to quickly ignore the alerts and move along to the next alert on the console.

Inevitably, the security monitoring program console is overrun with meaningless events that become a bore to manage. Analysts then push for a quick fix, spawning the classic arguments with security engineering. For example:
“The signature is too noisy”
“The signature provides no value”
“The signature floods the console.”

Through these fallacious arguments, we drive our teams to disable signatures and filter out a wide portion of our visibility, just so we can physically see other events in the console. Think about this for a minute. How crazy is it, that in this day and age we are blinding ourselves just so we can see?

The problem is clear; humans can’t deal with repetitive tasks across high volumes. If it’s nothing else, it’s boring — and people don’t do well with boring. I’m declaring that we need to move past this. It’s time to automate security operations.

It’s time to free the analysts. It’s the end of the console.