Today’s modern cybersecurity solutions must scale to unparalleled levels due to constantly expanding attack surfaces resulting in enormous volumes of diverse data to be processed. Scale issues have migrated from just the sheer volume of traffic, such as IOT led DDoS attacks and the traffic from multiple devices, to the need for absolute speed in identifying and catching the bad guys.
Long tail analysis is narrowed down to looking for very weak signals from attackers who are technologically savvy enough to stay under your radar and remain undetected.
But, what’s the most efficient and best way to accomplish what can be a time-consuming and a highly repetitive tasks?
What is Long Tail Analysis?
You might be wondering what the theory is behind long tail analysis, even though you’re familiar with the term and could already be performing these actions frequently in your security environment. The term Long Tail first emerged in 2004 and was created by Wired editor-in-chief, Chris Anderson to describe “the new marketplace.” His theory is that our culture and economy is increasingly shifting away from a focus on a relatively small number of "hits" (mainstream products and markets) at the head of the demand curve and toward a huge number of niches in the tail.
In a nutshell and from a visual standpoint, this is how we explain long tail analysis in cybersecurity: You’re threat hunting for those least common events that will be the most useful in understanding anomalous behaviour in your environments.
Finding Needles in Stacks of Needles
Consider the mountains of data generated from all your security sources. It’s extremely challenging to extract weak signals while avoiding all the false positives. Our attempt to resolve this challenge is to provide analysts with banks of monitors displaying different dashboards they need to be familiar with in order to detect malicious patterns. As you know, this doesn’t scale. We cannot expect a person to react to these dashboards consistently. Nor do we expect them to “do all the things”.
Instead, experienced analysts enjoy digging into the data. They'll pivot into one of the many security solutions used to combat cybersecurity threats such as log management solutions, packet analysis platforms, and even some endpoint agents all designed to record and playback a historical record. We break down common behaviours looking for those outliers. We zero in on these ‘niche’ activities and understand them one at a time. Unfortunately, we can’t always get to each permutation and they are left unresolved.
Four Long Steps of Long Tail Analysis in the SOC
If you are unfamiliar with long tail analysis, here are 4 steps of how a typical analyst will work through it:
Step 1: First, you identify events of interest like a user authentication or web site connections. Then, you determine how to aggregate the events in a way that provides enough meaning for analysis. Example: Graph user account by the number of authentication events or web domains by the number of connections.
Step 2: Once the aggregated data is grouped together, the distribution might be skewed in a particular direction with a long tail either to the left or right. You might be particularly interested in the objects that fall within that long tail. These are the objects that are extracted, in table format, for further analysis.
Step 3: For each object, you investigate as required. For authentications, you would look at the account owner, the number of authentication events, the purpose of the account. All with the intended goal of understanding why that specific behaviour is occurring.
Step 4: You then decide what actions to take and move on to the next object. Typically, the next steps include working with incident responders or your IT team. Alternatively, you might decide to simply ignore the event and repeat Step 3 with the next object.
Is There a Better Solution?
At Respond Software, we’re confident that long tail analysis can be automated to make your team more efficient at threat hunting. As we continue to build Respond Analyst modules, we move closer to delivering on that promise -- and dramatically improve your ability to defend your business.
John Petropoulos is a security architect with over 16 years of experience working with all types of security operation centers, large and small. Specializing in breach detection and incident response, John has designed content development strategies and integration approaches that support some of the largest security operations in the world. Here at Respond Software, John is developing probabilistic models based on his experiences that evolved while working with a wide array of products and environments.View all posts by John Petropolous