Keeping NCAA football fans safe from cyber threats on game day

Did you happen to catch the Clemson vs Alabama NCAA Championship game Monday night? While the Clemson upset was thrilling to watch, I bet most fans had no idea there was another team behind the scenes, hard at work, keeping attendees and the stadium safe from cyber threats.

Yep, our diligent Respond Analyst™ was working away in a state-of-the-art Security Situation Room live during the entire game day experience. We had the privilege to partner with students from Norwich University in this onsite cyber-offensive in one of the newer football stadiums.

Cybersecurity is becoming more critical than ever before at these big events. Keeping close to 100,000 fans streaming video, playing social streams on the jumbotrons, and communication on the field has turned sporting events into a serious technical infrastructure. The potential for malicious malware attacks or other security breaches during these live events keep IT leaders on edge, the entire time they are online.

Really, every event should be monitored and analyzed for security implications during live events to help ease the risk. The problem is there are never enough people to handle it. That is until now…

For the first time in my 18-year career, I witnessed 243,000 events analyzed, 431 events diagnosed as malicious, and 13 scoped incidents escalated—without one false-positive through the entire event! The Respond Analyst’s Decision Automation technology performed without any disagreements from the onsite security team. Bottomline, the Respond Analyst learned its environment in less than a few hours!

To put that into perspective, a traditional security team would need more than 125 trained analysts to cover this volume in the same timeframe. The Respond Analyst covered all of this, freeing up the 6 onsite Norwich students to focus on threat hunting and more strategic security concerns.

This was a humbling experience for all of us at Respond Software who work so hard to deliver solid results for our customers. For those who know my style, I’m not one to brag or fluff the numbers just to give marketing a win, and this time, even I had to sit back and feel good about our capability and results.

One of the best parts was how fast everything was up and running. We built a server loaded with the Respond Analyst and sent it into the Norwich University students to set up. Everything was handled remotely by the Respond Software team and the Respond Analyst was operational within 6-7 hours from start to finish. Immediately after the first escalation, the students realized the event was a real threat (not junk data)—right away it generated more interest with the students and security personnel.

This is why I love helping customers. Knowing that the Norwich team could go to bed knowing that their networks were safe. They could identify and address cyber threats quickly—covering more with a smaller security team, meant they could focus on higher priority tasks. I look forward to working on more events like this in the future.

For more information, check out the Respond Software and Norwich University Press Releases.

3 Trends That Make Automation a Must for Securing Industrial Control Systems

Every time I flip a light switch or run water for my daily shower, I’m not thinking of the potential security risks within our power plants or water suppliers. I just take it for granted that the computers working behind the scenes keep things running smoothly.

These computers, also known as Industrial Control Systems (ICS), control the physical world of our most critical infrastructure. They monitor and control the processes responsible for machinery used in power generation and distribution, manufacturing, water treatment plants, HVAC, and many other industries.

The reality is that some of these systems were not designed with security in mind. Historically, these systems were not connected to the Internet or an IT network. They existed in an air-gapped environments, disconnected from all other networks.

The disconnected nature of ICS is quickly becoming outdated. Systems are more connected than ever before and can be accessed remotely by operators. Three trends are increasing the vulnerability of our ICS environments.

Trend 1: Connected IT and Operating Technology (OT) environments are growing.
While these blended environments provide increased efficiency and reduced costs for operators, they also increase the potential for security threats. Threats that occur in OT environments generally originate in the IT environment and then traverse the boundary.

This is complicated by the fact that Industrial Control Systems were not built with security event logging in mind, they receive software updates infrequently, and they often exist within flat networks (where all systems exist in the same network).

Bottom line—if one system is infected, it’s easy to spread the infection to multiple systems.

IT has traditionally focused on securing the confidentiality and integrity of data or services while ICS security has focused on maintaining operational availability and ensuring safety. Given the changing nature of the environments, these responsibilities need to evolve.

Trend 2: Attacks are becoming more sophisticated in critical environments.
There have been numerous examples of nation-states disrupting Industrial Control Systems with cyber attacks. One particularly well-documented example (and worth the read from Wired!) is Russia’s repeated disruption of the Ukrainian power grid. Other examples include:

Trend 3: A shortage of trained security analysts.
There is already a limited population of security analysts, but there is an even smaller population who can triage the combination of cyber and operational threats.

IT security analysts cannot monitor an OT network without understanding how the ICS systems function normally and how they can be exploited. Also, ICS systems often communicate on proprietary network protocols not found in IT environments and therefore, require specialized detection technologies to alert an ICS related threat.

Stop these 3 trends from impacting your ICS environment

The increasing potential for threats, combined with the lack of specialized resources to detect these threats, leave us all vulnerable. The serious attacks on power and water supplies around the world demonstrate the urgency of staying ahead of the bad guys.

Help is on the way. Using Artificial Intelligence (AI) and Machine Learning, Respond Software has partnered with SecurityMatters (recently acquired by Forescout) to provide automated monitoring, decision making, and triage of network intrusions within ICS environments.

Respond Analyst provides 24×7 automated monitoring and triage, without requiring you to hire, train, and operate a team of security analysts. SecurityMatters provides in-depth visibility into ICS environments, classifying assets and detecting threats based on deep packet inspection of industrial protocols. By monitoring both your OT and IT environments, Respond Analyst is able to identify threats crossing that boundary, providing an earlier warning, and increased visibility into the earlier stages of the attack.

Security Matters and Respond Software partnership.

A new tool for defenders – Real-time analysis of Web Proxy data

When I got back into the office after taking a short break to recharge my batteries, I was really excited to be speaking with my colleagues at Respond Software about the upcoming release of our web filtering model for our Respond analyst. You see, over the last few months we’ve been working tirelessly to build a way to analyze web filtering event data in real-time. Now that I’m sitting down to write this blog, the fruit of all the hard work our team has put into making this a reality is really sinking in. We’ve done it! It’ s now available as part of the Respond Analyst!

This was no small feat, as most of you in the security operations world would know.

You may ask why we chose to take this challenge on.  The answer is quite simple, there is a ton of valuable information in web filtering data and it’s extremely difficult for security teams to analyze these events in real-time due to the sheer volume of data generated by enterprises. What a perfect opportunity for us to show off the Respond Analyst’s intelligence and capability.

Up until now, security operations and IR teams have pivoted to using web filtering data for investigations once they’ve already been alerted to an attack through threat hunting or some other form of detection.  Processing all of the web filtering data for an organization in a SIEM or similar has just been way too expensive to do. In fact, most organizations can’t even afford to store this data for a “reasonable” amount of time for investigators to dig through.

Think about it for a second, each web page visited can generate a number of new web requests to pull back content from different sources. Then picture each employee using the internet for most of day; navigating the web through their day-to-day tasks, a few personal items between meetings, all this amounts to hundreds of web page visits each day. If you have a few hundred employees, the volume of data generated by the web filtering solution quickly becomes unmanageable. Well now we’re able to process all of these events in real-time.

Consider the questions you are able to ask of the data without even taking the assigned web filtering category into account…

  • Analyze each component of the HTTP header
  • Perform user agent analysis
  • Take a look at how suspicious the requested domain is
  • Perform URL string comparisons to all other requests over an extended period of time
  • Compare each attribute to information you’ve gathered in your threat intel database

But why stop there…

  • What about looking at whether the pattern of behavior across a set of requests is indicative of exploit kit delivery?
  • Maybe you suspect that these requests are related to command-and-control activity
  • What about the upload of documents to a filesharing service, is that data exfiltration or simply everyday user activity?

Web filtering data can also leverage the power of integrated reasoning.  When web filtering data is combined with IDS/IPS sensors, Anti-malware technology and contextual sources like vulnerability data and critical asset lists, you are able to form an objective view of your enterprise’s threat landscape.  Beyond the analysis of each of these data sources, the Respond Analyst accurately scopes all events related to the same security incident together for a comprehensive incident overview.  The Respond Analyst then assigns an appropriate priority to that incident and documents all the details of the situation and presents this information to you.  This is, by far, the most efficient way to reduce attacker dwell time.

We have a long way to go and many more exciting Respond Analyst skills & capabilities on the way. I couldn’t be prouder of all the work we’ve achieved and the release of our Web Filtering model.

Way to go Respond team!

Respond Software Named Top 25 CyberSecurity Innovators

As the new Product Marketing Manager at Respond Software, I knew when joining the team they were doing some outstanding work. Simplifying the complexity of network security monitoring and triage and giving hope to small security teams working to defend their business.

The hard work and dedication from the team has been paying off!

We are proud to announce Respond Software has been selected as one of the Top 25 CyberSecurity innovators by Accenture Innovation Awards! The 25 leading innovations consist of a diverse batch of cutting-edge concepts, developed by pioneers in our eight global themes. These innovations are reshaping our world and unlocking new value and benefits for all parties.

I tip my hat to the amazing product and engineering teams that have developed Respond Analyst to tackle some of the complexity in security operations.

Thank you, Accenture Innovation Awards for recognizing Respond Software as a top CyberSecurity innovator! We are excited to be a part of such an amazing and forward-thinking group!

When Currency is Time, Spend it Threat Hunting

“Time is what we want most, but what we use worst.”
– William Penn

How many valuable cybersecurity tasks have you put aside due to the pressures of time? Time is currency and we spend it every moment we’re protecting our enterprises.

When we are constantly tuning, supporting and maintaining our security controls or chasing down an alert from an MSSP, only to discover it’s yet another false positive, we spend precious currency. When we create new correlation logic in our SIEM or decide which signatures to tune down to lower the volume of events to make it more manageable for our security team, we spend precious currency. When we analyze events from a SIEM to determine if they’re malicious and actionable or if a SIEM rule needs additional refinement, we spend precious currency. When we hire and train new analysts to cover churn, then watch them leave for a new opportunity – we waste currency and the investment hurts.

You can spend your “currency” doing pretty much anything, which is a blessing and a curse. We can (and do) waste an inordinate amount of time going down rabbit holes chasing false positives. We are forced to make choices: do we push back a request while we investigate the MSSP escalations or do we delay an investigation to provide the service agility the enterprise requires?

Both options are important, and both need addressing; forcing us to make a choice. In our gut we think the escalation is another false positive, but as cybersecurity professionals; we wait for the sword of Damocles to fall. It’s only a matter of time before one of these escalations is related to the thing we worry about most in our environments. Either way, something gets delayed…. hopefully just lunch.

Basing decisions on what we can neglect is reactive and unsustainable. It’s a matter of time until we choose to postpone the wrong thing.

We need to use our time more wisely.

Organizations need to spend precious “currency” focusing on higher value tasks, like threat hunting, that motivate their talent and provide value to the organization. But also need to maintain two hands on the wheel of lower value tasks that still need attention.

Organizations should implement automation tools to focus on the lower-value, repetitive tasks such as high-volume network security monitoring. Generating and receiving alerts from your security controls is easy, making sense and determining if they’re malicious and actionable is a different story. The decision to escalate events is typically inconsistent and heavily relies on the analyst making the decision. Factor in the amount of time required to gather supporting evidence and then make a decision, while doing this an additional 75 times an hour. As a defender, you don’t have enough “currency of time” to make consistent, highly-accurate decisions. Security analysts tasked with monitoring high-noise, low-signal event feeds is a misallocation of time that only leads to a lack of job satisfaction and burnout.

There is another way.

Employing Respond Analyst is like adding a virtual team of expert, superhuman analysts and will allow your team to, bring their talent and expertise to threat hunting. Adding Respond Analyst allows your talent to focus on higher value tasks and more engaging work so you can combat analyst burnout, training drains, and churn.

Autonomous network security monitoring; analyzing the ‘breadcrumbs’ that are hiding in your Palo Alto Networks IPS/IDS logs

Detecting an intruder at the point of entry can have the greatest impact on reducing system compromise. That is why Network Intrusion Detection and Prevention Systems (IDS and IPS), such as Palo Alto Networks, are essential tools for any security organization, whether they are protecting the data of a large financial services company or the sensitive research and valuable intellectual property of a university.

The downside is that these systems generate such a high-volume of data that even large, mature security teams do not have the capacity to analyze all the data collected.

A common strategy for dealing with this volume of data is to apply rules to filter data, such as, ‘only show me an IPS detection alert if it appears to be high priority’. The result is that a great deal of relevant security data is disregarded, limiting visibility into relevant clues and context which would save time in identifying and responding to actual threats.

Imagine being able to increase the visibility and depth of analysis by leveraging every IPS detection alert from your Palo Alto Networks’ devices.

An autonomous network security monitoring tool should seamlessly integrate into existing PAN IPS/IDS implementations and immediately begins providing value by taking over the monitoring, analysis and decision-making required to turn IDS/IPS data into vetted, actionable security incidents that are ready for human analyst response.

By the way, out-of-the-box, Respond Analyst’s accuracy rate is between 83-92%, after a month on the job, accuracy rates push higher – with many customers reporting nearly 100% accuracy after 60-90 days.

As Security Analysts, Instead of Threat Hunting We’ve Become Ticket Monkeys

We’ve heard repeatedly from security analysts (like those interviewed in Cyentia’s Voice of the Analyst Survey) that event monitoring is time-consuming, boring, and repetitive, that security analysts feel like ticket monkeys interfacing with IT, and only occasionally do they get to do the fun work of threat hunting.

But did you know that EPPs (Endpoint Protection Platforms, commonly called Next-Gen Antivirus, NGAV or AV) are a foundational data source in security operations but can also be a time sink for security analysts to evaluate and act.

Generally, EPPs generate high-fidelity alerts; the system is likely infected with malware. Given this alert, a security analyst must decide if:

1. the infected system presents a serious threat to the organization and an incident response procedure is

required

2. the system is in fact infected but the threat is not that serious and can be safely mitigated by creating a

ticket for IT or simply reimaging the machine

3. the alert can be dismissed because it is not a threat and no action is required at this time

And how does a skilled security analyst come to an accurate and appropriate decision?

Context. Context. Context.

A security analyst must understand the importance of the involved systems and accounts. Is this a server or a workstation? Is this the CEO’s laptop? Do the systems have any vulnerabilities?

Security Expertise.

Not all malware is created equally.  A security analyst must understand the type of malware, its function, potential harm, and ability to spread.  Analysts gain expertise on the job, through research, or arduous certifications (of which they need to keep maintained).

Experience.

Good security analysts won’t assume that the action taken by the endpoint agent (aka EPP) will fully remediate the issue, they will look for other indicators and evidence.   For example, corroborating and relevant network IPS alerts.  Experienced analysts know that when one malware is observed, likely more are lurking.

Awareness.

Of course, the security analyst must qualify if this threat is even relevant to their environment. Conversely, the threat could be part of something ongoing within their organization or an external campaign.

A thorough analysis of the situation and making the appropriate decision takes time.

On top of that, interfacing with IT and generating tickets to remove commodity malware from a workstation may not be meeting the expectations of hungry analysts eager to be hunting for bad guys.

It’s no surprise SOC teams are falling behind their unrelenting event loads and 1 in 4 security analysts express dissatisfaction with the current job.

But wait…

There is a solution besides wringing hands or hiring more analysts. Turns out, we created a Virtual Security Analyst to expertly analyze malware events and recommend a course of action. And get this, our virtual security analyst is fast, scalable, and 100% (yes, that’s right) 100% consistent in performing dozens of checks while evaluating every event.  On top of that, Respond Analyst integrates with most ticketing and case management solutions, elevating your analysts from time-consuming ticket creation processes.

Don’t you just want to learn more why we were named one of Gartner’s Cool Vendors?

Please reach out to learn how to augment your team with the Respond Analyst today.

SOC Events Per Analyst Hour aka (EPAH)

Early in my corporate career, I was promoted to the first SOC manager for IBM’s Managed Security Services. Prior to building that SOC, our security monitoring services were provided by the NOC.

This was back in 2000, the early days of security operations, before we’d figured it all out.
This SOC team was tasked with monitoring security events in real-time to determine which were important enough to be escalated to our customers for deeper investigation.

Within weeks, the team pointed out to me that they were completely overwhelmed by the volume of events.  In fact, when I pushed back on the team they said, “You try it!”  So, I did, and it turned out that 2,400 events per hour will make you crazy in less than 20 minutes (and they knew that).  This led me to the question, “What is too much volume for a SOC analyst and how do you measure it?”  This is when I began measuring Events Per Analyst Hour (EPAH).

Now that I have been measuring humans analyzing events for almost 20 years, the “green zone” for EPAH is between 75 and 150.   And, those events can usually be collapsed into 12-15 different simultaneous potential attacks to analyze.  An experienced analyst can usually handle around 150 events and a junior analyst will struggle with 75 events.

To demonstrate the challenge this limitation poses for Security Operations monitoring, let’s look at the hard numbers to bring this into focus:

  1. A mature SOC provides 24 x 7 x 365 coverage
  2. It takes a minimum of 10 FTE (really 12 with management) to cover that at 1 analyst per hour
  3. IDS/IPS alone can produce 100+M events in any given month
  4. Many other data sources are just as loud, so assume 300-500M events per month for many enterprise SOCs
  5. At 500M events per month, that equals 694,444 events per analyst hour

Clearly that scenario doesn’t work.  So, if you implement an event funnel to reduce the above by 4,629x down to 150 EPAH, what did you miss?  I’m not sure, how about you?

When the capacity of humans doing security monitoring is exceeded, it’s obvious.  On the other hand, what was ignored and missed, that should have been spotted?  Here’s the catch — you won’t know until it’s too late.  While EPAH is a critical measure of what human analysts are capable of, it doesn’t tell you anything about what you ignored.  This is the fundamental limitation of console monitoring.

Even if humans were perfectly effective at monitoring (see my previous article “How Human Factors Hurt the SOC”) catching capable attackers is nearly impossible due to the small amount of total possible attack traffic actually observed.  This is borne out by the consistent truth that most attacks are caught when new stolen items show up in the darknet.

What If Your Frontline Cybersecurity Ops Team Had Malware Detection Robots to Help?

What if I told you that you could give your front-line security analyst a robot that could automatically tell you which cyber-incidents were spreading, which systems were in question, how dangerous the malware was, how it was detected, and numerous other factors that you would want to know? How would that change your world?

Now what if I told you this robot could emulate the human reasoning and judgement which your expert security analysts use but with inhuman speed, scale, and consistency? In other words, could do what no human is capable of doing. Here’s what Raffael Marty, a world-renowned security expert and former executive at Sophos said:

“…understanding which [malware infections] do and what needs to be done about them is a very time-consuming process for today’s security operations teams…”

Well, there isn’t actually a robot to do this yet, but there is software that does. In fact, it’s the first-ever software system that automates your front-line security analysts monitoring and analysis tasks to determine incident severity based on endpoint protection telemetry. Perhaps since the software is here, the robots will follow!

To Accelerate SOC Speed It’s Time to Re-assigning Cybersecurity Analysts’ Repetitive Tasks to Machines

Ask any security analyst why it is so difficult for them to perform consistent, accurate and fast incident scoping and prioritization, and they will give you a very long list of answers.  It’s no secret that for human analysts, regardless of their skill level or experience, these mission-critical security tasks can be overwhelming due to the ever-increasing attack surface, avalanche of data to process, and not enough skilled personnel to do the job.

For those who are not familiar with the difficult job security analysts perform, the following overview will help to demystify what incident scoping, prioritization and escalation mean, along with what it takes for human analysts to do the job.

The Challenges of Scoping

Scoping is the process that aggregates all related events and systems into a common incident, or situation, based on shared attributes within an attack, such as the same systems, attack stages, or signatures.

Incidents can vary dramatically in both their duration and number of internal assets involved.  For example, targeted low and slow attacks can last weeks or longer, involving only a few systems.  Other incidents may contain many events and systems over a short period.

Now, consider that an analyst’s shift is typically eight hours long. Within that shift, a pair of analysts may trade-off between spending two hours monitoring the event console and two hours off console working on supplementary duties. This type of duty rotation is essential to try and prevent “console blindness” which can easily happen from monitoring events and from staring at a screen for too long.

To scope across these shift changes, analysts must rely on the shift log documentation from previous analysts.  In the shift log, the analyst annotates events they find interesting or decide to take action on. The strength of incident scoping relies on the accuracy and exhaustiveness of this documentation.  Moreover, given that humans are naturally prone to make mistakes (and documentation is as boring as it gets), shift logs are typically incomplete and inconsistent.

As a result, analysts scope “in time,” and primarily based on their memory.  In best case scenarios, they can tie disparate alerts together into a single incident while on shift. This means no more than a one to two hour (perhaps less) lookback scoping events together.  Here’s where the process breaks down since relying humans for scoping is faulty,  inaccurate and inefficient.

The Challenges of Prioritization

Prioritization evaluates the context, impact, and systems scoped within the incident and assigns a priority level.  Typically, priority levels dictate how quickly a response is required and who needs to be involved or made aware of the threat.

With each escalation decision, security analysts must run through a checklist of a few critical questions, for example:

  • How big is this event?
  • Is this a single compromised workstation (or are we announcing a major breach on tomorrow morning’s news)?
  • Are we handling the incident during the Monday 8 to 5 pm shift or executing a full-blown incident response process at 2 AM on a Saturday night?
  • Am I waking up the CISO and the CEO in the middle of the night?

 

The implications of assigning a high priority to an incident and being wrong can result in wasted time, along with some angry colleagues and management. As a result, analysts have become biased and overly careful, not wanting to escalate too quickly for fear of angering the receiver of the escalation.

 

However, it is apparent that this type of behavior introduces an enormous risk to the organization. Security analysts should not be afraid to use the proper callout procedures designed to protect the organization and enable rapid response.

At the same time, it is a catch 22 for analysts since they are under unrelenting pressure to identify something – anything — that could be a threat to the organization’s security.  Consequently, analysts often chase down lower priority, but easier incidents that should be ignored vs. focusing on real (and potentially more challenging to investigate) attacks.

The scoping and prioritization challenges identified above introduce avoidable risk into the organization.  It is time to consider automation to relieve human analysts and reassign these repetitive tasks to the power of software and machines.

Relieving Human Analysts

Our expert system, the Respond Analyst, is software which emulates human decision making and reasoning and is consistent in performing every check, every time, is immune to “console blindness”, and possesses the memory to scope over large durations.

The Respond Analyst can immediately accelerate your SOC and solve your scoping and prioritization challenges by:

  • Dynamically grouping related events and systems into actionable security incidents
  • Building actionable, detailed cases with decision-making transparency
  • Prioritizing incidents based on the incident likelihood, number and business importance of systems involved, and the observed attack stage
  • Reprioritizing incidents as attacks progress or new information is introduced
  • Integrating into existing security operations workflows and case management processes

For more product information on how the Respond Analyst can help you, to schedule a demo, or to view our new “Deploy a New Kind of Analyst” webinar, please contact us at info@respond-software.com or go to our Contact page.

Join our growing community! Subscribe to our newsletter, the "First Responder Notebook," delivered straight to your inbox.