Threat hunting is a regularly-occurring activity in any high-performance SOC. But for less savvy organizations, it’s a must-have activity that can mean the difference between a malicious hack or a normal, uneventful day. With the stakes so high, it’s time to look at the history of threat hunting, what it looks like today, and the future of threat hunting – particularly as adversaries become more advanced every day.

The ghost of threat hunting past

Just over a decade ago, security analysts shared information on malicious code amongst themselves only – no public announcement of a breach, and certainly no public reckoning. Instead, you would receive information about a threat from a colleague or friend, conduct an investigation to determine how the compromise occurred, and then enrich your security postures to prevent future similar compromises. This system inspired a new industry paradigm in how SecOps evolved to fight adversaries, at respective organizations as well as across the globe. A handful of vendors began to offer threat intelligence, and eventually, a whole slew of security providers developed threat intelligence for the industry as well as for different verticals.

Once these tools became widespread, SecOps needed to subscribe to threat intelligence to enrich detection tools, which is a failsafe approach to looking for future threats. But threats seldom appear in real time, so you need a robust historical log to search for past threats as well as potential future ones.

Today, many organizations rely on SIEM tools, which offer a single pane of glass view into security events and gives you a method of broadly loading events against it – a far cry from individually loading events, which can take months.

How threat hunting’s humble beginnings can hinder its growth

Make no mistake: threat hunting has come a long way in the last ten years. But then again, so have adversaries.

There are a few barriers to entry for achieving what today’s ideal SOC looks like:

Growth of the attack surface. Business expansion has created an almost incomprehensible number of endpoints that serve as attack vectors for breaches. This means that while SIEM became the norm for historical security data, attackers began to use all available data, not just security data. SIEMs can no longer handle the volume of structured and unstructured data available to the security operations center, and the landscape is changing. In other words, all data is security data. SIEMs are an important piece of the puzzle, but you need more.

Live data is more necessary than ever. Without live, hot data available to search, you need to search months of data in a data lake; this process can take hours, and it requires an exact search and syntax criteria that’s unlikely to happen without a data scientist present at all times. This data lake approach is much too time-consuming when the potential for data breaches can take mere seconds.

TCO was prohibitive, until now. The cost for centralized log management solutions to log all data can be astronomical based on the high levels of machine data digital businesses are generating. To keep costs in check, businesses generally keep one or two weeks of historical data in their logs, and back the rest up to tape or another backup storage type. The problem? You can’t threat hunt tape. However, security analytics tools today deliver full capabilities at a much lower cost, so you don’t have to let cost constraints compromise your security.

Today’s high-performing SOC

These days, the specter of attackers keeps CISOs up at night. To get a good night’s sleep, you should have access to all data, live, and you shouldn’t have to restore backups or manually insert other data sources that weren’t being logged. You need full, complete visibility without restriction. What’s more, you should be conducting consistent penetration testing of your systems to ensure threat hunting systems are functioning properly.

The future of the SOC and threat hunting

Unfettered visibility is important, yes. But you must also hone skills to combat the next phase of how our adversaries are targeting us: through artificial intelligence and machine learning. Attackers are beginning to dynamically shift on the fly; they no longer require an employee to click a phishing email to gain access to data. Next generation attacks can execute from previews, shut off antivirus systems, escalate privileges, and even disable logs to hinder threat intelligence.

What should I do?

I’m glad you asked. Continue to enrich your organization’s security posture with full data visibility and robust threat hunting and intelligence. Understand that malicious adversaries are hiding in plain sight – such as in Windows Powershell designed for sysadmins – so threat hunting must evolve to look for tactics, techniques, and procedures (TTP), instead of focusing purely on static indicators of compromise. But, don’t leave old rules behind, either: static information like hashes are still a relevant part of security.

Jason Mical is Devo CyberSecurity Evangelist.