Splunk ES Implementation Checklist

Introduction to ES

Congratulations!  Your organization has had the foresight to purchase Splunk’s Enterprise Security (ES) along with expert Professional Services to assure a successful implementation.

This guide will serve as a checklist to help ensure you are prepared for the most successful ES deployment possible. We will identify the Who, What, Why, and How of a successful Enterprise Security deployment.

Granted, not every company is ready for a SIEM solution. This checklist should help you gauge how prepared you are, and what you can do to better prepare. We will cover key data sources that will provide you with the most complete picture in ES from a least mature, to the most mature scale. We will also discuss things you can do prior to your consultant arriving on site that will help expedite the implementation and make the process go as smooth as possible.

Who is ES meant for?

Enterprise Security is meant to be the tool of a Security Operations Center (SOC). Meaning ES is designed under the premise that someone will be logging in, looking at, and triaging the events on a daily basis. Some customers will try to have each notable event send off an alert email. This doesn’t scale for several reasons. The number of notable events could inundate an inbox relatively quickly due to poor tuning of correlation rules; the number of correlation rules enabled could be high; or the amount of detects could be high.

At a minimum, I think each customer could expect to see at least 50 notable events triggered per day. Some of these could be quick closes, while others could warrant an investigation, and could eventually be declared an incident. The bottom line is, you should plan to use ES as your single pane of glass, and someone should be investigating notable events on a daily basis. Detection without response is useless.

There are some really good Security Orchestration and Automation Response tools (Phantom) that integrate well with ES and help to alleviate the shortcomings of team sizes. But it is important to remember that an automation tool is not a replacement for a SOC but rather, an augmentation. Also, remember to make sure you have a process in place to track these events that your team will work on. Many people forget that a security team is more about process and procedure than anything.

What Data Sources Should I Have?

It’s important to note, this list is not the end-all-be-all, but rather a guide to help you make the most out of your SIEM solution. 36% of organizations use anywhere from 25 to 49 point solution security tools. This not only inundates security teams, but makes it next to impossible to identify and prioritize security events.

Enter Splunk’s Enterprise Security. Making this your single pane of glass and sending all of your security tools to it will exponentially increase the efficiency of your SOC team, even without the use of a SOAR tool.  ES seamlessly integrates with both UBA (User Behavior Analytics) and Phantom (SOAR tool) as well, adding exponential detection and response capabilities in a single suite.

What is this Common Information Model I keep hearing about?

OK, so you’ve identified that your organization is ready for a SIEM solution. You’ve stood up Splunk Enterprise. You’ve identified the data sources.  You’ve even started sending some data over as part of a POC. Now, the next step in the process is normalizing that data and making it to where Enterprise Security will be able to read it.  I can’t stress this enough, but a successful Enterprise Security implementation depends on a STRONG DEPLOYMENT OF YOUR CORE SPLUNK INFRASTRUCTURE. This includes the data normalization. The way Splunk normalizes data is via the Common Information Model.

The Common Information Model (CIM) is Splunk’s way to normalize logs from the many different point solutions and vendors into one common language. And we do this at search time, so it does not require any changing of your data before it is written to disk. Part of that whole “schema on the fly” that makes Splunk so good at what it does. But essentially, CIM is nothing more than data models and data sets. (i.e., malware, vulnerabilities, web, authentication, data models). In these models are data sets that are made up of very specific field values.

Many Technical Addon’s (TA) on Splunkbase will handle the CIM mapping for you. This is why we recommend using TA’s wherever available, especially Splunk built, and vendor-built TA’s. When you find a TA on Splunkbase you are interested in, take a look at the right side of the page and look to see if there is a CIM version listed. If there is, then you know the add-on will handle your CIM mapping for you. If it does not, then you will need to handle all of your CIM mappings manually. It is important to note, that some of these TA’s are not named properly, so please check this before you deploy. Enterprise Security will generally only be able to read TA’s that are named either Splunk_TA_*/TA-*/SA-*/DA-*. You can modify the app imports update in ES, but I don’t recommend this approach. I think it is better if developers make a habit of following proper naming conventions for CIM compliant TA’s.

Where’s That ES Checklist?

If you follow the below checklist, it will help ensure you are prepared to implement Enterprise Security.  Just to reiterate, standing up a SIEM is just the first part of developing Security Operations. Detection without response is useless. So, enabling use cases in ES (Correlation Searches) with no plan as to how an analyst will react will more than likely leave you with untriaged notable events. Follow the below steps to ensure ES will be a critical tool of your Operations, and not just another checkbox added to the list of security tools your team owns.

• Security Policy
  • This should absolutely drive the implementation of every use case you implement in ES.
• Compliance and Security Framework
  • If you have not already, identify any form of compliance that you are obligated to follow (i.e. HIPAA, GDPR, SOX, SOC 2)
  • Industry framework helps to ensure complete coverage from a monitoring perspective. Identify one or more frameworks you’d like your monitoring to follow. MITRE, NIST CIS Top 20, Etc.
• Data Inventory
  • Do this prior to your consultant arriving onsite. Take an inventory of your security stack, document what your posture looks like.
  • Annotate security tools in your environment as it relates to each security domain (endpoint, network, Identity, and Access, etc)
  • If you already own Splunk enterprise, compare your documented security stack to what has been onboarded into Splunk for Gap Analysis
• Assets and Identities Prework
  • Ensure you can identify critical users and machines in your environment
  • Whether it is by subnet, users title, or some other logic, having an idea on how you can identify these things will greatly help in developing your Assets and Identities framework in ES
• CIM Compliance
  • Ensure data is CIM compliant as you onboard it into Splunk
• SOPs/IR Plan
  • I know I have mentioned a few times now but detection without response is useless
  • Your analysts will need to know what to do when a notable event triggers

Use Case Enablement

The checklist above will not only help you ensure your best possible chance at a successful ES deployment, but it will also help guide you in purpose-driven use case enablement. The biggest mistake I see almost every customer make is not having a rhyme, reason, or plan to their use case enablement. More often than not, they will decide on what sounds cool instead of following some kind of plan around what to enable and how to prioritize events.

The biggest mistakes we see when customers go through use case enablement in any SIEM are:

1. Too many correlation searches enabled for the given size of the server

2. Too many correlation searches enabled for the given size of the security team (detection without response, anyone?)

3. Enabling correlation searches that there is no data available for to satisfy

• Customers often love the idea of use cases that require Sysmon or an EDR solution with no plans to collect that data
• This wastes a search slot on your server

4. Enabling correlation searches that have no policy backing the enforcement of

• E.g. Concurrent logins detected. This is a common occurrence for IT personnel unless there is a policy saying otherwise. Why detect something that isn’t prevented by the policy.

The key here is to remember that use case enablement should be purpose-driven, planned out, and actionable. Analysts should know EXACTLY what that action is for each correlation search. And the numbers of notables need to be manageable for the size of the security team. This can relate to the tuning of the rules, but also to the number of use cases that are being monitored. Generally speaking, analysts often only triage 10% of received alerts. We need to try and correct and that average and not add to it. And it starts with planning your use case enablement and sizing your team accordingly.

About SP6

SP6 is a Splunk consulting firm focused on Splunk professional services including Splunk deployment, ongoing Splunk administration, and Splunk development. SP6 has a separate division that also offers Splunk recruitment and the placement of Splunk professionals into direct-hire (FTE) roles for those companies that may require assistance with acquiring their own full-time staff, given the challenge that currently exists in the market today.