Splunk is a log aggregation and analysis tool that can also serve as a SIEM (Security Information and Event Management) product when the Splunk Enterprise Security app (in most cases, simply referred to as Splunk ES) is installed. The Splunk Enterprise Security app provides prebuilt content, including correlation searches, to help security analysts streamline investigations within their IT environments. In this article, we will discuss the features that make Splunk Enterprise Security the high-powered SIEM tool that it is.
The first thing to know about Splunk Enterprise Security is that it runs on top of Splunk Enterprise (or Splunk Core). You need to have a mature, performant Splunk environment in place before you can reap the full benefits of Enterprise Security. Unlike some other Splunk server roles, the ES app requires its own dedicated search head. The minimum hardware requirement specifications to run ES efficiently are 16 CPUs and 32 GB. This also needs to be paired with sufficient disk I/O and storage space at the indexing layer (minimum 800 IOPs random seek). Under the hood, ES is performing data model acceleration and correlation searches that are very resource-intensive, especially on CPU. You can read more about the recommended hardware requirements here.
Getting Up and Running with Splunk Enterprise Security
Ok, so you have purchased Splunk Enterprise Security and you have the right hardware in place. You’re ready to go, right? Not so fast. The most important part of getting Splunk ES to work and getting its pre-built dashboards and other content to “light up” is ensuring that all your data is Splunk CIM (Common Information Model) compliant. The Common Information Model is Splunk’s method of data normalization. Only events that have been normalized to the CIM will be included in the data models that are being accelerated. All searches, dashboards, and reports use the data models to return results and events to users.
What work needs to go into making your data CIM compliant? Aditum’s Professional Services engineers estimate that roughly 50% of the Apps and Technology Add-ons (TAs) on Splunkbase are already CIM compliant. For the half that is not, there is a bit of legwork required by an organization’s Splunk Admin (or Aditum’s Professional Services could be utilized). It could take anywhere from 30 minutes to a full day per data source to modify the logs to be CIM compliant. Our PS staff prefaces these estimates with “if the person knows what they are doing”, and also notes that Aditum has built a library of Apps and TAs for common data sources (Palo Alto, Juniper, Checkpoint, Windows, etc.) that have been made fully CIM compliant. The majority of the time, this entails working with existing Splunk Apps and TAs and making modifications as opposed to writing custom apps. The modifications include things such as renaming fields, adding tags that are not available, fixing tags (tags translate log-speak into “plain English” for those not familiar with the logs, for instance, NOC or SOC Analysts), modifying data that may in the wrong fields, and other modifications. While CIM compliance is central to the use of Splunk Enterprise Security, it’s worth noting that this data normalization, or what we’ll call “data hygiene”, is recommended by Aditum’s Splunk Professional Services engineers outside of Splunk ES, as it ensures that all fields coming into Splunk are consistent, and as a result, all users get the same use out of the data.
Another consideration with the data you are sending to Splunk in preparation for using Splunk Enterprise Security is the different types of data sources. Splunk ES tries to build a holistic view of your security infrastructure by conducting correlation searches across different data types. To achieve the most value out of Splunk Enterprise Security, you should have a good mix of data types including web, network traffic, vulnerability, authentication, etc. For example, if you only have network traffic data coming into Splunk, most of the prebuilt dashboards in ES will not populate. The data models that they run their searches from will not be built due to a lack of data types. Splunk needs data to provide insights. If the data isn’t available, ES can’t help.
Splunk Enterprise Security and Correlation Searches
As mentioned, correlation searches are another big piece of how Splunk Enterprise Security functions. These are pre-packaged searches that (once enabled) run in the background to identify known threats, attacks, and some vulnerabilities. Each search is written to look for events in your data that may represent a possible attack. The Brute Force correlation search, for example, looks for multiple failed login attempts from the same user in a short period of time.
Splunk Enterprise Security comes with 59 more searches like this that will help identify possible attacks and vulnerabilities in your environment. As mentioned earlier, collectively these are resource-intensive searches so you should only turn on the searches that match the data in your environment. These searches can also be edited and extended to match the unique requirements of your environment.
Once you have found the correlation searches that you want to use, it’s important to set those to run on a schedule instead of in real-time, which occupies your CPUs longer. Splunk Enterprise Security also provides a framework for you to build your own correlation searches and display them in its Security Posture dashboards.
Notable Events
When a correlation search finds results, it creates a Notable Event. Those events are then grouped by event type and tags, and placed in the “notable” index. These events are used to populate the two main dashboards in ES, the Security Posture Dashboard, and the Incident Review Dashboard.
The Security Posture Dashboard lists all the information about the notable events that ES has found. This includes the amount over time, the urgency of the events, the sources they are coming from, and the events that are occurring in your environments the most. Like any other dashboard in Splunk, these panels can be modified to fit the time frames or information you would like to see.
The Incident Review Dashboard is where you investigate notable events. It lists notable events individually over a time period that you designate. Once you click on one of the notable events it gives you the correlation search which triggered the event and other information that will help you investigate that event. There is a Notable Event Action tab that allows you to assign an owner, add an adaptive response, or change the status of an event. This dashboard should be checked daily to understand the security status of your environment.
Understanding and Managing Urgent Events
On both the Security Posture Dashboard and the Incident Review Dashboard, you will see “urgency” referenced more than once. This value comes from the combination of the Event Severity which is designated in the correlation search and the Asset/Identity Priority which is designated in a list that you populate.
The Asset and Identity list is important in ES because it helps determine high-priority endpoints and users. Searches and responses centered around these endpoints or users should be treated differently than low priority objects. Most companies do not have either of these lists up to date or currently on hand. You can use a combination of tools and Splunk Enterprise Security to gather this list automatically and put together a process for maintaining it. Keeping your assets and identities up to date can be a real challenge but this provides an excellent context for ES. The first time you go through this process will be the hardest. After that, it simply turns into maintaining a spreadsheet.
When used correctly, Splunk Enterprise Security is the most powerful SIEM tool in the market, as validated by Gartner. However, like anything else, it’s important to understand the surface before you can take a deeper dive into the solution. In this article, we have covered the basics ranging from hardware specs to the importance of data normalization to assigning varying levels of urgency or priority to different assets in your environment. Once you understand these basics it will make the additional features that Splunk Enterprise Security has to offer, such as Threat Intelligence Feeds, Glass Tables, and Adaptive Response easier to understand and use.
Splunk Enterprise Security Best Practices at Your Fingertips
SP6 is a Splunk consulting firm focused on Splunk professional services including Splunk deployment, ongoing Splunk administration, and Splunk development. SP6 has a separate division that also offers Splunk recruitment and the placement of Splunk professionals into direct-hire (FTE) roles for those companies that may require assistance with acquiring their own full-time staff, given the challenge that currently exists in the market today.
