On Wednesday, March 28th, SP6 presented a webinar titled “How to Optimize Splunk License Usage.” During the webinar, I presented several options for understanding your organization’s Splunk license usage as well as ways to tune, reduce, and optimize the license usage within your environment.
I’ve gone through this exercise with many clients and have found that it’s a good opportunity for them to discover what’s in their environment. Sometimes customers are excited to find data they didn’t even know they had! As a Splunk admin, it’s essential to understand what those data sources are because it determines what you can do with Splunk. It also helps ensure that you’re not ingesting unnecessary data that will cost your business money in the long run.
The Path to Splunk License Optimization
To get the most out of your existing Splunk license, you must understand how licensing works, and also understand your current license usage. Splunk provides an out-of-the-box report to help you understand your usage. This can be further refined by source, sourcetype, host, or index. You can see a sample of the license usage report below. During the webinar, I also provided some additional suggestions on ways to filter this report to get the most out of it.
We also asked the question, “How did it get like this?” and reviewed some common decisions that can lead to overconsumption of data. Finally, we examined ways to reduce Splunk license usage by eliminating unnecessary data, focusing on key use cases, and tuning key data sources.
Key Questions on Splunk License Optimization
After the webinar, we set aside time for a question-and-answer session. Below is a review of the key questions and answers.
Q: What happens if I want to ingest a year of historical logs and it’s over my license limit for the day?
This does come up quite a bit with customers. They’ll want to onboard a data source and they’re worried that when they bring in the data, they’ll blow out their license. The answer is that Splunk gives you 5 license violations per month for a reason, and this is one of those reasons. Splunk understands that there will be days when you want to onboard a historical data source. So of course, that day you’ll be at a terabyte because you had to pull in all of the backfill data. But then you won’t go over for the rest of the month. This is a very typical scenario. If there is a problem, it’s usually as simple as reaching out to your Splunk Sales Rep for extra assistance.
When you have professional services personnel onsite with you, we can make the whole process easier by onboarding this data for you and ensuring that there’s no-hit to your license as you go through this process.
Q: How does data get re-indexed by Splunk?
The most common situation for re-indexing is when file inputs are misconfigured. Data will be re-indexed if for any reason Splunk doesn’t know that it has re-read that data. If your file input is misconfigured or if you’ve changed the cyclical redundancy check in any way, then Splunk may re-read that data because it affects how Splunk knew it had read that data before.
Another challenge is that the way Splunk tracks which logs it’s read and which logs it hasn’t read is with an index called the Fish Bucket. This is included in every Splunk install. Every time something gets read, it gets written to the Fish Bucket as a hash. If you were to completely delete a forwarder, for example, if you had a forwarder on a server and you completely deleted it and installed a new one, the Fish Bucket would be deleted too. Splunk would end up re-indexing the events you told it to read because it has no idea that it read them before. If you’re upgrading a forwarder, don’t delete that forwarder and reinstall it. It’s better to update that forwarder.
The bottom line is to watch your file inputs and be mindful of your upgrades and installs for Splunk forwarders.
Q: Where do I apply event filters to send data to the null queue?
Splunk has numerous options for routing and filtering data. And really, you can do that with a regex definition. You can route and filter any pattern you want to. This routing and filtering normally takes place on a heavy forwarder, and as a best practice, that’s really the place for it, because there’s a bit of CPU usage associated with this activity.
If that’s not possible it should be done at the point where the data is indexed. So that would be on your indexers. They can also handle this route and filtering operation.
Additional Resources on Splunk License Optimization
There are many additional resources that may be helpful to you and your organization as you’re exploring Splunk License Optimization:
- Choosing the Right Splunk License
- Reducing Your Splunk License Usage
- Splunk Docs on How Licensing Works
- Splunk Docs on Routing and Filtering Data
- Windows Security Event Code Encyclopedia
Do you need assistance with Splunk license optimization or with your Splunk deployment? SP6’s Splunk Professional Services consultants manage successful Splunk deployments, environment upgrades and scaling, create dashboards, searches, and reports, and perform Splunk Health Checks. SP6 also has a team of accomplished Splunk Developers that focus on building Splunk apps and technical add-ons.
