One of the best ways to ensure that you’re getting the most out of any tool is to get a handle on others’ mistakes and lessons learned. Our team of Splunk Professional Services engineers is continuously gaining insight into how to best deploy Splunk, including how to overcome technical challenges as well as internal hurdles. We’re sharing these insights to help your organization gain faster value from your Splunk deployment.
This post is the second in a series of articles on “Things I Wish I’d Known” about Splunk. In today’s post, Roman Lopez, Splunk Professional Services Consultant with SP6, and Jon Papp, SP6’s Professional Services Manager, explore lessons learned that can improve Splunk deployment and adoption. For more Splunk best practices and lessons learned, you can also view the first post in this series, “Splunk Data Management – Things I Wish I’d Known.”
Have a Good Plan to Drive User Adoption
Once Splunk has caught on with a few key users, its use, functionality, and value-add start to take off rapidly. But how do you get those key users to adopt Splunk?
Many organizations will plan internal demos or provide official Splunk training to employees in hopes of encouraging them to take advantage of Splunk. This is a great starting point – but it doesn’t go far enough. When bringing Splunk into a new environment, you need a plan that connects the specific pains of your users with the solutions Splunk can provide.
Start by gathering requirements – what kinds of problems does a team have that can be solved by Splunk? Then solve some of those specific problems – build a dashboard to show user account lockouts, or build an email alert for unexpected server shutdowns.
Now that users see Splunk can solve their problems, they’ll be more open to attending something like an internal hack-a-thon – book a conference room for 4 hours and work with interested users in solving their problems – by writing their own Splunk queries – in Splunk! Plans for user adoption will vary by organization, but if you want to drive increased value with Splunk, you need your users to take full advantage of it – have a plan!
Build a Test Environment
The reality is that clients (even big corporate clients) ignore the test or dev environment requirement. If you have a multi-site, multi-cluster environment in production you should have something similar in the test. I have seen this at a client where it was implemented after an ES upgrade went very badly south. A test/dev environment is critical for testing upgrades, replicating and fixing errors in your prod Search Head cluster between sites, etc.
The majority of clients do not have a test environment but it is so easy to set up, even on a local laptop using VirtuBox or VMware.
Utilize Splunk Docs (Correctly!)
Use Splunk docs, and when doing so, remember to change the version in the top right corner. On my last engagement with a Medical equipment provider, they were running 6.5.0 and we were running some commands on the SHC. These differed significantly between 6.5.0 and 6.6.2
- https://docs.splunk.com/Documentation/Splunk/6.5.0/DistSearch/RestartSHC
- https://docs.splunk.com/Documentation/Splunk/6.6.2/DistSearch/RestartSHC
Once you select a topic, the next page comes up:
The Splunk wiki and blog is another excellent resource. Topics include:
- Troubleshooting Your Splunk Installation
- Deploying Splunk
- Getting Data Into Splunk
- Searching, Alerting, and Reporting
- Much More!
Measure Twice, Cut Once
This is something that comes naturally after your first handful of deployments once you’ve been bitten by inefficiencies that you unknowingly built into your platform!
Every time you develop a new component you need to ask yourself: How will this scale? How likely fragile is it (think dependencies and assumptions) and how gracefully will this fail? How easy would it be for another person to understand and fix? Can I say this is “Enterprise” level? What will the maintenance overhead be?
SP6 – Expertise for a Successful Splunk Deployment
SP6 is a Splunk consulting firm focused on Splunk professional services including Splunk deployment, ongoing Splunk administration, and Splunk development. SP6 has a separate division that also offers Splunk recruitment and the placement of Splunk professionals into direct-hire (FTE) roles for those companies that may require assistance with acquiring their own full-time staff, given the challenge that currently exists in the market today.
