Person working on a laptop

Understanding Splunk Custom App Development

Many customers believe that the Splunk apps they download from Splunkbase are either created by Splunk itself or provided by product vendor companies directly. This is not entirely true. Splunk has empowered developers to write their own custom apps and make them available to the world by uploading them onto Splunkbase.

However, developing a Splunk app and uploading it on Splunkbase or distributing it through external repositories (Git, Bitbucket, etc.) is not enough. Only Splunk Certified Apps can be installed on Splunk Cloud; and as Splunk Cloud gains wider adoption, uploading a custom app on Splunkbase and not getting the “Splunk AppInspect” badge is not a wise option.

Today, I am going to share some tips around custom app development which will help in adhering to Splunk guidelines and best practices for Splunk apps. Before getting into that, I would first like to shed some light on the nuances of Splunk Apps and Add-ons.

Terminology and Definitions Related to Splunk Custom App Development

In practice, customers/users use different terminologies including Splunk Connector, Splunk Integration, Splunk Module, and Splunk Plugin. All of these words can be classified in one of two categories in the Splunk world: 1. Splunk Technology Add-on (often referred to as a “TA”) or 2. Splunk (Main) App. You will also see Supporting Add-on (SA) and Domain Add-on (DA) available with premium apps like Splunk ES and ITSI but these are twists on TA’s and we’ll leave them outside of the scope of this discussion due to their rarity.

So, what is the difference between the two? Well, there are significant differences but they both complement each other to provide a complete solution for their specific use case.

Splunk Technology Add-on – In practice, a Splunk Technology Add-on is a Splunk app that works primarily on the back end. It consists of data ingestion logic, data parsing, field normalization, and CIM mapping configuration. There are some advanced scenarios where Technology Add-ons contain custom commands, alert actions, or other things as well.

Splunk (Main) App – The Splunk App primarily delivers the front-end solution to the Splunk users. It consists of dashboards, reports, alerts, scheduled searches, and lookups (lookups can be part of TA also depending on the use case).

Tips for Successful Splunk Custom App Development

Now, below are some of the tips which are basically lessons I have learned while developing various Splunk Apps and Add-ons over the past few years and also based on the feedback received by customers who use dozens of apps on a single Splunk instance

1. Avoid using macros in eventtypes. It’s standard practice to define the custom index that needs to be searched in a macro (for ex: [get_my_index] index = my_custom_index and use `get_my_index` macro in all search queries for better index management purpose). First of all, it needs to be noted that Splunk doesn’t replicate the macros to indexer clusters in distributed environments. You won’t face any problems with this approach on dashboard queries because the Splunk search head replaces the macro declaration with the macro definition before sending a search query to the indexers.

However, the search head doesn’t behave in the same manner when it comes to eventtypes and sends search queries of eventtypes as is to the indexer layer (with the unexpanded version of the macro). This behavior will break your eventtype definition – so it’s better to avoid using macros in eventtypes.conf. There is a workaround for the curious – you can make macros work in eventtypes.conf by adding macros.conf to the “replicationWhiteList” stanza in distsearch.conf. However, when you’re developing a convenient and simple-to-install product, you don’t want to push this burden onto your users or their Splunk admins.

2. Make knowledge objects of Add-ons available globally so that App dashboards/reports/alerts can leverage them. Most of the time, a Splunk Add-on will not be visible unless it has a custom setup page for input configuration. So not exporting a knowledge object globally would make it practically inaccessible for the users at the web UI level.

3. Export knowledge objects of an App only at that App level to avoid any conflict with other Apps. Generally, dashboards, reports, and alerts need to be accessed by going to the App Navigation menu only – so sharing them globally will typically not add much value. Additionally, it can cause conflicts in cases where dashboards or reports from two different apps have the same name, For ex: two different dashboards with the same name “Malware Attacks Summary” created from the data collected through two different anti-virus products in one organization. This can lead to unexpected behavior on the Splunk Web UI.

4. Always try to normalize events and make them CIM compatible even if there is no immediate requirement for that. Many times, Splunk developers ignore the importance of CIM compliance and just focus on data ingestion and data parsing when developing Add-ons. However, at later stages when this Add-on reaches the masses, many Splunk Users/Admins expect that the Add-on will already be compliant with the Splunk CIM so that they can see the results with CIM queries they have already designed. This situation can often lead to another almost full development cycle just to satisfy CIM compliance.

5. Stick with search time extraction over index-time extraction wherever possible unless it is absolutely required (for example, masking PII information which has to be done at index time). It’s standard practice to not put any additional burden on Indexers other than indexing and data replication. For this reason, many customers keep storage optimized (and not compute-optimized) indexers in their production environments.

6. Use easily recognized prefixes in Saved Search and report names (for ex: Cisco – Network – <report name>, Cisco – Firewall – <alert name>). If possible, but the purpose of the Saved Search as a prefix in its name, for ex: Lookup Gen – <saved search name> and Alert – <Saved Search name>. In production environments when there are tons of apps installed on a search head along with user-created saved searches and reports, it is common to see hundreds of entries under Setting -> Search, Reports and Alerts. It’s the developer’s responsibility to make sure saved searches and reports have unique names with all the required details in them.

7. Use the “Splunk package” command to create the .spl bundle for your custom app. A custom app can be packaged using tar and gzip commands also, but it comes with its own risks and limitations. As you know, tar and gzip are not available on the Windows platform natively. Also, using the tar command on Mac OS creates an extra ._foo file for each foo file unless you tell Mac OS to not copy metadata files by exporting the COPYFILE_DISABLE=true environmental variable. The Splunk package app is platform-independent and always fulfills its purpose without any hiccups.

8. Use Splunk Add-on Builder to build your Add-ons, especially when 1. you are making external connections using custom python code or 2. You want to make Splunk events CIM compliant so that ingested data can be leveraged by other Splunk apps (e.g. Splunk Enterprise Security). The Splunk Add-on Builder can give a solid, consistent look to your apps. It takes care of naming conventions and many Splunk best practices, making the developer’s life much easier. You can download the Splunk Add-on Builder here and check out the user guide for it here.

About SP6

SP6 is a Splunk consulting firm focused on Splunk professional services including Splunk deployment, ongoing Splunk administration, and Splunk development. SP6 has a separate division that also offers Splunk recruitment and the placement of Splunk professionals into direct-hire (FTE) roles for those companies that may require assistance with acquiring their own full-time staff, given the challenge that currently exists in the market today.