Splunk has multiple methods in regards to Getting Data In (GDI). One very popular method is the HTTP Event Collector (HEC). The use of the HEC allows data ingestion into Splunk via HTTP POST messages. Two popular methods that send POST messages out of AWS into Splunk are the AWS services: Lambda and Firehose. A common question is when to use one versus the other to get data into Splunk.
Below is a high-level overview of each service as it relates to Splunk. Lastly, a small decision table illustrates which features each AWS service can utilize.
Lambda and why do I care?
Lambda is an AWS service that allows serverless functions. The functions that Lambda can perform are up to the coding and the constraints of time and size that are given as timeout and memory, respectively. When using Lambda to send data, additional services and logic are needed for storage and error handling.
An advantage for Lambda is that the throughput for sparse or very low volume sources is higher than for Firehose.
Firehose and why do I care?
Kinesis Firehose allows data to be streamed natively or altered to configurable endpoints. The altered data format is performed by a LAMBDA function which passes the data back to the Firehose. The Firehose has Storage and built error handling as part of its base service.
One additional feature that can help to meet requirements is the option to backup all data or just errors to an AWS S3 bucket.
When should I choose one or the other?
Decision table below:
| Feature/Function | AWS Lambda | AWS FireHose |
| Splunk HEC EndPoint | x | x |
| Transform Events | x | x |
| Auto Error Handling | x | |
| Navtive AWS Bifurcation to S3 | x | |
| Ease of Administration | x | |
| Large Data Set Efficiency | x | x |
| Small Data Set Efficiency | x | |
| Cost | Dataset Dependent | Dataset Dependent |
In Summary
If you are looking for ease of administration, then AWS Firehose is the easiest way to reliably load data into Splunk. Its fully managed service automatically scales and requires no ongoing administration. Most importantly it’s considered Splunk best practices, a good rule to live by would be, “Firehose first, Firehose Last, and Lambda only when required.”
If you are currently hosting your Splunk instance in AWS, or considering hosting Splunk in AWS, and could use some guidance, contact us directly. One of our Sr. Splunk/AWS experts can help guide you through any technical hurdles, or questions you may have.
About SP6
SP6 is a Splunk consulting firm focused on Splunk professional services including Splunk deployment, ongoing Splunk administration, and Splunk development. SP6 has a separate division that also offers Splunk recruitment and the placement of Splunk professionals into direct-hire (FTE) roles for those companies that may require assistance with acquiring their own full-time staff, given the challenge that currently exists in the market today.
