Logging for Cloudwatch Events using Splunk HEC
Welcome to another installment on how to log multiple AWS accounts into Splunk, when the recommended method is not feasible. In this article, we will take a look at Splunking AWS Cloudwatch Event data using an HTTP Event Collector (HEC) input. The most common method to retrieve Cloudwatch Events data is to use the Cloudwatch Logs input configuration through the AWS Add-On.
Pull (near real-time)
The pull method involves using the AWS Add-on for Splunk and initiating the API request from a primary Splunk system. While the method is straightforward, there is sometimes the desire to use a push method for AWS logging into Splunk. The process is further complicated when a single point of log consolidation is required or desired for multiple AWS accounts. We are going to examine the ‘push’ method that involves Splunk HEC as our data receiver.
AWS Lambda
Options available to send messages involve setting a subscription to AWS Lambda. This step is offered as an action once the associated Cloudwatch Logging group is selected. The data is processed by an AWS Lambda step function that processes the data and lastly sends it as an HTTP Post message to the Splunk HEC endpoint, configured in the Lambda config. As with other Lambda functions, care should be taken when planning this method due to the high volume of messages that will be traversing the function.
The processing of the Cloudwatch Events should still utilize the existing sourcetype of aws:cloudwatch logs provided by the AWS Add-on for Splunk. Therefore, the AWS Add-on should still be utilized however no inputs need to be created via the add-on as done with the traditional pull configurations.
As with nearly all AWS services there will be a cost associated with using Lambda.
AWS Account consolidation
There are references available in the AWS environment that detail the options and processes needed to allow account consolidation for the purpose of logging. One solution is to create a specialized account with the purpose of logging into external or internal software for further analysis and processing. The steps and methodology to centralize AWS logging are in documents/blogs/articles on the AWS site.
A couple of great links from AWS about AWS Account Consolidated Logging are:
https://aws.amazon.com/answers/logging/centralized-logging/
https://aws.amazon.com/blogs/architecture/central-logging-in-multi-account-environments/
In Conclusion
AWS Cloudwatch Events help organizations onboard data into Splunk from AWS Services such as Lambda, Batch, Kinesis, and so forth. The ability to reliably log the information in a reliable and highly available implementation establishes the value of the data as well as the information that it contains. Using Splunk HEC to log the data gives customers a solution that can withstand growth and failures. There are multiple methods that can be used to get the data into Splunk HEC, the method above allows organizations without AWS external endpoints and CA-signed certs to still utilize the power of Splunk HEC.
If you are currently hosting your Splunk instance in AWS, or considering hosting Splunk in AWS, and could use some guidance, contact us directly. One of our Sr. Splunk/AWS experts can help guide you through any technical hurdles, or questions you may have.
About SP6
SP6 is a Splunk consulting firm focused on Splunk professional services including Splunk deployment, ongoing Splunk administration, and Splunk development. SP6 has a separate division that also offers Splunk recruitment and the placement of Splunk professionals into direct-hire (FTE) roles for those companies that may require assistance with acquiring their own full-time staff, given the challenge that currently exists in the market today.
