Consolidated AWS Account Logging of Cloudtrail and Config Services Into Splunk

AWS Cloudtrail and AWS Config micro-services are vital to companies as they migrate to or begin the process of using Amazon Web Services (AWS).  Both services allow the management and monitoring of security, audit, or configuration domains within the AWS Cloud.  The Splunk Add-on for Amazon Web Services allows AWS customers the capability to utilize the many features of Splunk software and adds the availability of premium apps such as Splunk Enterprise Security and Splunk IT Service Intelligence.

In this post, we will look at two scenarios where multiple AWS accounts are configured to log Cloudtrail and Config into a consolidated S3 bucket to be ingested into your Splunk system.  The first scenario is where the traditional Splunk Add-on resides on a Splunk Heavy Forwarder.  The second scenario involves using the Splunk HEC input with the AWS Lambda service.

The following two scenarios were chosen because they are the most common acceptable solutions to the Multiple AWS Accounts logging AWS Cloudtrail and AWS Config use case.  There may be other methods that will work better for a given customer and these two are not the only solutions.

Scenario 1

This logging method involves a Splunk heavy forwarder where the AWS add-on is installed and configured.  Splunk recommendations such as using S3 buckets, SQS queues, and potentially SNS should be utilized where possible. The option to invoke the assumeRole feature can make administration easier than entering and managing the AWS accounts with secret keys individually.  The basic steps and flow for this scenario are documented within the Splunk documentation and there are videos on YouTube on this subject as well.

Scenario 2

Assumptions:

In this scenario, it is assumed that Splunk HEC is configured for use in the Splunk environment, AWS Lambda service is enabled within the environment, and AWS VPC’s and policies have been configured to allow traffic between Lambda and the VPC where the Splunk HEC Endpoint is configured.  In this environment, there is a desire to move from pulling logs to pushing logs.

Flow Steps:

1. Cloudtrail or Config is assigned to log in to a particular S3 bucket
2. S3 bucket sends notification of a new object to a Lambda function which will process the notification and pull the data mentioned in the notification
3. In the case of Cloudtrail or Config data, the payload is compressed, the Lambda function decompresses the payload and converts it to JSON format. After the conversion, the data is sent via an HTTP Post to the Splunk HEC endpoint.
4. In the event of an error, the payload is not discarded but placed into a designated SQS Queue (Dead Letter Queue)

Advantages:

• Capability for redundant logging endpoints for AWS Cloudtrail and AWS Config
• AWS Cloudtrail and AWS Config AWS API throttling for throughput risk mitigation
• AWS Cloudtrail and AWS Config AWS API Key dependency risk mitigation
• Reduction in the number of AWS modular inputs on a given HF and therefore their management

Caveats:

• Consolidated logging to a central logging account will require custom policies and permissions on S3 buckets and possible IAM roles that will allow the assumeRole feature
• Lamba is NOT a free service and there will be an additional cost associated with it every time the function runs. The cost will depend on the memory allocated and the timeout for the function.  Check your AWS contract or contact them for better pricing information
• Basic knowledge of Python, Javascript, or other languages allowable by AWS Lambda is highly beneficial when modifying or creating a blueprint (code) for Lambda function execution

Summary

Cloud computing services have experienced a growth curve over the past few years.  The need for security, audit, and configuration domain insight is an important component that will help organizations as they migrate or add assets to their cloud environment.

Splunk’s AWS integration is a solution that can greatly help with logging AWS Cloudtrail and AWS Config services so that those insights are available easily and when needed.  The complexity of the AWS environment may include the logging of multiple accounts into one Splunk environment.

In the use case of one (1) Splunk environment with multiple AWS accounts there are at least 2 options available to ensure that data is indexed and available for analysis.  Option 1 involves the traditional use of a Splunk heavy forwarder while Option 2 does not require a Splunk heavy forwarder and uses Splunk HTTP Event Collector (HEC).

Both options are very robust, however, Option 2 has the capability to mitigate risks and is easily suited to growth in the number of AWS accounts and in the amount of data being indexed.

If you are currently hosting your Splunk instance in AWS, or considering hosting Splunk in AWS, and could use some guidance, contact us directly.  One of our Sr. Splunk/AWS experts can help guide you through any technical hurdles, or questions you may have.

About SP6

SP6 is a Splunk consulting firm focused on Splunk professional services including Splunk deployment, ongoing Splunk administration, and Splunk development. SP6 has a separate division that also offers Splunk recruitment and the placement of Splunk professionals into direct-hire (FTE) roles for those companies that may require assistance with acquiring their own full-time staff, given the challenge that currently exists in the market today.