Building a Security Audit Pipeline with AWS CloudTrail, CloudWatch & Athena

In today's cloud-native world, security is paramount. Organizations need robust tools to audit their AWS environments effectively. By leveraging AWS CloudTrail, CloudWatch, and Athena, we can set up a security audit pipeline that ensures complete visibility, proactive monitoring, and real-time alerting.

In this guide, we’ll use these three powerful tools to build a security audit pipeline that aggregates, stores, and analyzes logs from AWS services.

What You’ll Need

1. AWS CloudTrail: AWS CloudTrail records API calls made on your account, providing a complete activity history.
   
   

2. AWS CloudWatch: CloudWatch allows you to monitor and trigger alarms based on specific metrics and logs.
   
   

3. AWS Athena: Athena enables querying of log data stored in Amazon S3 using SQL queries without the need for provisioning infrastructure.
   
   

---

Step 1: Setting Up AWS CloudTrail

CloudTrail captures all API activity within your AWS account and stores it in S3. This is the foundation of our audit pipeline.

1.1 Enabling CloudTrail

1. Go to the CloudTrail Console.
   
   

2. Click on Create Trail.
   
   

3. Name your trail (e.g., SecurityAuditTrail).
   
   

4. Select All Regions to ensure that CloudTrail tracks activity across every region.
   
   

5. Choose S3 as the destination for your logs.
   
   

6. Optionally, enable CloudWatch Logs integration for real-time monitoring.
   
   

1.2 Verifying CloudTrail Configuration

Ensure the trail logs all management events, including Write and Read actions, across your entire AWS account. The logs are stored in an S3 bucket for easy access and querying.

---

Step 2: Setting Up CloudWatch for Real-Time Monitoring

CloudWatch offers powerful features to track security-related metrics and trigger alerts.

2.1 Create Custom CloudWatch Logs Group

1. Go to the CloudWatch Console.
   
   

2. Under Logs, create a Log Group (e.g., SecurityAuditLogs).
   
   

3. Set up Log Streams for CloudTrail logs to be streamed in real time.
   
   

2.2 Set Up CloudWatch Alarms

1. Navigate to CloudWatch Alarms.
   
   

2. Choose Create Alarm and select the metrics you want to monitor.
   
   

• For example, track failed AWS Console logins, unauthorized access attempts, or resource creation events.
  
  

3. Configure the alarm actions (email notifications, Lambda invocations, etc.).
   
   

2.3 Establish Log Insights Queries

CloudWatch Logs Insights allows you to run queries on your log data. Create custom queries to detect security anomalies, such as unauthorized access or API calls from unusual IP addresses.

---

Step 3: Using Athena for Advanced Log Analysis

Athena allows you to query log data stored in Amazon S3 directly, enabling you to perform deeper investigations into your AWS CloudTrail logs.

3.1 Set Up Athena to Query CloudTrail Logs

In the Athena Console, configure a database and table in the Athena Console Configure to point to your CloudTrail logs stored in S3.

Example SQL query:


CREATE EXTERNAL TABLE IF NOT EXISTS cloudtrail_logs (

  eventTime STRING,

  eventName STRING,

  userIdentity STRING,

  sourceIPAddress STRING,

  userAgent STRING,

  awsRegion STRING

)

ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'

WITH SERDEPROPERTIES ("separatorChar" = ",", "quoteChar" = "\"")

LOCATION 's3://your-cloudtrail-logs-bucket/AWSLogs/';

3.2 Querying CloudTrail Logs for Security Audits

You can now run SQL queries to identify suspicious activities. For example, you can look for specific API calls such as DeleteBucket or CreateNewUser:

SELECT * FROM cloudtrail_logs

WHERE eventName IN ('DeleteBucket', 'CreateUser')

AND eventTime BETWEEN '2024-08-01' AND '2024-08-30';

3.3 Automating Query Execution

Set up Athena to run scheduled queries for recurring security audits. This will save you time and ensure that nothing slips through the cracks.

---

Step 4: Integrating the Pipeline

Now that we have set up CloudTrail, CloudWatch, and Athena, integrate these tools into a cohesive pipeline.

4.1 Integrating CloudTrail with CloudWatch

Ensure CloudTrail logs are streamed to CloudWatch Logs in real time for continuous monitoring.

4.2 Automated Responses Using CloudWatch

Configure CloudWatch alarms to trigger Lambda functions, which could automate responses, such as notifying your security team, isolating compromised resources, or running cleanup procedures.

4.3 Running Scheduled Queries in Athena

Schedule Athena queries to run periodically, and save the results to S3 or even send them to a dashboard for regular monitoring.

---

Step 5: Reporting & Dashboards

Create a CloudWatch dashboard or an Athena-based reporting tool to consolidate logs and alarms for improved visibility. This will allow you to see security metrics at a glance and track trends over time.

---

Conclusion

Building a security audit pipeline using AWS CloudTrail, CloudWatch, and Athena allows you to track, monitor, and analyze AWS activity in real-time. With automated alerts, historical querying, and in-depth analysis, you can proactively identify and respond to security threats, ensuring your AWS environment stays secure.

By integrating these powerful AWS services, you establish a solid foundation for cloud security that audits activity and helps in continuous monitoring and compliance.