L6. Centralized Logging & Analysis: CloudWatch, Athena & Security Lake
Video generating
Check back soon for the video lesson on Centralized Logging & Analysis: CloudWatch, Athena & Security Lake
Course outlineLesson 6 of 18
From real-time alerting with CloudWatch metric filters to ad hoc SQL over S3 log archives with Athena, to org-wide normalized security telemetry with Security Lake.
Exam scope: Use the current SCS-C03 guide linked below; this lesson covers the topic rather than the complete exam syllabus.
Real-Time Alerting with CloudWatch
CloudWatch Logs centralizes log aggregation from AWS services and installed agents into log groups you can search and retain. On top of raw logs, metric filters turn matching log patterns into numeric CloudWatch metrics. For example, a metric filter that matches the string "AccessDenied" converts every occurrence into a data point on a metric. A CloudWatch Alarm watching that metric can then notify your team, typically via SNS, within minutes of a spike, giving you near-real-time detection without building a separate alerting pipeline.Ad Hoc SQL Over Massive Log Archives
Amazon Athena runs standard SQL directly against log files sitting in S3, including CloudTrail logs, VPC Flow Logs, and ALB access logs, without provisioning any servers. When you need to answer a one-off historical question, such as "which IAM principal called this API last quarter", across terabytes of archived logs, Athena provides serverless SQL access, with no cluster to manage or keep running. Choose a pricing model for the workload rather than assuming all queries use the same billing model.Normalizing Security Telemetry Across the Board
AWS Security Lake centralizes security telemetry from AWS services, on-premises sources, and supported third-party security tools into a single, customer-owned, S3-based data lake. Its defining feature is normalization: every source is converted into the Open Cybersecurity Schema Framework (OCSF), a common schema, so a single query language and a single set of detection logic works consistently across sources that would otherwise each have their own log format.Security Lake supports query access through AWS Lake Formation tables, for services such as Athena, and data access for subscribers that consume S3 objects after HTTPS or SQS notifications. Choose the access model per subscriber and grant only the sources and Regions it needs. Custom sources must supply data in the supported OCSF schema and Apache Parquet format; Security Lake does not parse arbitrary third-party logs automatically.
Balancing Investigation Speed Against Cost
High-volume raw logs like VPC Flow Logs are expensive to keep in hot storage indefinitely. The common pattern is a lifecycle policy: keep recent logs in CloudWatch Logs or S3 Standard for an initial investigation window, when queries need to be fast, then transition older logs to S3 Glacier once that window closes. Choose the exact storage class for the required retrieval time. Athena cannot query archived Glacier Flexible Retrieval or Deep Archive objects until you restore them. On Athena engine version 3, supported Hive tables also need the read_restored_glacier_objects table property set to true; Athena does not perform the restore.
| Tool | Purpose |
|---|---|
| CloudWatch Logs metric filters + alarms | Near-real-time alerting on log patterns |
| Amazon Athena | Serverless SQL over historical S3 log archives |
| AWS Security Lake | OCSF-normalized, centralized, multi-source security data lake |
| S3 lifecycle to Glacier | Cost control for high-volume raw logs after the hot window |
References
- CloudWatch log metrics
- Athena restored archive queries
- Athena pricing models
- Security Lake subscriber access
- Security Lake custom-source requirements
- AWS SCS-C03 exam guide
Continue Learning
- ✓CloudWatch Logs metric filters convert matching log patterns into numeric metrics, which CloudWatch Alarms can use for near-real-time alerting
- ✓Amazon Athena runs SQL directly against log files in S3 (CloudTrail, VPC Flow Logs, ALB logs) with no servers to provision, ideal for ad hoc historical investigation
- ✓AWS Security Lake normalizes security telemetry from AWS, on-premises, and supported third-party sources into the OCSF format inside a customer-owned S3-based data lake
- ✓Security Lake query subscribers use Lake Formation tables; data subscribers consume S3 objects using object notifications
- ✓Lifecycle policies typically move high-volume raw logs (like VPC Flow Logs) from hot storage to S3 Glacier after an initial investigation window, balancing cost against query speed
1. A security team wants to be paged within minutes whenever "AccessDenied" errors spike in application logs already flowing into CloudWatch Logs. What should they configure?
2. A team needs to run one-off SQL queries against two years of archived VPC Flow Logs stored in S3, without standing up any servers. What is the best tool?
3. What is the primary benefit of AWS Security Lake over sending logs directly to a SIEM from many separate sources?