L18. Audit Logging and Incident Response in Kubernetes

Kubernetes Audit Logs

Audit logs record API server requests selected by the audit policy. They answer the critical forensic questions: who did what, when, and from where.

Each audit event includes:

• User: The authenticated identity (user, service account, or anonymous)
• Verb: The operation (get, create, delete, patch)
• Resource: What was affected (pod, secret, role)
• Namespace: Where it happened
• Source IP: Where the request came from
• Response code: Whether it succeeded

Audit Levels, Stages and Backends

Levels are None, Metadata, Request (metadata and request body), and RequestResponse (also response body). The first matching policy rule wins. Stages are RequestReceived, ResponseStarted (long-running requests), ResponseComplete, and Panic. Not every request emits every stage.

On self-managed control planes, set kube-apiserver --audit-policy-file and enable the log backend with --audit-log-path, or the webhook backend with --audit-webhook-config-file. Without an audit policy, no events are logged. Restrict log access and retention; Falco complements API auditing with runtime detection.

Configuring Audit Policies

A good audit policy balances visibility with log volume:

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  # Log Secret access without leaking secret values into logs
<ul class="list-disc pl-6 mb-4 space-y-2">
<li class="text-slate-300">level: Metadata</li>
</ul>
    resources:
<ul class="list-disc pl-6 mb-4 space-y-2">
<li class="text-slate-300">group: ""</li>
</ul>
        resources: ["secrets"]
  # Log RBAC changes with full details
<ul class="list-disc pl-6 mb-4 space-y-2">
<li class="text-slate-300">level: RequestResponse</li>
</ul>
    resources:
<ul class="list-disc pl-6 mb-4 space-y-2">
<li class="text-slate-300">group: "rbac.authorization.k8s.io"</li>
</ul>
        resources: ["clusterroles", "clusterrolebindings", "roles", "rolebindings"]
  # Log pod exec and port-forward
<ul class="list-disc pl-6 mb-4 space-y-2">
<li class="text-slate-300">level: Request</li>
</ul>
    resources:
<ul class="list-disc pl-6 mb-4 space-y-2">
<li class="text-slate-300">group: ""</li>
</ul>
        resources: ["pods/exec", "pods/portforward", "pods/attach"]
  # Log node and namespace operations
<ul class="list-disc pl-6 mb-4 space-y-2">
<li class="text-slate-300">level: Metadata</li>
</ul>
    resources:
<ul class="list-disc pl-6 mb-4 space-y-2">
<li class="text-slate-300">group: ""</li>
</ul>
        resources: ["nodes", "namespaces"]
  # Log everything else at Metadata level
<ul class="list-disc pl-6 mb-4 space-y-2">
<li class="text-slate-300">level: Metadata</li>
</ul>
    omitStages: ["RequestReceived"]

Log Shipping

Send audit logs to a centralized SIEM for analysis and alerting:

• Managed clusters: EKS sends audit logs to CloudWatch (enable in cluster logging settings). AKS sends to Azure Monitor. GKE sends to Cloud Logging.
• Self-managed: Ship logs from the audit log file to Elasticsearch, Splunk, or your SIEM using Fluentd or Vector.

Incident Response Playbook

When a security incident is detected in Kubernetes: 1. Contain

# Isolate the compromised pod with a deny-all network policy
kubectl label pod compromised-pod quarantine=true -n production
# Apply isolation policy
kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: quarantine
  namespace: production
spec:
  podSelector:
    matchLabels:
      quarantine: "true"
  policyTypes: ["Ingress", "Egress"]
EOF

NetworkPolicies are additive: this policy isolates only when no other policy allows traffic to or from the pod, and the CNI enforces NetworkPolicy. Review existing policies and verify containment; escalate to node or network isolation if needed. 2. Preserve evidence

# Capture pod details before deletion
kubectl get pod compromised-pod -n production -o yaml > pod-evidence.yaml
kubectl logs compromised-pod -n production --all-containers > pod-logs.txt
kubectl describe pod compromised-pod -n production > pod-describe.txt
# Copy /tmp artifacts (not a complete filesystem snapshot)
kubectl cp production/compromised-pod:/tmp ./evidence/tmp/

3. Investigate Query audit logs for the compromised identity:

# Pseudocode: translate to the query language and schema of your SIEM
user.username == "system:serviceaccount:production:web-app"
AND verb in ("create", "patch", "delete")
AND objectRef.resource in ("secrets", "roles", "clusterroles")

4. Eradicate

• Rotate all secrets the compromised pod had access to
• Revoke the service account's RBAC permissions
• Delete and recreate the compromised pod from a known-good image
• Scan the image for vulnerabilities and verify its signature

5. Recover and improve

• Tighten RBAC to remove over-permissions found during investigation
• Add Network Policies to prevent the lateral movement path
• Add Falco rules to detect the attack pattern
• Update Pod Security Standards if the attack exploited weak security contexts

Key Detection Queries

References

• Kubernetes auditing
• NetworkPolicy behavior