EKS Forensics & Incident

Response

Cado Security | 1
 How do you respond to a compromised EKS
Container or Node?
If you’ve identified a potentially compromised container in EKS, there
are two potential ways forward:

● If the container is running on an underlying EC2, then refer to the

suggested steps above for immediate actions.

● If the container is running on Fargate, then collect any data required

for later analysis before subsequently suspending it.
 What EKS GuardDuty Detections are
there?

https://aws.amazon.com/blogs/security/how-to-use-new-amazon-guardduty-eks-protection-findings/
https://medium.com/@cloud_tips/guide-to-aws-guardduty-findings-in-eks-62babbd7da88
 Container Investigation Data Sources in AWS?
Amazon S3
EKS Audit / Control Plane Logs
● Shows: API Level Calls
● Usefulness: Medium
● Collected by: S3
CloudTrail Logs
● Shows: API Level Calls
● Usefulness: Low
● Collected by: S3
Amazon EC2 - Hosting EKS/ECS
Docker Container Filesystems
● Normally overlay2 versioned filesystem
● Contains all the files from all the containers
● Usefulness: High
● Collected by: EC2 EBS (API) or Cado Host (SSM/SSH)
Docker Logs
● Logs what containers were started, stopped
● Usefulness: Medium
● Collected by: EC2 Import or Cado Host
Inside Container - EKS/ECS on Fargate/EC2
Container Filesystems
● Live filesystem as seen by the container, Memory
● Contains all the files from all the containers
● Usefulness: Very High
● Collected by: Cado Host (ECS Exec/kubectl exec))
How do you Acquire an Amazon EKS System
in Cado?
 What is overlay2?
Overlay2 is the file system you are most likely to see.
It’s also versioned, which helps preserve evidence of attacks.
Separate containers are kept in their own folders:
 What AWS EKS Logs are Stored in AWS?
It's important to also analyze AWS logs that are generated for EKS systems.
These contain metadata around starting and stopping containers.
Below you can see a view of AWS logs collected in Cado Response:
 What Resources are available?

kube-forensics allows a cluster administrator to

dump the current state
Community of a running pod and all its containers so that
Resources security professionals can perform offline forensic
analysis.

We previously published a playbook dedicated to

investigating compromises in EKS environments.
Check out the GitHub repository with sample data
Cado Security
taken from a compromised EKS system, and an
Resources
associated talk on how to analyze it.
 What Remediation is available?

https://aws.amazon.com/blogs/security/how-to-investigate-and-take-action-on-security-issues-in-
amazon-eks-clusters-with-amazon-detective-part-2/
Cado Response
Free 14-day trial
Receive unlimited access to the
Cado Response Platform for 14
days.

www.cadosecurity.com/free-investigation/