CRITICALSupply Chain30-60 min containment16 steps across 5 phases

    Supply Chain Attack

    A malicious package, library, or container image has been introduced into your software supply chain. This could be a typosquatted npm/pip package, a compromised open-source dependency, or a poisoned container image in ECR. The malicious code may exfiltrate secrets, create backdoors, or mine cryptocurrency.

    Phase 1: Detection

    $ tail -f /var/log/cloudtrail/events.log
    1

    Check Inspector for vulnerability findings

    Amazon Inspector continuously scans Lambda functions and ECR images for known vulnerabilities.

    aws inspector2 list-findings \
      --filter-criteria '{"findingType":[{"comparison":"EQUALS","value":"PACKAGE_VULNERABILITY"}]}' \
      --max-results 20
    2

    Scan ECR images for malicious content

    Check ECR scan results for critical and high severity findings in recently pushed images.

    aws ecr describe-image-scan-findings \
      --repository-name <repo-name> \
      --image-id imageTag=latest
    3

    Review CodeBuild logs for suspicious install commands

    Check build logs for unexpected network calls during dependency installation (npm install, pip install, etc.).

    aws codebuild batch-get-builds \
      --ids <build-id> \
      --query 'builds[0].logs'
    4

    Check for unexpected outbound network connections

    Monitor VPC Flow Logs and DNS queries for connections to unknown external endpoints during build or runtime.

    GuardDuty:Trojan:Runtime/BlackholeTrafficUnauthorizedAccess:Runtime/TorRelay

    Phase 2: Containment

    $ ./containment.sh --isolate --immediate
    1

    Stop all deployments using the compromised dependency

    Halt CI/CD pipelines to prevent further deployment of the compromised code.

    2

    Isolate affected workloads

    Apply restrictive network policies to containers/functions running the compromised code.

    # For ECS: update service to desired count 0
    aws ecs update-service \
      --cluster <cluster> \
      --service <service> \
      --desired-count 0
    # For Lambda: set concurrency to 0
    aws lambda put-function-concurrency \
      --function-name <function-name> \
      --reserved-concurrent-executions 0
    3

    Lock the ECR repository

    Set the repository to immutable tags and add a deny policy to prevent new pushes while investigating.

    aws ecr put-image-tag-mutability \
      --repository-name <repo-name> \
      --image-tag-mutability IMMUTABLE

    Phase 3: Eradication

    $ ./eradicate.sh --purge --verify
    1

    Identify and remove the malicious dependency

    Review package.json/requirements.txt/Cargo.toml for suspicious packages. Check for typosquatting (e.g., "lodash" vs "l0dash").

    2

    Pin all dependencies to exact versions

    Replace version ranges (^1.0.0) with exact versions (1.0.0) and use lock files.

    Use package-lock.json (npm), Pipfile.lock (pip), or Cargo.lock (Rust) to ensure reproducible builds.

    3

    Rebuild and rescan all container images

    Rebuild all images from scratch with verified dependencies and re-scan with Inspector.

    aws ecr start-image-scan \
      --repository-name <repo-name> \
      --image-id imageTag=<tag>

    Phase 4: Recovery

    $ ./recovery.sh --restore --validate
    1

    Deploy clean versions

    Push rebuilt, scanned images and redeploy all affected services.

    2

    Rotate all secrets the compromised code could access

    Any secrets available to the compromised workload (environment variables, Secrets Manager, IAM role) should be rotated.

    3

    Re-enable CI/CD pipelines with safeguards

    Resume deployments with additional scanning steps in the pipeline.

    Phase 5: Lessons Learned

    $ cat POST_INCIDENT_REVIEW.md
    1

    Enable Inspector continuous scanning

    Ensure Amazon Inspector is scanning all ECR repositories and Lambda functions continuously.

    aws inspector2 enable --resource-types ECR LAMBDA
    2

    Implement a software bill of materials (SBOM)

    Generate and maintain SBOMs for all applications to quickly identify exposure to compromised dependencies.

    3

    Use private registries with curation

    Mirror approved dependencies in a private registry (AWS CodeArtifact) rather than pulling directly from public registries.

    supply-chaindependencycontainermalicious-packageecr

    Need Help with Incident Response?

    When an incident strikes, every minute counts. We help AWS teams prepare, detect, and respond to security incidents with proven expertise.