Skip to content

Incident Response / Security Automation

Automated Incident Response & Escalation

Built automated incident response workflows that escalated security events to the right teams via PagerDuty and Slack, reduced manual triage, and preserved forensic evidence automatically.

Incident ResponsePagerDutySecurity AutomationAWSPython

Problem

Security events required manual triage before escalation. On-call engineers had to determine severity, find the right team, page them, and collect evidence, all while the clock was running. Critical context was lost between handoffs.

Constraints

Multiple on-call rotations across engineering and security, no dedicated SOC, incidents could span AWS accounts, and evidence had to be preserved before containment actions changed the state.

Approach

Built Python-based automation that listened for security events, classified severity using detection metadata, automatically paged the right on-call rotation via PagerDuty API, created structured Slack threads with context, and triggered evidence preservation jobs (CloudTrail log exports, IAM credential reports, instance metadata snapshots) before any containment action.

Outcome

Reduced mean time to escalation from hours to under 5 minutes. Eliminated lost context between handoffs. Every incident now starts with a structured thread, pre-collected evidence, and the right people already paged.

Terminal

~/security/notes
# security event triggers automated response
$ python3 incident_router.py --event guardduty-credential-exfil
[+] Event classified: severity=CRITICAL, type=credential-exfiltration
[+] PagerDuty: paging security-oncall (P1)
[+] PagerDuty: paging infra-oncall (P2, informational)
[+] Slack: created incident thread in #sec-incidents
# evidence preservation runs before containment
[+] CloudTrail: exported last 24h logs to s3://forensics-vault/incident-2026-0531/
[+] IAM: credential report snapshot saved
[+] EC2: instance metadata captured for i-0a1b2c3d4e5f
[+] Ready for containment. Evidence preserved.

Tools

PythonPagerDuty APISlack APIAWS CloudTrailAWS LambdaEventBridgeS3

Key Highlights

  • Automated escalation via PagerDuty API
  • Severity-based routing to on-call teams
  • Automatic evidence preservation (CloudTrail snapshots, log exports)
  • Slack-based incident coordination with structured threads
  • Reduced mean time to escalation from hours to minutes
Details are intentionally sanitized to protect employer and client confidentiality.
← All WorkDiscuss Security Work