The Problem
Drift detection sounds great in principle and becomes useless in practice. Run terraform plan against production once a day and you’ll get a delta. Run it across 80 accounts and you’ll get a delta the length of a novel.
The teams that turn it on usually turn it off within a month. The signal-to-noise ratio is brutal: out of two hundred drift entries, maybe two represent something you’d actually want a human to look at. The other 198 are autoscaling resizing a node pool, a managed service updating its own backing IAM policy, or a tag your costing system added after the fact.
The fix isn’t better detection. It’s classification.
The Approach
Treat drift as a continuously scored risk surface, not as a binary state mismatch. Score it at detection time into three tiers:
Tier 1: Authorization, audit, and availability drift
Changes that, left alone, change who can do what, what gets logged, or what happens when something fails. Every entry in this tier pages the on-call:
- IAM policies, roles, or trust relationships
- Security group rules (especially anything opening 0.0.0.0/0)
- KMS key policies or rotation settings
- CloudTrail or VPC Flow Log configuration
- S3 bucket policies, public access blocks, encryption settings
- Backup plan deletion or retention reduction
- Production load-balancer listener changes
Drift in any of these means either an attacker is moving laterally or a human bypassed your change-control process. Either is worth waking someone up.
Tier 2: Capacity, cost, and consistency drift
Not dangerous on their own, but indicators of a process gap. These go to a weekly digest:
- Instance type changes (someone resized for cost)
- Tag additions or removals (cost-allocation drift)
- Auto-scaling group min/max adjustments
- RDS parameter group tweaks
- Lambda memory or timeout adjustments
Once a week, an engineer reviews the digest. If a pattern is real (“the data team is rightsizing RDS by hand every Friday”), the IaC catches up. If it’s one-off (“someone changed a memory limit during an incident”), the engineer either backports or accepts.
Tier 3: Known-good drift
Expected changes you don’t want cluttering the digest. Suppress them at the detector:
- Autoscaling instance counts inside declared bounds
- AWS-managed IAM policy version updates
- Service-linked role updates
- Default route table associations created by service launches
- CloudFront distribution metric subscriptions
Suppressing Tier 3 is what makes the system usable. Without it, the digest is unreadable and Tier 1 alerts get ignored.
Implementation
Tooling: terraform plan in CI on a schedule is the baseline. For scale, driftctl or Pulumi’s pulumi refresh --diff produce the same signal with less compute.
Run the detector hourly for Tier 1 (cheap, narrow scope), daily for Tier 2 (broader plan), weekly for the full estate. Route Tier 1 to PagerDuty via a webhook that includes the resource ARN and the field that changed. Route Tier 2 to a Slack channel. Route Tier 3 to /dev/null.
The classifier is regex on the planned change. Five lines of Python wraps the terraform plan JSON output into a tier judgment. No machine learning. No fancy heuristics. A static table maps aws_iam_* and aws_security_group_rule and a few others to Tier 1; everything else gets reviewed by the digest.
The Template
Below is the GitHub Actions workflow that runs the hourly Tier 1 check against an estate of AWS accounts. It uses an OIDC-federated role to assume into each account, runs terraform plan -detailed-exitcode, and routes anything matching the Tier 1 pattern to a webhook.
# .github/workflows/drift-check.yml
name: Infrastructure drift · Tier 1
on:
schedule:
- cron: '0 * * * *' # hourly
workflow_dispatch:
permissions:
id-token: write # for OIDC role assumption
contents: read
jobs:
detect:
runs-on: ubuntu-latest
strategy:
matrix:
account: [prod-us-east-1, prod-us-west-2, log-archive]
fail-fast: false
steps:
- uses: actions/checkout@v4
- uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::${{ secrets.ACCOUNT_ID }}:role/DriftDetector
aws-region: us-east-1
- uses: hashicorp/setup-terraform@v3
with: { terraform_version: 1.9.x }
- name: terraform plan
id: plan
working-directory: ./environments/${{ matrix.account }}
run: |
terraform init -input=false
terraform plan -input=false -lock=false \
-out=plan.bin -detailed-exitcode || exit_code=$?
terraform show -json plan.bin > plan.json
echo "exit_code=${exit_code:-0}" >> $GITHUB_OUTPUT
- name: Classify and route
if: steps.plan.outputs.exit_code == '2' # changes detected
run: |
python3 .ci/classify-drift.py plan.json > tier1.json
if [ -s tier1.json ]; then
curl -X POST -H 'Content-Type: application/json' \
--data @tier1.json \
${{ secrets.PAGERDUTY_WEBHOOK }}
fiThe classify-drift.py script is a 30-line filter over the JSON plan that emits Tier 1 entries only:
import json, sys, re
TIER1_RESOURCES = (
r'^aws_iam_',
r'^aws_security_group',
r'^aws_kms_key',
r'^aws_cloudtrail',
r'^aws_s3_bucket_policy$',
r'^aws_s3_bucket_public_access_block$',
r'^aws_backup_(plan|vault)',
)
plan = json.load(open(sys.argv[1]))
tier1 = []
for change in plan.get('resource_changes', []):
if change['change']['actions'] == ['no-op']:
continue
if any(re.match(p, change['type']) for p in TIER1_RESOURCES):
tier1.append({
'resource': change['address'],
'type': change['type'],
'actions': change['change']['actions'],
})
if tier1:
json.dump({'alert': 'Tier-1 drift', 'changes': tier1}, sys.stdout)Operating Notes
Run the digest review on Mondays. Anything in Tier 2 that’s been there three consecutive weeks becomes a Tier 1 candidate, or gets accepted and moved to Tier 3 with a comment explaining why. The tiers aren’t static. They’re the team’s working classification of what matters, and they should evolve as you learn what your environment actually does.
The number to watch isn’t drift incidents. It’s how often Tier 1 fires. More than once a week on average and your change-control process has a gap. Less than once a quarter and the classifier is too lenient. Someone is changing security-relevant settings without it surfacing.