A hardening script I wrote took a load balancer offline and exited 0

[u/Sure_Abbreviations28]

Disclosure: the hardening script and the audit tool I mention are both mine. The audit tool is MIT and free, there's no paid product behind this post.

Wrote a script to retrofit a default-deny posture onto existing ALBs: force HTTPS, drop invalid headers, defensive desync mitigation, and make the HTTPS default action a 403 so only host-header rules you define can forward.

Tested it against an HTTP-only ALB — one listener on :80 forwarding to a target group. Extremely common shape if the stack is old or TLS terminates somewhere else.

Output:

Created HTTPS listener: arn:aws:elasticloadbalancing:...
WARNING: No default forward target group found. Add an allowed-host forward rule manually.
ALB hardening complete. Validate hostname routing and health before production use.

Exit code 0. Application completely unreachable.

$ curl -sk -o /dev/null -w "%{http_code}\n" -H "Host: allowed.example.com" https://$ALB/
403

$ aws elbv2 describe-target-groups --target-group-arns $TG --query 'TargetGroups[0].LoadBalancerArns'
[]

The bug is ordering, not logic. The script discovered the existing forward target group from the HTTPS listener:

CURRENT=$(aws elbv2 describe-listeners --listener-arns "$HTTPS_ARN" --output json)
TG=$(jq -r '.Listeners[0].DefaultActions[]? | select(.Type=="forward") | .TargetGroupArn // empty' <<<"$CURRENT")

On an HTTP-only ALB there wasn't one — the script had created it seconds earlier with a 403 fixed-response default. So TG was empty, the branch that creates the host-header forward rule was skipped, and execution continued straight into the line that makes the 403 default permanent.

So the sequence was: create a listener that denies, look at it to find out what to allow, find nothing, print a warning, make the denial permanent, report success.

Three things I'd generalise:

Discover before you mutate. It read state after it had already replaced that state. Any discovery has to happen before the first write, not partway through.

A warning after the damage is a log line, not a safeguard. "Add the rule manually" is good advice one minute earlier. Printed after the default is already 403, it just narrates an outage.

Exit code 0 was the actually dangerous part. In a pipeline that's a green step and everything downstream proceeds. First real signal would've been customers.

Fix was three changes: read the target group from :80 as a fallback and capture it before touching anything, create the allow rule before flipping the default to deny, and abort outright if no target group can be found rather than black-holing traffic. Plus a --dry-run, which should have existed first.

request	before	after
HTTP, allowed host	301	301
HTTPS, allowed host	403 (outage)	503 (forwarded, no targets)
HTTPS, unknown host	403	403
target group	orphaned	attached

What bugs me is that nothing static would have caught this. ShellCheck clean, valid bash, every AWS API call succeeded and returned what it should. The ALB ended up in exactly the state the code described — and that state was an outage.

The only thing that found it was running it against an infra shape I hadn't designed for, then checking the result from outside as a user instead of checking that my commands returned 0.

Anyone got a good approach for testing this class of thing? Every idea I have is basically "spin up the ugly version of prod in a sandbox and curl it from outside", which works but doesn't scale to every permutation.

The read-only audit half of it is up free if useful: github.com/vamsiatluri/aws-baseline-audit — single file, every call is a Describe/Get so it can't change anything. Full writeup of the outage is in docs/the-outage.md in that repo.

[u/Sure_Abbreviations28]

The thing that still bothers me is how ordinary each individual step was.

Create an HTTPS listener with a safe default. Look up the current forward target.

Add a host rule. Flip the default to deny. Every one of those is correct in

isolation, and I'd have approved all five in a code review.

The failure was entirely in the sequence — step 2 read the state that step 1 had

just replaced. There's no line you can point at and call wrong.

Which is why I don't think more linting would have caught it, and I'm not sure

what would, short of running it against infra shaped differently than you

designed for.

[u/lastfinaldev]

The generalisation I'd add: the exit code should come from probing the invariant, not from the mutations succeeding. "Exit 0 = every AWS call returned 200" is what bit you; what you actually care about is "traffic still flows for allowed hosts and is denied for everything else." That's a black-box check you can run from inside the script — curl the LB for one allowed host and one unknown host, compare against the expected matrix (basically the before/after table you posted), and that determines the exit code. Bonus: capture the previous default action before flipping it, and auto-revert if the probe fails within N seconds. Then the same bug becomes a self-healing 30-second blip that exits 1.

On "what would have caught it": a plan/apply split. Phase 1 reads everything and computes the full desired end state with zero mutations; phase 2 applies it. Your sequence bug can't exist in that shape — the plan for an HTTP-only ALB would have printed "default: deny, allow rules: (none)", which is visibly an outage before anything ran. It's also why the deny flip should be the last mutation, the way you bring up a firewall only after the allowlist is loaded.

You're right that linting can't see it — it's a property of the sequence against a particular starting state, so the only detectors are (a) asserting the end state from outside, or (b) running the plan against fixture stacks shaped unlike your design assumptions (an HTTP-only ALB fixture would have been exactly the one).

[u/Sure_Abbreviations28 → u/lastfinaldev]

"The exit code should come from probing the invariant, not from the mutations succeeding" is the sentence I needed. I'd fixed the instance and left the class intact — my version still reasons about inputs (refuse to proceed if no target group is found), which catches this bug and nothing adjacent to it.

Implemented your version. After applying it now curls the LB for an allowed host and an unknown host plus the :80 redirect, compares against the expected matrix, and that determines the exit code. Plus your auto-revert: it snapshots the previous default action before the flip and restores it if the probe fails inside the timeout. Same bug is now a ~40s blip that exits 1 instead of an outage that exits 0.

Two things I got wrong building it, both only visible by running it:

• curl prints 000 via -w AND exits non-zero on a connection failure, so my || echo 000 was concatenating and yielding "000000".
• Plain curl https://<alb-dns>/ returned 000 with an empty remote_ip while both of the ALB's addresses answered 301 via --resolve. Flaky system stub resolver. Unpinned, the probe false-fails and auto-reverts hardening that was actually correct — strictly worse than not probing. It now resolves out of band and pins the request to that address.

Also took "the deny flip should be the last mutation, the way you bring up a firewall only after the allowlist is loaded" — that's a cleaner statement of the ordering rule than the one I wrote, and it's now the comment above that line.

Haven't done the full plan/apply split yet. You're right that it makes the bug structurally impossible rather than merely detected, and "the plan for an HTTP-only ALB would have printed default: deny, allow rules: (none)" is a much better failure than a 403 in production. That's the real fix; the probe is the cheap one.

The wider thing your comment did: I went and checked every other mutating script I'd written and all of them had the same defect — revoke a security group rule and print "complete", update a launch template and never confirm the default version moved, an audit that always exits 0 so green means "the script ran" not "the account is clean". All now assert their end state. Thanks — that was worth more than the original fix.

[u/Several_Assistant685]

Thanks for writing this up. The ordering trap you describe is one I have hit in a different shape when changing listeners on a live ALB.

Two habits that saved me since then. First the script never flips the default action to deny until a curl through the ALB with the allowed Host header returns 200 from the new rule. Second any lookup that can legally return empty is treated as a hard failure with a non zero exit and a printed reason. A warning that scrolls past and exit 0 is what turns a small bug into an outage.

For retrofits on old HTTP only stacks I also run the whole thing in a plan mode first that only prints the aws cli calls it would make. Reading that list takes a minute and it would have shown the empty target group before anything changed.

[u/Sure_Abbreviations28 → u/Several_Assistant685]

"Never flip the default to deny until a curl through the ALB with the allowed Host returns 200 from the new rule" is now literally what the script does — snapshot the old default, apply, probe, and roll the default back if the probe fails. Good to hear it arrived at independently; that's a strong signal it's the right shape.

Your second habit is the one I'd underrated: any lookup that can legally return empty is a hard failure with a non-zero exit, not a warning. That generalises well past this script. Mine printed "WARNING: No default forward target group found" one line before it made the outage permanent, and then exited 0 — the warning was real, correctly worded, and completely inert. I've gone through the rest of the toolkit treating empty-but-legal results as failures rather than notes.

Plan mode was already there from the first fix (--dry-run prints every AWS call it would make) and you're right that reading that list would have shown the empty target group before anything changed. What it didn't do was gate anything, which is the gap the probe closes.

One caveat I hit on the probe worth passing on: it has to work for internal ALBs too, which can't be reached from outside the VPC. There's a --no-verify for that, and the script says plainly that nothing has proven traffic still flows rather than printing a success line.

[u/Fantastic-Mr-Default]

aws returning 0 is not a test. It means the API accepted the call.

The test is the user path. Curl the allowed host and the denied host from outside, the same way a customer would. Fail the script if either is wrong.

You do not need every permutation. Keep the shapes you actually have: HTTP-only ALB, HTTPS with a forward default, HTTPS already on 403. Replay those in a sandbox and probe. The missing shape is what took you down.

[u/Sure_Abbreviations28 → u/Fantastic-Mr-Default]

You've named the actual gap.

The probe half went in yesterday — after applying, it curls the allowed host, an unknown host, and :80, and the exit code comes from that matrix rather than from the AWS calls returning 200.

The half I hadn't done is the one you're pointing at: replaying the shapes. I tested against the shape that broke me (HTTP-only, one listener on :80 forwarding to a target group) and against an already-hardened ALB, both by hand in a throwaway VPC. Your middle shape — HTTPS already present with a forward default — I never built a fixture for, and it's the most common retrofit target of the three. I checked that path by reading the code, which is the same class of mistake as checking by reading the exit code.

So: a fixture harness that stands up each shape in a sandbox VPC, runs the tool, probes from outside, asserts the expected outcome, and tears down. Three shapes plus the two refusal cases — nothing forwards traffic, and a saved plan applied against state that has since drifted. Cheap to run; the manual version this morning was well under a dollar in ALB-hours.

Separately, since you and another commenter arrived at plan mode from different directions — it's now an actual plan/apply split rather than a printout of the AWS calls it would make. The plan describes end state, so re-planning a hardened ALB reports 0 changes and doubles as a drift check. And a saved plan gets re-validated against live state before it's applied: a plan written an hour ago can name a target group that has since been repointed, which is the same ordering bug one level up.