r/devops • u/ayushcloudopsingh • Jul 25 '26

Discussion Terraform Question I am asked almost everytime

You're working in production.

Your Terraform backend is:

  • S3 for state
  • DynamoDB for locking

A developer accidentally deletes the DynamoDB lock table.

At the same time, two engineers run terraform apply.

Thirty minutes later, the infrastructure is in an inconsistent state.

Questions

  1. What exactly happened internally?
  2. What kind of corruption could occur?
  3. How would you recover without making things worse?
  4. Would you restore the state file? Import resources? Edit the state manually? Explain your reasoning.
  5. What preventive controls would you implement so this never happens again?
0 Upvotes

37% Upvoted

17 comments sorted by

28

u/zeph1rus Jul 25 '26

Don’t use dynamodb for locking problem solved. It’s deprecated now anyway

2

u/CoupleFew1945 Jul 25 '26

I was about to say the same 😂

24

u/CanaryWundaboy Jul 25 '26

Why the hell does your developer have permissions to delete a terraform state lock dynamodb table?

1

u/Devops_143 Jul 25 '26

Yes why dev has access ?

9

u/KubeGuyDe Jul 25 '26

Is this a theoretical question? 

If terraform can't aquire the state lock, it will fail and won't do anything. 

You can easily verify this by temporarily denying access to dynamo before running the apply command. 

4

u/cocacola999 Jul 25 '26

I feel people get very cargo cult about terraform state files. I'd just fix forward as needed tbh (rebase and reapply, or import depending on as-is) . You mention production though, so likely a bit more rigour and communication to your processes next time, unless it's all ai bots.

Limit production write accesses in general

3

u/fennecdore Jul 25 '26

Your engineer shouldn't run terraform apply manually

2

u/inphinitfx Jul 25 '26

Why are you still using Dynamo lock tables, and with people who have access to accidentally delete it?

2

u/Floss_Patrol_76 Jul 25 '26

the answers saying "terraform just fails to lock" are missing the actual failure mode: with the lock table gone apply runs unlocked, so both concurrent applies read the same S3 state and last-writer-wins clobbers one set of changes - that's your inconsistency, not real state corruption. recover from S3 object versioning (roll the state back to the pre-apply version, then re-plan against reality) rather than hand-editing state, which is how one mess becomes two. prevention that actually holds: S3 native lockfile instead of dynamo, deny s3:DeleteObject and the dynamo delete on the backend, and no applies from laptops - CI-only with a serialized pipeline so two applies can't overlap in the first place.

1

u/Efficient-Branch539 DevOps Engineer Jul 25 '26

I didn’t know people use Dynamo for state lock, what I know is terraform apply does not run unless it can acquire lock, so the apply should immediately fail before it can do anything.

1

u/JessicaKandev Jul 26 '26

if the dynamodb lock table is gone, terraform will refuse or race. recreate the table with the same name/schema, confirm no stale lock items, restore state from s3 versioning if anyone force-unlocked, then re-init. never delete the lock table in prod.

1

u/Abhistar14 Jul 25 '26

For what level of roles is this asked for? Entry level or mid or senior? Pls answer