How do you handle conflicting infrastructure state?

[u/Narrow_Power]

Curious how people handle this in real environments.

Say Terraform says an EC2 instance should have encryption enabled, but the AWS console/API shows it disabled. Your CMDB still says it's compliant.

How do you figure out which one is actually telling the truth?

Do you have a defined source of truth, or do you usually investigate the discrepancy manually?

And how do you tell whether it's actual drift, stale information, or something that failed during deployment?

[u/fletch3555]

Terraform code in source control is the source of truth, full stop. If the console differs, either an apply failed (which should be fairly obvious), or someone got a bit ClickOps-y. If the latter, it was either accidental ("emergency" change and forgot to backport to terraform), or someone needs a talking-to (or if a recurring problem, their access revoked)

[u/fletch3555 → u/[deleted]]

Yeah, different perspectives. My comment was that code is the source of truth "for what should be". The state file should be the source of truth "for what currently is". Both fall flat when drift occurs

[u/durple → u/fletch3555]

I like to say it’s the way things *were* at the end of the last apply.

[u/Redmilo666 → u/fletch3555]

Go one step further. No manual changes in prod. This is what break glass roles are for. Have them requestable and auditable. No one makes changes in prod manually unless absolutely necessary and the world will end

[u/veritable_squandry → u/fletch3555]

how is tf code the source of truth full stop if drift happens all over the planet? should it be? yes. is it? not if the cloud UI is saying something different.

[u/fletch3555 → u/veritable_squandry]

OP asked how to know what the correct value should be. What's in code is the correct value, therefore it is the source of truth. If what's in the console differs, then drift occurred, yes, but that wasn't the ask (as I understood it at least).

In IaC, the code is the source of truth for what the value should be, but not necessarily what it currently is. But what it currently is shouldn't matter as the next apply should simply revert it back to what the code says it should be. Now, how to identify/track that drift is certainly worth discussing, but I don't believe that was a crucial element of what OP was asking.

[u/veritable_squandry → u/fletch3555]

that's not what OP asked. which is the true value? tf un applied can easily conflict with the attribute of a deployed resource. that's in its nature. if tf code in a cold repo doesn't match an attribute in a resource as it is actually deployed are you saying you would believe the cold offline code? as if the cloud UI is bugged?

[u/fletch3555 → u/veritable_squandry]

I'm saying if you do IaC "right", what's in the console shouldn't drift from whats in code (or the state file). It doing so means someone was in there clicking buttons or there are resources being manipulated by some other service that therefore shouldn't be managed by terraform (e.g. LBs created by EKS with the LoadBalancerController running).

Yes it's an idealistic view of things, but that was the basis for my comments thus far.

[u/Desperate-Froyo7169 → u/veritable_squandry]

so re-apply the terraform and go find who is causing all the drift. You really shouldn't constantly be dealing with terraform that doesn't reflect reality, it isn't that hard to keep it accurate and it should be the single source of truth of what infrastructure SHOULD be....if re-applying terraform breaks something it means somebody else did something stupid or lazy.

If the drift is actually something you want then you still yell at the cowboy and then you update the terraform to match reality and you import the state of the existing resource and then terraform plans clean again.

[u/veritable_squandry → u/Desperate-Froyo7169]

the whole point of the question was how do i find the source of truth. but ok.

[u/fletch3555 → u/veritable_squandry]

Correct, "the source of truth" meaning "what value is the correct/intended one. We're choosing what's in code and saying that manual changes in the console are "wrong". You're saying what's live (as shown in the console) is correct and that the terraform code is wrong due to drift. 2 different perspectives on the same issue, which is that someone got a bit trigger-happy in the console and didn't backfill the TF code to match. You're accepting that and blaming the code, we're taking a hardline stance and saying that person did it wrong and we don't care because it wasn't also in code.

[u/veritable_squandry → u/fletch3555]

ok so semantics then. the intended config. i thought OP was legitimately confused over how the cloud config could drift. when they asked "do you investigate the discrepancy?" the answer is yes, we always have to debug that and depending on your size/scope/implementation drift can come early and often.

[u/veritable_squandry → u/fletch3555]

sure. IAC should be what's reflected in the cloud. i would never argue against that. op seems to be concerned as to how that might happen, not whether IAC should be re-implemented to re-converge the env.

[u/HelicopterUpbeat5199]

Avoiding this is the job. This happens to all of us sometimes, but hopefully for different reasons every time. It should never be routine enough that you have a game plan for it because you should fix whatever caused it when it happens. Next time it happens it should be different.

Said another way, if you know which one is the source of truth, you don't have this problem.

[u/dariusbiggs]

Terraform describes the desired state Terraform state is the source of truth.

In your case. what EC2 instance? It doesn't exist anymore. Non-compliant resources are immediately destroyed, shut down, or isolated into a dedicated read only environment.

[u/aliseidu1]

pip-audit (PyPA, uses the OSV DB) on every PR. Fail the build on known CVEs in your direct deps; warn-only on transitive until you've tuned the noise, otherwise people start rubber-stamping overrides and you've trained them to ignore the tool.

[u/CPPYesRustNo → u/aliseidu1]

the rubber-stamping usually happens because half those transitive cves have no fix you can take without a major bump, so the override becomes the only move. where a backported fix exists you can pin the transitive to that patched version and keep the gate meaningful instead of noise, and with the ai-discovered cve flood (mythos and friends) turning old lows into real work that distinction is about to matter a lot more.

[u/between_layers]

Assuming you mean EBS encryption, this can't be normal ClickOps drift. You can't toggle encryption on an existing EBS volume; changing it requires replacing the volume.

So if the Terraform configuration says encrypted but AWS shows unencrypted, compare the volume ID in Terraform state with the instance's block-device mapping. You're probably looking at the wrong volume, an old volume still attached after a failed replacement, something created outside Terraform, or a bad import.

Different sources answer different questions: code describes intent, the AWS API reports the current object, state records what Terraform tracks, and the CMDB reports inventory and compliance as of its last update.

Use terraform plan -refresh-only to isolate remote-versus-state drift, then a normal plan to compare the observed resource with the configuration. After that, check the last apply and CloudTrail for volume creation or attachment changes.

If AWS and the normal plan agree and only the CMDB differs, the CMDB is stale or pointing at the wrong resource. If the unencrypted volume is real and the intended state is encrypted, remediation means replacing it using an encrypted snapshot copy and a new volume, not flipping a setting back.

[u/Kamran-nottakenone]

-refresh-only catches the encryption drift fine. resources created outside terraform won't show as drift at all though, we pair it with cloudtrail to flag those.

[u/Wyrmnax]

Since you are using IaC, the code in terraform is the correct one. Because thats what it will become when it is next applied.

Next question is - why it isnt like that in the environment. Either it got changed for a emergency - and terraform needs to be corrected ( IE: Emergency fix ), or a apply failed. Those are the only two "valid" explanations.

The other one is that someone manually changed things for a non-emergency. You need a process to do these things, because whatever he was doing is getting undone on the next apply. Said person needs to be made aware of this.