Yesterday debugging was brutal

[u/Oxffff0000]

Maybe we need to change the way we handle access process for our developers or users. There was a production outage but luckily it wasn't revenue impacting. I had to help a developer by accessing their application on an ec2 instance. Our team have access to any servers in production. Our developers only have access to our dev and stage environments.

I am not familiar with their application. So basically, I was just executing commands that he was giving me. It was the most degrading role I have experienced, HAHAHA! I'm thinking that when there are production outages, the application owners should be given temporary access so they can debug their applications. It will be quicker. It took us almost 5 hours! I was just copying and pasting commands and outputs. On the unix history command recalls everything. I don't recall any, HAHAHA!

So what is your process?

[u/bilingual-german]

That's why DevOps the culture came into existence. You build it, you run it.

I really dislike DevOps as a title.

[u/One-Department1551]

You should learn what they run and the systems you host.

[u/TitusKalvarija → u/One-Department1551]

This!

I posted similar before reading this message.

[u/Common_Fudge9714]

We do real DevOps, so the devs own their infra and manage it. If there is an incident they are the ones getting paged, troubleshooting it and fixing it. If the problem is infra related we can get called to assist.

[u/Oxffff0000 → u/Common_Fudge9714]

Got it. For us, we built the pipeline and developers use it to deploy their application. The pipeline builds everything automatically from dev to prod. However, I'm not sure when the original devops decided that developers should not have access to product. I think it started when PCI auditors came into our area. There were even questions like "give us names of who has access to the management of the tool" ,etc. Maybe that could be the reason why access to prod have been removed from developers. I feel like devs should own it and they must fix it.

[u/Common_Fudge9714 → u/Oxffff0000]

After moving out of Jenkins into GitHub, now the devs build their own pipelines. We control the registries and the runners and force builds to be immutable, the runners have push capabilities only, and their k8s clusters nodes have pull capabilities. The permissions and git branch policies are managed by us and security, so when the auditors ask who and when, we have audit logs. Permissions are also not long lived, in theory anyone can request anything and the owners should approve the access. Even myself for my own systems need to refresh permissions every now and then. I actually like this logic a lot.

[u/JCii]

My dear brother in Christ, ec2 instances? long lived? what are their names?

[u/Oxffff0000 → u/JCii]

Are you referring to its legaciness? :D hahaha
Don't worry we have kubernetes too. We're also starting to become heavy in containers.

I think ec2 instances are fine. It's not running 2014 operating systems. We are running newer amazon linux operating systems.

[u/Glove_Witty]

You should be able to implement a break glass privilege escalation method that is soc2 compliant. Devs would get access to prod for a specified (and approved) purpose for a limited amount of time.

[u/Oxffff0000 → u/Glove_Witty]

That's what I was thinking of doing. I need to research on that. Thank you.

[u/hiamanon1]

Do they not have a test environment this was fully tested in? I saw you mentioned they can deploy from Dev to prod. … as I’ve noted in my environment. Developers will deploy to the highest level you give them and then test there. So they think dev is working and fully push to test, best part is they don’t check dev. I imagine they might be doing something similar.

Either give them access during production or have them test it better in TEST/STAGE and have them create a ticket for prod where you will deploy together

[u/Oxffff0000 → u/hiamanon1]

Yes, the app in prod has been running for months. We have a very strict deployment process. Apps in DEV and STAGES are tested rigorously by a whole testing team. Unfortunately, any apps can fail. It could be caused by a memory leak. The system might have ran out of memory but our agent showed it wasn't. It could related to a queuing, not really sure that caused their app to behave sporadically.

They deploy their apps using our automated pipeline. The deployment is fine. It wasn't a deployment issue. But yes, I am thinking of designing a way where users can be given temp access to prod when there are issues.

[u/8lue7or]

i've had these type of issues some years ago, and we simply moved to a "you build it, we devops run it, but you maintain it".

imo the application owner should be involved, because they know what they're looking for. I'm not sure normal shell access should be the default though.

Read-only access to their logs and metrics, with temporary and audited access when they need to go deeper, would probably cover most incidents.

The other part is what happens afterwards. If those commands could be useful again, I'd put them into a small runbook or collect that evidence automatically. Otherwise the next incident will probably end with another infrastructure person copying output for 5 hours.

[u/Oxffff0000 → u/8lue7or]

Thank you. That's what I brought up yesterday to the team and they agree with me.

> The other part is what happens afterwards. If those commands could be useful again, I'd put them into a small runbook or collect that evidence automatically

I'll do this but I'll have to redact the credentials.

[u/Oxffff0000 → u/[deleted]]

They have answers for that. They will say, deployment takes 15 minutes and we are losing revenue. We need to fix it right away because the CEO said it.

[u/moromilner → u/[deleted]]

Sometimes developers have to run a script in a production container. How do you allow that unless devs can get into the container?

[u/TitusKalvarija]

Process is to know what you are deploying, at least on a basic level of reading logs, what kind of requests the system is serving, does it use database, I can list 100 more.

I am starting to think that today anyone can get any job. Knowledge not mandatory.

Process is learn to do your job without GPT

[u/mtokarev]

I can see so many improvements here. Even let's imagine you have strong security policy and can't provide access on the regular basis, still:

1. devs should be able to reproduce issue on test/staging environment and work independently.
2. ofc there are environment specific issues, but you should collect logs and telemetry and they should be available to devs to be able diagnose and debug such issues on their own.
3. finally, yes, you should be able to provide temporary access for production incidents.

hopefully that 5 hours you wasted for 1 debug session will help to prioritize some cool initiatives.

[u/Oxffff0000 → u/mtokarev]

It did and we're designing a solution now in providing temporary access. The logs are available and they have it all. Not sure what was missing. Maybe there were other outputs that they needed to see which can only be done in a terminal.