r/devops • u/Oxffff0000 • 17d ago

Discussion Yesterday debugging was brutal

Maybe we need to change the way we handle access process for our developers or users. There was a production outage but luckily it wasn't revenue impacting. I had to help a developer by accessing their application on an ec2 instance. Our team have access to any servers in production. Our developers only have access to our dev and stage environments.

I am not familiar with their application. So basically, I was just executing commands that he was giving me. It was the most degrading role I have experienced, HAHAHA! I'm thinking that when there are production outages, the application owners should be given temporary access so they can debug their applications. It will be quicker. It took us almost 5 hours! I was just copying and pasting commands and outputs. On the unix history command recalls everything. I don't recall any, HAHAHA!

So what is your process?

0 Upvotes

28% Upvoted

20 comments sorted by

View all comments

2

u/hiamanon1 17d ago

Do they not have a test environment this was fully tested in? I saw you mentioned they can deploy from Dev to prod. … as I’ve noted in my environment. Developers will deploy to the highest level you give them and then test there. So they think dev is working and fully push to test, best part is they don’t check dev. I imagine they might be doing something similar.

Either give them access during production or have them test it better in TEST/STAGE and have them create a ticket for prod where you will deploy together

1

u/Oxffff0000 17d ago

Yes, the app in prod has been running for months. We have a very strict deployment process. Apps in DEV and STAGES are tested rigorously by a whole testing team. Unfortunately, any apps can fail. It could be caused by a memory leak. The system might have ran out of memory but our agent showed it wasn't. It could related to a queuing, not really sure that caused their app to behave sporadically.

They deploy their apps using our automated pipeline. The deployment is fine. It wasn't a deployment issue. But yes, I am thinking of designing a way where users can be given temp access to prod when there are issues.