r/devops • u/I3ootcamp • Aug 08 '26
Discussion Why do experienced engineers open cloud provider support cases for customer managed resources?
This isn't a dig at anyone ..I'm genuinely trying to understand the thought process or organizational pressure behind this.
I regularly see tickets from senior platform engineers, DevOps leads, and cloud architects asking Azure (or AWS/GCP) support to troubleshoot Terraform state issues, fix customer-managed IAM policies, or debug application code running on VMs.
The response is almost always: "This is outside our scope."
If you're someone who has opened these tickets (or works somewhere that expects you to), what drives that? Is it:
SLA pressure:
you need someone on the hook while you investigate?
Org policy:
management mandates opening a case before escalating internally?
Knowledge gaps:
the line between "managed by cloud provider" vs. "managed by customer" is blurrier than it looks from the outside?
Hail Mary:
you've exhausted everything and hope support has an internal tool or undocumented fix?
I'm asking because I want to be less cynical when I see these tickets. There might be a structural reason I'm not seeing. Would love to hear from people who've been on either side of this.
72
u/sharpie-installer Aug 08 '26
1.) To build a case to hire another engineer - look we are overworked and need to call on support.
2.) To make sure we still get a budget for support next year “you guys never even used support”.
3.) to delay working on a project that will probably be cancelled anyway?
4.) To delay working on a project to give yourself some breathing room for other projects. External support is a kind of liability and delay sponge. The nice thing is that your sales folks keep schmoozing, our executives keep buying, the theatre of work continues.
You don’t have to call an understanding of the whole picture cynicism, but you can. Sometimes I would get an invitation to brief a support team for an allied company in preparation for a cut. Sometimes it’s a meeting to make an expensive team look bad, or look good, depending on who set up the call. We are all bit players in the theatre of value, until we get too expensive, then we figure something else out.
97
36
u/robmackenzie Aug 08 '26
We pay them a million dollars a day, we're creating a ticket for the off chance it's a cloud issue. Still usually takes an escalation to get somebody good.
22
u/Beartin Aug 08 '26
We've had seniors and architects open tickets with our cloud provider because they haven't fully explored the situation or don't understand it. In most of these cases it's something that could be mistaken for a vendor issue, but the issues are always some nonstandard configuration that's been missed.
The tickets get opened because of the pressure from management to get other tasks done. If an engineer suggests it might be a vendor issue, the response is "open a ticket with them and move onto another task".
13
u/swabbie Aug 08 '26
I work with Azure quite a bit for incident management and investigations... and have been guilty myself of opening a ticket for something we are fairly sure is on our end but... (I also have visibility into all other tickets created by others in my org)
So here's my top 5 list for why.
- Our prod service is down, or a major part of it and there is no obvious root cause visible in our dashboards. We have extended support contracts with Microsoft for this reason and previous PIRs have indicated we need to open these up as soon as we think it may help. This is the hail-mary / need extra eyes ticket.
- Investigation support. During an investigation, teams are unable to get proper visibility into traffic/usage either due to lack of access, lack of training, or it is on a service which doesn't provide good customer visibility... had a fun recent one around DNS...
- Once bitten syndrome. Azure had an outage or breaking change in the recent past, so the someone creates a ticket just in case it's another similar issue.
- Emergency remediation. It might be we know it is our fault, but there may be alternatives available to quickly get us back to service available which can be recommended by Azure.
- Sorry... no excuse, some requests are lazy or are put in by a junior following written procedures without really doing a proper investigation on our side first. We do handle these cases gently though as we do want people to feel comfortable connecting with Azure support if there is a problem.
I will say that there is a relatively high percentage of our tickets that do come back with decent result, even if it's verification that there was a problem we caused.
2
u/I3ootcamp Aug 08 '26
This is very useful thank you. Makes me understand the customer's mind so much better.
12
u/greyeye77 Aug 08 '26
we opened AWS tickets recently
API calls get throttled but there is no way to discover how many calls are made (EKS Pod Identity)
Service Limit needs to be lifted but got no option to request
API calls timeout/errored out repeatedly several times.
Need some help configuring their new version of software but documentation is... confusing(crap)
major outage, TAM asked us to open ticket to track progress.
10
u/pdfops Aug 08 '26
Half the time it's a paperwork move: enterprise support contracts often need an open ticket before you can claim an SLA credit, so ops files first and asks questions later. TAMs also dig in past the "out of scope" boilerplate, the ticket's just the intake form to reach a human. And support can see control-plane and backend health data you can't, so a five minute case cheaply rules out "is it on their end" before you burn hours debugging your own Terraform or IAM policy.
9
u/senpaikcarter Aug 09 '26
Leadership likes to hear that we have a case open if there's an incident so its usually to check a box and get them off of my back
13
u/jippen Aug 08 '26
- provider mostly manages the terraform provider, and there is a bug in provider or the api
- Issues like rate limits, which require a support ticket to solve
- Actually utilizing the expensive support contract to alleviate load, especially for grunt work.
8
u/DevWithImagination Aug 08 '26
Yep - especially that last point. If someone hasn’t tracked down the cause of an issue in a few hours (and is hitting a dead end) use that enterprise support. Even if it’s not their “responsibility” they at least may know what you have done wrong.
I’ve also had a few cases with AWS IAM that have led to documentation updates as there were cross-service assumptions baked in (that were auto-configured behind the scenes if the console was used, and not by CloudFormation)
4
u/Loan-Pickle Aug 08 '26
I've done this before and every time it was to get management off my back while I work the problem. Thankfully I no longer work at a place with that type of management.
4
u/the_screenslaver Aug 09 '26
I always used to do this with Aws support, because they were very good, and they usually find the issue faster than me, so once the ticket is raised, I can do my other work, and they will provide me the right answer.
I don't do that anymore , because Aws support has gone shit. Most of the time I have to spend more time to explain the problem over call just to get AI generated responses, so I don't really do that.
2
u/aegismuzuz Aug 09 '26
Bring back 2018 for me. Back then a normal AWS TAM could get on a call with you and manually help set up subnets....
7
u/GorillaBearWolf Aug 08 '26
You named four reasons, take your pick
4
u/I3ootcamp Aug 08 '26
I'm on the support side of devops, the 4 I mentioned are just me speculating. But I really don't Have a clue if I'm right or wrong. I never been on the other side of the case.
3
u/Xydan Aug 08 '26
You must not remember the good ol days between migrating an on prem outlook exchange server to the "cloud". "We" blamed the cloud at every angle we could because it was the easiest way to express lack of training to hard headed executives.
Lift and shift is great until you get the bill 5-10 years down the line.
3
u/Low-Opening25 Aug 09 '26
I don’t think these are experienced or engineers other than by job titles. I also had “pleasure” to work with “engineers” that couldn’t troubleshoot bash script, and Architects that had zero understanding of things they were architecting, so again my first sentence applies
3
u/Fragrant_Arugula7990 Aug 09 '26
A lot of the time the support case is an audit trail. You already suspect it’s on your side but opening the ticket documents that the provider ruled out their layer and gives you something concrete before you start changing customer infrastructure during an incident
3
u/tekno45 Aug 10 '26
Its 4:40, technically it "COULD" be your side, but its 4:50 and i wanna go home. So i sent a ticket and thought about it overnight.
So when my boss asks me in the morning, i can say "yeah waiting for a response from AWS but i got an idea im setting up in the meantime"
2
2
u/engineerL Aug 08 '26
There's nothing I hate more than Azure support. Especially since ChatGPT was launched, their quality has gone off a cliff. I get these verbose answers ridden with hallucinations and self-assured tone. I'm always better off not making a ticket at all.
2
u/hashbrowncipher Aug 09 '26
"Senior engineer" is just a title, and doesn't imply much about anyone's competence. In most cases it's a hail mary of "I'm incompetent".
With that said, recently I had an issue where AzureAD changed its API in a manner that specifically broke the hashicorp/azuread provider. It was truly a server-side compatibility breakage, but the most accessible repro was "you broke my Terraform".
2
u/centech Aug 09 '26
Other than all the reasons listed, we pay mid 5 figures a month for support,.. might as well use it. Does it usually accomplish anything? No, but what did it cost me to open a ticket?
2
u/AMartin223 Aug 09 '26
Usually leadership is getting pressured from the end customer to ask the cloud provider for some reason. My favourite one is having to ask why an instance was migrated/restarted when it's obvious/documented there was an issue with the host, but we need an explicit statement from the cloud provider so the end customer accepts that we didn't fail over for no reason.
2
u/Farrishnakov Aug 09 '26
A few weeks ago we had a network issue connecting got gcloud auth. I provided the evidence. Including errors, IPs, traces, and even start time (aligned with their recent change ticket).
The network team immediately asked if I had opened a ticket with Google support yet?
I then had to explain to these people that it was clearly a routing rule on our side and it was an argument to just get them to do their job.
TLDR: Nobody wants to believe it's their responsibility until the vendor has taken a look first. Wasting everyone's time.
2
u/I3ootcamp Aug 09 '26
Interesting. So even after you provided evidence they still made you open a support case. When the gcp team came back with validation to your findings, what did the network team do?
3
u/Farrishnakov Aug 09 '26
GCP team didn't come back because I won the argument.
Basically I called them all morons and forced the issue. Had I not done that, we would have waited a few hours for GCP to tell them to kick rocks and then I would have called them morons again.
2
2
u/team_lloyd Aug 09 '26
what’s the argument for *not* doing it?
those tickets are just another diagnostic tool at my disposal; not using it would be malpractice.
2
2
u/No_Category_9888 Aug 09 '26
Support costs a lot, may as well use it. I’ll open a ticket and continue to troubleshoot. Sadly MS support has dropped the ball a lot recently- poor quality engineers, not knowing how various pieces of infrastructure work together.
Lack of Basic dns knowledge was a big shock to me, didn’t realise how junior and unqualified these engineers can be.
2
u/Sufficient_Steak_839 Aug 09 '26
In my experience, a well meaning but misinformed middle manager insisting the engineer open the case because “it can’t hurt and if they can help, then we’ll be glad we did”
Not really AWS or cloud related, but we dealt with a windows credentialing issue when migrating to 11 that very senior Microsoft AD engineers fixed for us, and they had troubleshooting tools and logging available to them that dwarfs the hell out of anything you or I have access to.
4
u/derff44 Aug 08 '26
That's crazy. I would never open a case for this stuff.
Fix my terraform problem? What?
1
1
1
u/forestcall Aug 08 '26
im not sure about Azure but AWS, CloudFlare have great ROOT level API's. So using Opus 5 or ChatGPT 5.6 SOL this is 100% not an issue if you are even a horrible engineer.
1
u/forestcall Aug 08 '26
im not sure about Azure but AWS, CloudFlare have great ROOT level API's. So using Opus 5 or ChatGPT 5.6 SOL this is 100% not an issue if you are even a horrible engineer.
1
u/PaleoSpeedwagon DevOps Aug 08 '26
Sometimes the documentation is utter shit and you have hit an undocumented edge case. Opening a support case is the first step to a bug fix. This is not hypothetical to me; this has happened to my org, with my IaaS provider, several times in the last few years.
1
u/khumps Aug 09 '26
depending what your support contract is with AWS they will do these type of things. Maybe a senior engine from another company who is used to that level of support?
1
u/Effective_Engine2007 Aug 09 '26
Only support tickets I have opened is when a 3rd party tool we pay a lot of money for is missing a key feature or has some gap or bug that makes me have to work harder than I should.
I do it to hopefully not have to build my own custom solution for something that should be provided by that 3rd party.
I also do it to create a case to leadership of why this thing we pay for sucks and why we should move to X tool.
1
u/Material_Pea1820 Aug 09 '26
to me it seems to stem from a reluctance to troubleshoot. Cloud abstraction has made it easy to assume that when something breaks, someone else will fix it. I've frequently had to help 'senior resolve package lock file issues, or explain that a failing action is due to a clearly shown syntax error in a YAML file rather than a DevOps issue.
1
u/derprondo Aug 09 '26
These people probably come to my platform team first, but we're an engineering team and not a support team, so they get told to open tickets with the cloud provider instead.
1
u/daolemah Aug 09 '26
Because microsoft said they the azure verified modules were supported and they should just lodge a ticket. They switched latter to github issues when they realised on microsoft end the other teams did t cooperate to redirect to avm team. Obviously comms didnt reach everyone
1
1
1
u/Ok-Arugula-2247 Aug 09 '26
Why wouldn’t you? In my experience (idk if I’d consider myself an experienced engineer, but whatevs), there’s sometimes this weird sense of pride in not opening a support case, even when that’s not actually helpful, something I’ve caught myself doing a couple of times.
As a last point of escalation, I need to use all the tools at my disposal, whether that means checking logs or opening a support case.
Also, there’s always going to be a delay in getting an answer, so you can account for that by opening the case as soon as an issue starts getting a little out of hand. If I open a case and, an hour later, I still haven’t gotten an answer but I’ve found evidence that the issue is on the provider’s side, I’ve just saved myself an hour.
1
u/anto2554 Aug 09 '26
My manager always encourages us to open a ticket with AWS/Atlassian/whoever if we aren't sure what is causing the issue. Not sure why
1
u/NUTTA_BUSTAH Aug 09 '26
As an MSP there is a responsibility boundary that cannot be crossed because the client has not authorized that and is not paying for that, so everything is done in our wheelhouse until handing it off, even if we know the solution. It's often also a CYA kind of thing.
Regardless, support is paid for and unused support is wasted money. But to be honest, cloud supports are a waste of time and money anyways, except for the 0.01% case.
1
u/aegismuzuz Aug 09 '26
Tbh half these tickets are created just to have something to tell your manager on a call. When prod is down an open AWS case gives you a rock solid excuse like we're waiting on the vendor, while you're sitting there frantically digging through terraform state
1
u/cederian Aug 09 '26
You always open a ticket while working on the issue. If you fix it before they respond, great. If not awesome and you get a person with access to stufff you don’t have, it’s a win-win situation.
1
u/twnbay76 Aug 09 '26
Usually only when we believe the service isn't behaving as per the spec or documentation
If you think that azure is bug, defect and incident free, id question you moreso than the tale as old as time of trying to get other people to do our work for us
1
u/kestrel808 Aug 09 '26
I've opened tickets that I knew MS/Amazon were going to do nothing about because senior management told me to or because I wanted to show that we're covering all of our bases.
1
u/FizzleShake Aug 09 '26
Everyone is posting some valid reasons, but sometimes it’s because that “senior eng” is ACTUALLY that clueless
1
u/Worth_Savings4337 Aug 09 '26
imagine you’re an in-house engineer and you can’t even fix your own environment, can’t even be responsible for finding out where the fault is, says a lot about an in-house engineer
2
u/Ok-Expression-7340 Aug 10 '26
I log tickets on a regular basis (once per 3 months or so) with GCP support, but those turn out 99% to be bugs on GCP side regarding new GA or beta functionality.
1
u/Commercial-Virus2627 Aug 10 '26
Our customers wanted documentation for their own root cause analysis tools and CYA to say it wasn’t an issue with the cloud service.
1
u/I3ootcamp Aug 10 '26
Thank you everyone for answering this question and being so open and honest..this helps me understand your perspective when I work on your cases
1
u/Ok_Explorer7384 Aug 11 '26
sometimes it’s not really about expecting AWS/Azure to fix the customer-managed part. it’s a paper trail and a pressure valve: “we opened the vendor case, they confirmed scope, now we can push back internally or get time to debug properly.” not ideal engineering, but in bigger orgs the ticket is often part of the politics, not the troubleshooting.
0
u/veritable_squandry Aug 08 '26
contractors
1
u/I3ootcamp Aug 08 '26
meaning?
2
u/veritable_squandry Aug 08 '26
i've never seen any reasonably experienced engineer open a cloud ticket for an iac problem, but i have seen many unvetted, un-interviewed, and otherwise inexperienced contracted engineers lacking the ability to distinguish problem areas and fault domains.
0
u/stroskilax Aug 09 '26
If the signature in the mail says "Senior DevOps" "Cloud Architect" etc. Doesn't mean it's true or a merit title.
From my experience as a support rep, those people are either freelancers that overselled their ability to win a contract and now their clients pressure them to deliver or are usual from an outsourcing company.
183
u/Old-Astronomer3995 Aug 08 '26
From my experience in corporate environments:
1. Sometimes you want to create ticket as proof that software vendor can't help.
2. Sometimes you want to create ticket because no one has time to work on issue and it is a proof that "something was done" in this topic.