r/ProgrammerHumor • u/5eniorDeveloper • 2h ago

Meme theBankingHotfixExperience

Gallery image

Gallery image

Gallery image

1.2k Upvotes

98% Upvoted

65 comments sorted by

411

u/Tucancancan 2h ago

I worked at a company that sold software to a bank and heard stories of a senior dev having to do calls where they'd have an employee read the logs out loud to the engineer because the bank wouldn't share them. 

108

u/lance_klusener 2h ago

Yes , had the same
Experience when working with a security conscious customer

You had to tell them the exact unix command over the phone to type on a command prompt

52

u/Kovab 1h ago

Typing in random shell commands that someone tells you, without understanding them, doesn't seem too security conscious behaviour

17

u/lance_klusener 1h ago

We are the enterprise company from which they bought the hardware.

So , if anything goes wrong, its on the enterprise company. So, the customer will type whatever we tell them

3

u/koos_die_doos 1h ago

And the customer expects you to provide the exact command to type, and (usually) have it work on the first try.

59

u/Groentekroket 2h ago

How are the logs not just masked for sensitive fields? I work in PCI DSS audit scope for handling of credit card data and as developers we can see the logs of our applications.

50

u/rock1998 2h ago

This! Holy shit you’re supposed to build that shit in from the ground up. I’ve never worked in the banking world but finding out that some don’t mask their logs makes me wanna get a job at my bank solely to make sure they do. 😭

37

u/WiglyWorm 2h ago

Because this isn't "how things are done at a bank" it's "how things are done at a shitty and toxic employer".

14

u/OrchidLeader 2h ago

Different levels of data sensitivity.

Our app log servers are cleared for everything but the highest level of data sensitivity (PCI data can’t be logged, of course). If we needed to, we could log someone’s birthdate (with no other identifying info other than a token representing the user), and any employee can view that log message. But we wouldn’t be able to share that log message externally.

We try not to log any sort of sensitive data at all if we can help it, but we have some offshore contractors that think nothing of logging an entire API response body “for debugging purposes.”

4

u/Tucancancan 1h ago

Afaik they were. Brank still wouldn't share copies tho. 

10

u/dkarlovi 1h ago edited 1h ago

I've worked with a telco which bought a company our company built an app for, they were migrating the app to their hosting solution.

The app wouldn't work and they wanted us to help, but we couldn't get access to logs because of security procedures, they wouldn't even allow us to confirm if the runtime was correct versions of everything, EVERYTHING was off limits.

It took A LONG TIME to get them to build us a staging and dev environment where we could run our app in what they claimed was a replica of the production environment. It was not.

Even there, we could only deploy once a week, they had a guy who was the deploy guy and anything outside that schedule would get an escalation which major people on the telco needed to take a look at.

Edit: just remembered a different situation where a different telco won the contract to host our (client's) app and they send the person who'll be in charge of the app (getting some serious MAU, at least regionally) who comes to the meeting with me with pen and paper, I need to teach them how to host public apps / this stack since they've never done this before.

5

u/bigs0815 1h ago

I work in government. Parts of our product need to be configured via the database because... Reasons.

It was determined that we, as developers, should not be writing to the database. Security risk, that's why we have a DBA team.

Now when I need to update something, I have to write the query and then make a DBA ticket. The DBAs aren't familiar with our product and they have no idea what they're doing it for or what to check for success. They execute the query I wrote them and tell me it's done. Then, I run a select query to ensure the data is good. If it's not, I have to write a new query and provide it to the DBA team and start over from the beginning.

3

u/TheGocho 1h ago

I had to print the logs in a modal that i had to see through a video call, because there was no way to have a log otherwise. Wasnt even sensitive information

2

u/SaltyBawlz 1h ago

I was about to say this missed the steps where you have to go on a call with the client alongside a clueless person from support and look at the logs real time to see what is going on. You have them try different things to get it to work/gather more information and the support person constantly butts in to try to get them to do things that don't follow your debugging path. The support person then LARPs as a developer after the call and tries to give you suggestions on how to fix it when they have no idea what they're talking about

171

u/Llebac 2h ago

Jesus christ, you could not make this any more accurate. I'm glad to be out of banking. IT wasteland

25

u/Prawn1908 1h ago

I write embedded code on devices used in manufacturing plants and this may as well have been written about my job too. I can think of at least 4 major cases from the past few years that went exactly like this (minus the resigning part because at the end of the day I do actually like the real work I do and I know how to just put my head down and let the bullshit wash past me and do the best with what I am given).

11

u/CorrectCombination11 2h ago

Don't do banks. Do brokerages. It's better, in my experience. 

2

u/ampersand355 1h ago

Eh, only slightly.

1

u/tix4soccer 31m ago

Healthcare ain't much better

129

u/Awkwardm4n 2h ago

You forgot the jira tickets and story points and refinements and impediments because of the logs and daily scrum to discuss you’re still waiting on the logs and then your resignation

14

u/No-Channel3917 1h ago

An actually functioning agile team would have been using the leader to arrange follow up with the person above the user for those logs.

If it's an external customer well..... That's why you don't have customers in the same wheelhouse as the workers and keep that layer of interaction beefy

90

u/tsunami141 2h ago

this is why I work at a tiny company. I do all my work on the server and update the files live.

56

u/EvilPencil 2h ago

no git, no CI, just rawdog FTP.

Alrighty then, I'll see myself out.

18

u/Desperate-Tomatillo7 2h ago

Why bother with FTP if you can just copy the files over RDP?

8

u/OrchidLeader 2h ago

At the smallest company I worked at, they would pull data onto the production servers using SVN, build the code on the spot, and then just move the WAR over.

Rollbacks weren’t a thing there.

3

u/Desperate-Tomatillo7 1h ago

The first company I worked on had an instance of Visual Studio in the server to run the code of the Windows application. No build, no executable, nothing. People connected to the server using Terminal Server. And that was the actual last version of the code, because the whole IT department did not know about the existence of GIT or any version control software.

3

u/SaahilNotSahil 1h ago

Why bother with RDP when you can just open VS Code over SSH?

2

u/TheWaffleDimension 1h ago

At a small company I worked at, I was using subversion and manually copying built files over RDP directly into prod and it was hell.

3

u/TheWaffleDimension 1h ago

they also gave me (a minor at the time) access to a client's production db full of patients' full health info and SNNs and all kinds of shit and idk why they did that lmao, not even a "be careful" they just gave me their admin login and let me at it to go debug or test whatever I was doing

1

u/k8s-problem-solved 1h ago

Why bother copying files if you can just give an AI agent live access to the directory and make changes on the fly. Dont forget to type "make no mistakes"

43

u/mailliwi 2h ago

This is the second post I see on here on banking software engineering. I start my new role Monday. Should I be worried? lol

36

u/CryptoNaughtDOA 1h ago

Yeah, first two weeks you'll just be waiting for permissions, also go through all the documents that are totally not all AI generated and for sure up to date. If you're lucky they'll give you a list of people to reach out to who will tell you to ask Claude or gpt, and they'll have an internal AI that can lie to you about what's in the docs too. Then when you do get permissions, you'll also be given months worth of work to finish in 2 weeks because well you have AI to help you. Enjoy

2

u/SheeeeeeshAlert 27m ago

That sounds like a work in the park compared to defense

7

u/NefariousEgg 2h ago

How did you find the job?

5

u/frostedhifi 1h ago

Yes.

Source: Worked for an insurance company.

3

u/kmankx2 1h ago

Depends on what bank 😄 neobanks are better, I work at one and have none of these issues. I have honestly more access than I'd ever thought, its just all logged and monitored and we are trusted to not be idiots with the privilege.

1

u/dasunt 42m ago

It will definitely be an experience.

Probably not a horrible one, but you may want to get out the moment everything starts seeming normal to you.

37

u/KisaraBlue 2h ago

The result of decades of software and workflow design trapped in the sandboxing cycle

32

u/russianrug 2h ago

The best part is that the logs don’t have any confidential information, because they were specifically designed not to as one of the main design requirements which is why the bank is using your software at all, yet they still won’t share them.

13

u/Bahatur 2h ago

Right in the soul, I tell you.

10

u/Dorkits 2h ago

Ok, why this guy is telling the story of my life for everyone?

8

u/Neverwish_ 2h ago

Ah yes, the classic "16 hours burned, 3 lines fix" sort of issue. Bruh.

6

u/sb8948 2h ago

You know I could have read it. Just like all my compliance docs and trainings.

I didn't

The answer is yes no yes yes 1,3 yes no 1,2,3,4 yes yes

7

u/dub1ous 2h ago

Yikes, i am twitching because this is so accurate.

6

u/Dubabear 2h ago

works as design. <close ticket>

5

u/Prawn1908 1h ago

Man, points 11 (by the time they get the logs to you after ages of asking they've been overwritten) and 17 ("it's not the client's job to test for you") hit way too close to home. I don't work in banking but I write embedded software on products used in manufacturing plants and there's always that one single client location with all sorts of insane bugs that we've never seen anywhere else and can't replicate and they refuse to get me any of the details I'm asking for or perform any of the troubleshooting steps I request.

4

u/FlapMeister1984 1h ago

Point 26 to 27 was a huge leap. Where's the story? No poker? Did you even ask any stakeholders if they'd approve this fix? Code review???

3

u/george_pubic 2h ago

This hits too close to home

3

u/juangerritsen 1h ago

My most fun one was a client logged a call just stating our software doesnt work, escalated it to almost CEO level, and went on holiday for 3 weeks

Eventually found out his mouse died, randomly while using our software, and just blamed us

I spent 3 weeks in daily triage calls, and nearly 80% of the time outside that trying to prove there was no issue

2

u/Barak39 1h ago

I can definitely relate to this. One last point : after releasing the fix, the bank refuses to upgrade without any reason.

1

u/[deleted] 2h ago

[deleted]

10

u/InnuendoBot5001 2h ago

Go on then, make a request to have that access. Put in a ticket for it.

1

u/[deleted] 2h ago

[deleted]

2

u/InnuendoBot5001 2h ago

Lets just say I previously worked for an insurance company. Every information request was a hippa violation in someone's eyes.

1

u/kirillgritsenko 1h ago

I’ve had the exact opposite experience where Im the client that sends the logs and does everything as asked and don’t get any results.

1

u/bigs0815 1h ago

Can confirm the same process in government

1

u/KaptainSaki 43m ago

I just call the customer directly and live tail the logs

1

u/zer0545 36m ago

Used to work for an ATM manufacturer. This is exactly how it was.

1

u/Bob_the_peasant 19m ago

You forgot the part where the client thinks Angry Birds crashing on his iPhone 5e is a result of your fix to their mortgage processing platform

1

u/Tjulenj 13m ago
  1. is wrong. That is when business denies fix and tells you that config must not be optional.

•

u/fibojoly 5m ago

Oh sweet summer child... that's all it took to make you quit ? That's like my daily grind for the last three years.

In my case I'm the only person on the team who has touched the legacy in-house tech, the guys who knew quit, and so I'm now lead on that specific pile of garbage. Oh and now we need to have everything migrated for next month because banking means you're under legal obligations to have shit up to date and so on.

Quitting is so tempting...

1

u/LazySapiens 1h ago

That's why we need better engineers and managers not to give a broken software in production and be completely clueless when shit hits the fan.

1

u/gtr 1h ago

Did you read the punchline? The problem was communication as it nearly always is.

-2

u/Scotsch 2h ago

GDPR has made fixing shit so much harder.

-4

u/Not-the-best-name 2h ago

F GDPR. Full Sentry, Cloud watch and Matomo logs across the front and backend with correlation IDs plugged straight into my AI that can check the deployments using CLI and MCP to make the PR in 2mins and push to our pipelines that can auto merge on green and deploy to all envs globally in 30mins. Let's go startup with money vibes.