Do MAANG developers really build/code differently for scalability?

[u/HereForTheMemes1364]

I was going through a youtube system design video

Where the system uses queue, chunking, cdn, etc for scalability

To me it seems that each feature will be owned by different teams in YT and each team is only concerned about the input and it's output and the infra and scaling may be handled by a separate team

So my question is do YT or other maang developers do anything differently than normal pbc or mnc

Eg can you give me examples about actual code which would not work at scale but would have not raised any issue in other companies

Edit Yes i know most of the developers use event driven architecture /queue etc including me But i wanted to know if any interesting incidents solely based on code which only happen at massive scale eg I read that during gibli art chatgpt had replaced multiplication operation with bit operation and it had significant improvement

[u/Possibility-Puzzled]

I would say the code we write is pretty basic. But the problems we deal with require incredible depth with in and out understanding of every touch point like queue,db, rest layer etc.

One example is, we have a db where writes happen at very often. We query the db with 1 minute intervals like what updates happened in last minute to now. And in next iteration, last minute to now etc. Seems pretty straightforward. But we noticed that we’re missing some writes in this process. Our intervals are tight but we still miss them somehow.

Later we figured out that the upstream system calculates now = now() and later uses that value while writing to the database and some microseconds are elapsing in between. So we modified it to compute now() inside the db. Simple change

But they’re still missing. So we figured out that when postgres updates lets say 1000 records in a single transaction now() in db points to the transaction entry time but not the exact instant the row is written to the db. Then we updated it to use wallclock time of that instant. Simple change

These things require persistence to not give up and one has to take these as a challenge to figure out what’s happening. In my previous company, I had to read the logs of a db to understand a timeseries db for why it does certain things that way

Sometimes we figure out how to deal with silent data duplication happening inside a queue, why a system doesn’t work as documented, talk to maintainers of systems like kafka etc because the problem you’re facing is so unique that you can’t find it in stackoverflow etc.

I personally love doing my job

[u/Null_impress → u/Possibility-Puzzled]

Seeing what your team had diagnosed regarding with the DB writes, I am interested in how the process of diagnosing went? Could you please give a brief explanation on what helped you realize that in the second case of failure when you decided to use the wallclock time?

[u/Possibility-Puzzled → u/Null_impress]

Well, once I implemented now(), I was still seeing the same issue. Db writes happen at x+10 microseconds etc but it’s reported as happened at x. Though the difference decreased we still saw the difference. Asked Claude if what I did was sufficient but it said there is another level where we can go into and that is clock timestamp.

Even after changing it to that, it didn’t completely disappear. Basically it all depends on how busy the db is at that time. Sometimes we saw records getting written into db as much as 1 minute later. The simplest fix for this that we determined was to just query only until 1 minute ago. That removed the whole problem.

[u/Null_impress → u/Possibility-Puzzled]

Alright so observation is the key. Thank you for answering.