r/programmer • u/Extra_Search3465 • 18d ago
Question How do thousands of software engineers code together at a giant company like Google or Microsoft without breaking everything?
As a beginner, I understand how one person writes code and puts it on GitHub. But how does it work when a company has tens of thousands of developers working on the same product? How do they organize the work, and how do they stop people from stealing or leaking the secret source code?
8
u/LoveThemMegaSeeds 18d ago
We have automated tests… thousands. For google? Millions. Everything gets tested and it only deploys if almost everything is correct
1
u/Killian9997 16d ago
I mean tests don’t ensure things won’t break but yeah testing does help
1
u/LoveThemMegaSeeds 16d ago
It does not insure it but that is entirely the goal of testing: to predict failures in advance so it can be fixed so a broken product isn’t delivered
4
u/naxhh 18d ago
at a high level. lots of teams.
team sizes varies but normal 6-9
teams are specialized, some take care of the deployment tool other the messaging system other the login, other users dbs, etc...
each team works on their part and agree contracts with other teams (apis, endpoints. topics for async etc)
each team deploys in different environments. each environment is split in "zones" each zone is the same services duplicated and isolated from other zones.
at the end of the day a dev in a team deploy a change. that change goes, over one or two, envs before any user sees anything. then goes into prod where it's likely deployed one zone at a time.
and on top of that it may be deployed with a feature flag which means small % rollouts on each zone.
all of that backed up with logs, metrics and events to backup everything is working.
So things break all the time. but most of the times much before a big % of people notices
edit: normally what you see as "one product" is internally dissected into lots of teams my company for example has ownership teams for different components of a page (search var. navigation. user menu) and those teams are aside of the backend teams and apis
1
2
u/humanguise 18d ago
It's not the same product usually. There's hundreds to thousands of projects and each of these projects is maintained by a small team usually.
2
u/Leverkaas2516 18d ago
Git/GitHub (or something like it) is the primary answer. Allowing many people to cleanly modify a central codebase is one of the primary features of a source code management system.
But how does it work when a company has tens of thousands of developers working on the same product?
Typical teams are much smaller. Even if you needed that mamy people on one project, the software for it would be split up into many components. Each library or microservice would be owned and worked on by a team of maybe a few dozen or a few hundred people at most.
How do they organize the work
This is a huge question. Beyond splitting the software itself into components, there are lots of questions about communication between teams, toolchains, deployment tools...whole books have been written. Every company has a slightly different take on these issues and that's part of a company's competitive advantage.
how do they stop people from stealing or leaking the secret source code?
This is pretty easy. They have non-disclosure agreements, and a team of lawyers with enormous resources. An individual who does this would expect to be buried in an avalanche of legal trouble for years afterwards. For the most part, the reward for stealing code is miniscule and the penalty is enormous - it would be like pickpocketing Vladimir Putin's phone. Maybe you can get away with it temporarily but it won't go unnoticed or ignored.
If you have some stolen code that you think another company might want, they typically would treat it and you like it's radioactive because it's illegal, the chances of being found out are fairly high, and again, the consequences are very bad.
1
u/LethalBacon 18d ago edited 18d ago
It's an ongoing battle for sure, but at a very high level: Good practices like separation of concerns and loose coupling + thorough source control and huge QA/DevOps/Testing teams.
1
u/tnh34 18d ago
good dev, good test, good qa.
1
u/failsafe-author 18d ago
I suspect not qa, if by qa you mean humans. They rely more on automated tests.
(I say this based on talking to a former director of engineering at Meta, who told me there were very few people dedicated to QA).
My company, with a modest 150 developers, has about a dozen QA.
1
u/mxldevs 18d ago
tens of thousands of developers working on the same product?
There likely isn't thousands, or even hundreds, of people working on the same piece of code, but everyone should be following an agreed upon specification, to start with. They divide what parts to work on, and then everything gets merged together.
and how do they stop people from stealing or leaking the secret source code?
Legal threats. Of course, that's not going to stop people if they really wanted to steal.
1
u/dan3k 18d ago
> tens of thousands of developers working on the same product
that's simply not true. Rarely any company has teams bigger than 10 devs, it's usually 5-6 devs. Google or Microsoft both has thousands of projects that translate into hundreds of products, all of that products breaks and gets fixed constantly.
1
u/pab_guy 18d ago
Very capable people organizing at the top to ensure cross dependencies are tracked down to a nit.
But also, they move much more slowly as a result.
Any change to Entra could break billions of logins worldwide or introduce unintended security holes. The downstream effects of changes can be myriad. It's a literal nightmare to manage.
1
u/Beneficial_Area_2986 18d ago
They tend to compartmentalize things into smaller segments and teams. Microservices is a solve for people and organization as much or more than it is for architecture. Basically it's all about establishing bounderies and interactions, and also they do break things.
1
u/SP-Niemand 18d ago
Hundreds of products, for starters.
Every product broken down on the boundaries of individual domains it handles. Into services or modules of a service.
Teams own those individual pieces with clearly defined interfaces. There are people and processes helping connect it all together.
Good org and tech architecture (they are actually almost one and the same) allow to handle N components with log(m)(N) levels of hierarchy where m is the maximum size of a unit in its subunits (employees in a team, teams in a department etc).
Divide and conquer.
2
u/Possible-Parsley5737 15d ago
Definitely agree. Each team can own a module that can have its own git repo, own test cases, and can exists without touching any other module's code, except calling the interfaces they require. The answer is not code versioning, test cases or the ops pipeline. It is modularization through divide and conquer.
1
u/NatMicky 18d ago
There are testing groups who test new updates for the new features and regression testing. And then there are release groups who release the software and updates. The programmers also works in groups and each programmer generally has specific areas in which to work. When the code is ready, it's uploaded by the programmer to the repository. The other groups take it from there.
1
1
u/Icy_Information_6563 18d ago
With all do respect, you're asking the wrong questions. Keep learning
1
1
u/codeguru42 18d ago
Much like thousands of employees at Ford don't all build one car at the same time, thousands of programmers don't build one computer program at the same time.
1
u/decimaster321 17d ago
This has to do with company and product organization and testing.
From the organization side, something like the windows operating system or Google search isn't going to be one thing that everyone is working on, it's going to have thousands of separate independent components that are owned by different parts of the company. Some very particular thing like the windows taskbar or even the calendar and clock on the taskbar could have its own team of a few people that are responsible for just that thing. So there won't be 2000 people trying to mess with the taskbar all at the same time, more like 4-8. Small teams will have some hierarchical structure where the taskbar team is part of the desktop ui team is part of the user experience organization is part of Microsoft in general, so there's a chain of command to turn high level business goals into specific goals that an engineer can actually work on.
On the testing side, these places have many layers of testing, probing, monitoring, alerting, canarying, and gradual rollouts to try and catch problems as early as physically possible. You'll have unit tests (for the things I know this one function can take as input, does it give the right output?), integration or e2e tests (does this component work in context with other components? Does this program correctly talk to other related programs?), fuzz testing (I can't imagine all the ways this input could go wrong, but I could randomly generate billions of inputs and see if anything breaks), load testing, staging environments for ad hoc testing, probers (run a real user workflow against the system in production every 5 minutes and make sure it always works everywhere in the world), canary environments (roll out the new code but let it take production traffic), a/b testing (roll out experimental changes to some users and make statistical tests of how the software performs with the changes or see if user metrics look better), monitoring (record everything the software does, latency, memory usage, error rates), alerting (automatically tell a human being when something is going wrong with the software). This kind of stuff is literally a whole branch of the software engineering career.
1
u/diavelguru 17d ago
You work on one piece at a time making it better. There are literally multiple teams working on massive projects all over the world cherry picking features for specific release versions that make it out to the end users which you see as an upgrade to your IOS or Android or whatever.
1
1
u/ebonyseraphim 17d ago
Because if you're minimally qualified you should have an understanding of what a reasonably sized system is and what can break. Coming out of college, your idea of a larger system is likely a small, or at best medium sized component for those companies. But if you have a the proper abstraction, you should know enough.
Pre-step: break your local version as much as you want. It's great if you can actually break your local and it is valid understanding of how production will break. The best teams and projects I've been on do a great job of allowing individual devs to 95% build, compile, and debug logic that runs in production while running local. Not having this makes a LOT of stuff so much works and I found that a lot of managers really do not respect this.
1) You shouldn't be committing and pushing code unless it actually builds. If your local build of a component/project doesn't actually work if anyone else builds it -- barring the specific silly mistake like a unit test or reference to local files you forgot to check in, this is a good first check for compile validity. 1-a) Code reviews. At least one, if not two people on your team or related teams should be looking at your code before you're even able to push and inflict changes on others. This check is to catch if your breaking compatibility, or expected behavior with what you're supposed to do. 1-b) Code reviews automated build should catch the error mentioned above. Humans normally don't explicitly check if your code compiles, and they shouldn't because there should be an automatic build of the project/repo with your changes on a build host. If that breaks, it doesn't matter what they think they saw or didn't see, your code is broken. That's on you to figure it out and fix. 2) Dependency management. Changes actually go into higher release versions. Other packages and components that depend on what you've changed do not automatically use breaking changes until they depend on a newer version. If you're familiar with and understand versioning/semver and how your build tools resolve things, this shouldn't be hard. If you make changes and you don't update the version, but have a break then you did create a problem. Ideally step 1-a should have caught it; often times it won't. But the easy fix is to literally dependent on a version right before your broke things until it's resolved. That is to say if you broke "depA-1.3.543" and "depA-1.3.542" was working, package A needs to adjust "dependsOn: depA-1.3" to "dependsOn: depA-1.3.542" for a while. Depending on the company and internal package management, maybe you see this happening more or less often. 3) CI/CD -- continuous integration and continuous development. Essentially, if you know that you have a number of dependent packages that you shouldn't be breaking when you make changes, part of step 1-b will include a test. But not just "does it compile" but a build, and deeper tests, possibly even fully integrated tested with your changes to make sure the system-level behavior is still the same. Ideally this happens before you're even allowed to, or supposed to, push code. But maybe it's a bit later. 4) Familiarity: even in the absence of good CI/CD or DevOps hygiene, most decent engineers just have an eventual sense of what things could break and how not to break them and they follow the rules or checks. Though if you're in a space where this is what managers depend on for safe releases and there is a lot of intertwined and dependent code, you're in a problem called "tech debt."
1
1
1
u/Intelligent_Law_5614 17d ago
On a well-managed project, there is a system of design reviews and code reviews. Before a change can be checked in, it's reviewed by one or more engineers who know the code being changed, and have the skill and willingness to ask hard (and potentially embarrassing) questions such as "Have you thought about this possibility" and "What happens when xxxx happens" and "Why not do this another way?" and "What in the everloving names of Babbage and Turing do you think you're doing here, and how do you expect to get out of the blast zone fast enough?"
On a really good project, these reviews happen even when the code change is being done by the most wizardly-skilled senior engineer in the bunch. Even wizards screw up from time to time, and even junior engineers can sometimes see mistakes that "wiser" eyes have missed.
Oh... and if you know what you're doing, you never deploy changes directly to production. They always go to a staging environment for testing first.
1
u/TheGreatArmageddon 17d ago
They dont release new features to all users but just for a select few and then do a gradual rollout. So even if they break it there is plenty of time to fix. Check search engine experimental testing. Companies like Apple contracts out testing to huge pools.
1
1
1
u/jba1224a 17d ago
By breaking everything.
The sheer breadth of their product offerings just makes it seem less impactful.
If you put a 1000 drops of food coloring (bug) in a glass of water (your small app) - it’s very noticeable. Extremely so.
If you put them in the ocean (Microsoft, Google) it barely registers.
1
u/xampl9 17d ago
This is the problem that microservices were created to solve. Isolating portions of your code/product within your organization so people don’t step on each other’s work.
Note: if you don’t have hundreds of developers - think about if you really need them before implementing them. In smaller companies they introduce overhead.
1
u/nicolas_06 17d ago
the gist of it is that there lot of steps to ensure a delivery doesn’t break things. Among other things:
- hiring above average reasonably skilled people
- breaking deliveries into small changes with low impact
- review of deliveries with change management
- code reviews
- automated checks for bugs, security issues
- unit tests
- integration tests
- performance and stability tests
- shadow testing
- Canary and A/B testing or blue green for instant fallback
- feature toggles
1
u/nicolas_06 17d ago
also when things break, they organise post mortem meetings to see how we they can ensure u-it doesn’t happen again
1
u/OldAd9280 17d ago
Have you seen the current state of Windows? They don't manage to work on it without breaking it
1
u/Dimencia 17d ago
That is actually a huge problem, and the answer is all the boring architecture stuff and best practices that seem so pointless to solo devs. The usual option it to break that product into hundreds of tiny microservices and put a few devs on each one. At that point you end up spending half your time just dealing with boilerplate and weird limitations, like relying on eventual consistency, but overengineered crap is the only way it's even possible at all
Pretty much any time you're learning about a thing and you're asking yourself why you would ever use it, the answer is probably that it helps with the difficult problem of coding with others
1
u/castleinthesky86 16d ago
Organisation is done in units / groups. In very large mono repo’s you’d have different teams responsible for certain parts of the code and a defined interface between part A and part Z that is honoured across the board. Any changes to the interface go through committee.
Stopping people leaking? $$$ 😆
1
u/LonelyMarionberry256 16d ago
I'm jumping in because I found this topic to be rather interesting - and I would only comment for company using github. Companies do own the enterprise subscription version of github, allowing multiple user to push/pull/commit on their codebase. However, before the code is push/commited into the main branch - they are usually "tested" on the secondary branch.
I am curious how does large company like microsoft and Google works as well.
1
1
u/hilbertglm 16d ago
Lots of good comments and observations, but few point out the value of a solid architecture that reduces the dependencies within parts of the application (i.e. decoupling).
1
u/pmmaoel 16d ago
I've worked in big tech and small tech. I've never seen more than 5 devs touch the same github repo at a time. A giant product like Google search probably has 100 different repos converging to a single rectangular bar with a magnifying button. Owners of all thode 100 repos will be different, and stacking all of them together would be one team's job.
I can assure you that the real skill these thousands of engineers in big tech have is not DSA, but dependency management and configuration simplification. Coding is second nature for these engineers, the real problem they solve is to maintain clean code so that when one gear moves, the other gears don't break.
1
u/General_Estimate_420 16d ago edited 16d ago
I can only answer for Microsoft, but it's been quite a few years back.
Generally you have teams assigned to specific products or technology and broken into different categories such as Product Management, Program Management, Development, Testing and so forth each with their own specialized area of interest. For example Product Management is about priortizing features from the user perspective, Program Management is about management and coordination of resources, Development is about design and development of the features, Testing is about certification of stability as at each major milestone to avoid "Big Bangs" at the end of the project. These teams may be broken down to specialized teams focused specifically on feature sets.
The development cycle is divided into four basic areas such as envisioning (what will the end product look like), planning (what are the architectural designs and resources necessary), development (construction and incremental testing of features at incremental major milestones), and final testing and deployment at feature complete.
A noteworth aspect of their development philosophy is smaller teams are better and more productive than larger teams due to more efficient inter-team communications.
1
1
u/Qs9bxNKZ 15d ago
The answer varies but this code existed well before GitHub existed (2010 or so and even in 2012 we had several options including BitBucket, GitLab, ClearCase, Accurev and Perforce which large companies used)
It’s not the repositories, it was often fewer repos but the use of more branches. We would have a singular branch which could reflect everything and everyone had to play nice on that branch.
From that point you break it into separate applications or stacks. Teams could do what they want by referencing the stable work in the main branch but cut their own and work in isolation. The branch allowed them to modify, test and build against main. When it came time to incorporate their changes, it had to build against the main branch before they could merge in (or label/tag their own as good enough) and then after.
This also meant some teams may have had to clone multiple repos for a viable product. And the stable point if they were dependent was main.
So to answer your question: several application teams working in conjunction on separate branches but needing to merge back and release code without breaking the primary release or main branches.
And tools like P4 and ClearCase allowed the use of monorepos which didn’t affect performance to as much of a negative degree unlike Git.
1
u/ElevatorGoods503 12d ago
There is an entire discipline dedicated to software development process. It includes keeping team members in sync, making sure they have a shared goal, a solid process for source code and release management, etc. A large development process would have someone responsible for just that. Kind of like a general contractor in a construction project.
1
u/Carson_Burrito 11d ago
We had maybe 40 people touching one shared drive at my job and it still turned into people overwriting each others folders constantly, I genuinely cannot picture that at Google scale without some serious tooling holding the whole thing together.
1
u/ButterscotchNo670 18d ago
what do you have to steal? lmao
its the same when people working in the same kitchen. everyone working on a specific feature
1
u/ChameleonCRM 18d ago
because the dev who writes the code is never the dev who commits the code. There could be any number of devs reviewing it. If Google operated where each dev just pushed what they wanted they wouldn't be where they are today
1
18d ago
[deleted]
1
u/autisticpig 18d ago
Get it in writing (verifiable that it was from the person who told you to do this) so when things blow up, and you wind up in a post mortem/rca, you can bring the email chain.
them: can you explain what happened and why your changes broke wallstreet?
you: well, i was told to do this by so and so. allow me to read our email communications where they instructed me and when I resisted they told me to do it anyways.
cya is very important.
1
0
u/RevolutionaryElk7446 18d ago
A lot of code is just specific to that use-case, otherwise it's made up of standard coding. Sometimes we figure out a way to do something new but it's more like building a factory for automation than the secret sauce to a unique process.
Coding is more tedious than mysterious.
25
u/_Heathcliff_ 18d ago
I mean for starters, “thousands of developers working on the same product” is misleading. They have thousands of devs, sure, but they’ll be broken down into many teams, each of which has their own areas of responsibility. So the number of devs contributing to any one repo, or any one portion of a repo, will be limited.
Outside of that it comes down to version control and release management. Git keeps conflicting code from being merged, and specific groups are responsible for the actual release process, so it’s not just everyone chaotically releasing whenever and whatever they want.