Have you ever worked at FB / Google / whatever other company has huge mono repo with great tooling?
I went from many years at FB, to a place like you describe - hundreds of small repos, all versioned. It’s a nightmare to change anything. Endless git cloning and pulling and rebase. Endless issues since every repo ends up being configured slightly differently, and very hard to keep the repo metadata (think stuff like commit rules, merge rules, etc) up to date. It’s seriously much harder to be productive than with a well-oiled monorepo.
With a monorepo, you wanna update some library code to slightly change its API? Great, put up a code change for it, and you’ll quickly see whether it’s compatible or not with the rest of the whole codebase, and you can then fix whatever build issues arise, and then be confident it works everywhere wherever it’s imported. It might sound fragile, but it really isn’t if the tooling is there.
I have worked at a company that has huge monorepos and bad tooling.
Tooling isn't the problem though, the problems are:
- multiple monorepos copying code from each other, despite that code should be a library or installable python package or even deb package of its own
- you will never understand the entire monorepo, so you will never understand what things you might break. with polyrepos different parts can be locked down to different versions of other parts. imagine if every machine learning model had a copy of the pytorch source in it instead of just specifying torch==2.1.0 in requirements.txt?
- "dockerize the pile of mess and ship" which doesn't work well if your user wants to use it inside another container
- any time you want to commit code, 50000 people have committed code in-between and you're already behind on 10 refactors. by the time you refactor so that your change works, 4000 more commits have happened
- the monorepo takes 1 hour to compile, with nothing to compile and unit test only a part of it
- ownership of different parts of the codebase is difficult to track; code reviews are a mess
I think all of your problems are root caused by the phrase "multiple monorepos". This sounds more like "polyrepos" or something, multiple siloed repos that themselves might contained disparate projects, languages and tooling.
Google3 is a true monorepo. 99% of code in the company is one repo, with minor exceptions for certain open source projects and locked down code.
Edit: for example, you can change YouTube code, Google Assistant code, a Maps API, some config for how URLs are routed at top level load balancers, etc all in one CL if you really wanted to/needed to.
- use blaze or similar and run all downstream tests, the binary for your ml model includes pytorch whether you build from source or requirements.txt.
- other people committing doesn't matter if you are isolating presubmits and similar.
- using a blaze-like you never compile the whole monorepo
- code owners etc. makes this straightforward.
Like, as someone whose career has been mostly at Google, these are not problems I encounter at all, or only in cases where you'd have similar scope of problems no matter the repo structure.
If you only need to run the tests you affect, and only need to sync and update files touched in your CL, external changes are generally not impactful, sync & submit is a quick process even if people are submitting things elsewhere.
It's only a problem if someone submits a file you're touching, in which case you just have standard merge conflict issues.
I went from many years at FB, to a place like you describe - hundreds of small repos, all versioned. It’s a nightmare to change anything. Endless git cloning and pulling and rebase. Endless issues since every repo ends up being configured slightly differently, and very hard to keep the repo metadata (think stuff like commit rules, merge rules, etc) up to date. It’s seriously much harder to be productive than with a well-oiled monorepo.
With a monorepo, you wanna update some library code to slightly change its API? Great, put up a code change for it, and you’ll quickly see whether it’s compatible or not with the rest of the whole codebase, and you can then fix whatever build issues arise, and then be confident it works everywhere wherever it’s imported. It might sound fragile, but it really isn’t if the tooling is there.