I’ve always found that the biggest issue with monorepos is the build tooling. I can’t get my head around Bazel and other Blaze derivatives enough to extend them to support any interesting case, and Nix has too many usability issues to use productively (and I’ve been in an organization that gave it an earnest shot).
Yeah, in my experience it takes X years of experience at a company that is already using a monorepo + blaze-derivative tooling, for the costs of setup and maintenance to make it worth adoption.
That X is mostly a function of how open you are to the idea that maintaining + using a centralized, language-agnostic build graph is a Useful Thing.
But once that X is crossed, I don't think it's possible to happily go back to a non-monorepo environment. :)
FWIW, we've used Pants successfully for years, which has excellent Python support. However, we're in a long-haul migration to Bazel, so you might be better off revisiting Bazel if you want to invest in something that's extremely likely to issue dividends over the next 10 years.
We used Pants as well, and it worked alright for Python. But it’s codebase was a mess and it was also difficult to extend. I get the strong feeling that there is a better way to make a build system—perhaps something like Nix but without so many usability problems—but no one is investing in that right now.
With GitHub actions you can quite easily specify a workflow for parts of your repo (simple file path filter https://docs.github.com/en/actions/reference/workflow-syntax...). So you can basically just write one workflow for each project in the monorepo and have only those running where changes occured.
That’s what I’m doing now, but you still have to push artifacts up to a repository and downstream jobs have to pull them down. Further still, managing the boilerplate for GitHub actions is no small task (e.g., you have a dozen Python projects and you want the actions for each to look similar), so to solve for that you have to build some hack that involves generating the Actions YAML before you commit or similar. Not the end of the world, but still far from my ideal state.
I’m not doing json net, but I am generating them effectively from a pre-commit hook. One of the jobs that I generate validates that the actions that were committed are up-to-date. This works well because I don’t have too many dependencies, but if you have deep dependency trees then the builds will end up taking a long time and you end up reinventing Nix or Bazel poorly.
Are nix and Bazel in the same category? I have used nix very little and never used Bazel. But I have used Make and jsonnet pretty often. Can you re-run Make in a pipeline to verify that the actions in use match what would be generated based on the repo state?
Nix bills itself as a fully reproducible package manager and Bazel as a fully reproducible build tool. In the abstract, both of these reproducibly build software, so that puts them in the same category in general.
> Can you re-run Make in a pipeline to verify that the actions in use match what would be generated based on the repo state?
Yes, and I’m doing something similar although not with Make. The problem is that this home-grown build system doesn’t do incremental builds, so everything will be fully built from source every time, and that can take a while for very deep dependency graphs.
Conceivably your “Actions generator” script could generate only the jobs that need to be run based on what changed since the previous commit, referencing artifacts from the previous commit for those build steps which weren’t invalidated. This is a neat concept, but we’re well on our way to reinventing Nix or Bazel.
An interesting case that's hard to support with Bazel.
Note: I don't quite like Bazel; but my take is that Bazel supports too much configurations and options: my workflows are always much simpler. Which is why I am curious to hear what do your workflows require.
It’s been a while since I tried it (maybe 2018?), but Python 3 support straight up didn’t work despite the documentation. It also seemed like adding support for your own toolchain required you to write Java plugins in a very confusing, inheritance-for-it’s-own-sake, raptly-2000s-OOP way.
Got it. I can relate: while Python 3 support is probably fixed now, I agree that Bazel tooling for adding toolchains has been always abysmal. Things somewhat get better (https://docs.bazel.build/versions/master/toolchains.html) since it's now in Skylark, not in Java, but I still don't like it (aka not productive)
Overall, custom toolchains are common (especially, in the embedded world), and a good build system should make that very easy.
