> Torvalds: [hot take that almost no one else agrees with]
Considering the popularity of vendoring and/or static linking dependencies, I'm not sure where you're getting the "almost no one else agrees with" from.
glibc's hostility towards static linking is arguably a substantial part of why musl has attracted so much interest. Static linking support is explicitly stated as one of musl's core tenets [0].
One can also argue the popularity of containers has in part been due to glibc's dynamic linking requirements, and containers have become the bloated "modern" version of static linked binaries, bundling all the dependencies without eliding all the dynamic linking overhead.
Torvalds is definitely not alone in this particular opinion.
> Considering the popularity of vendoring and/or static linking dependencies, I'm not sure where you're getting the "almost no one else agrees with" from.
Vendoring is hugely popular for projects in year zero, because it makes it easier to ship. Until they learn that packaging specific versions of all your dependencies with your project implies taking on responsibility for backporting all security updates to that version of every dependency when upstream is only actively maintaining the latest version. Or you have a project riddled with known published security vulnerabilities, which is what most of those turn into.
You use a library, you've already taken on responsibility for it. You're application has bugs due to bugs in a library, you're users aren't going to accept "we reported it to the library and they said it wasn't important" as the resolution.
At which point you either submit a patch to the library or switch to a different library. That much of the work isn't avoidable, except maybe by choosing a better library at the outset.
But having to fix it yourself when the library maintainers fail once is a far cry from having to backport every security update in every library yourself even if the library maintainers are properly doing it for a newer version than you're using.
The problem is that doing the backporting is typically more work than updating your software to use the newer version of the library, but not maintaining the library version you use at all is invisible in the way that security vulnerabilities are until somebody exploits them.
Vendoring libraries isn't really different from keeping a version lock file around. You need to review and update the dependencies regularly and monitor them for security issues. Only difference is that instead of a file with the versions you have the full libraries.
I personally don't like to vendor as it bloats the revision control system, but as far as dealing with bugs in the dependencies it doesn't really make a difference.
> Vendoring libraries isn't really different from keeping a version lock file around. You need to review and update the dependencies regularly and monitor them for security issues. Only difference is that instead of a file with the versions you have the full libraries.
In the first case, using a version lock with dynamic linking, the maintainer of the library package is the one paying attention to and shipping patches for the library. And if the version of the library you're using stops being supported, your software breaks and you have to fix it immediately.
That's the thing vendoring "makes easier" -- you're not dependent on the third party to maintain that library version. Except that now it's on you to do it, and the breakage that would have told you that the version you're still using may no longer be secure is now absent. So the developer has to pay more attention. Which they regularly do not.
You have two choices here. Keep updating as you go along and amortize the price for so much open source deps inability to avoid breaking change; or don’t do that and then eight years later lead a heroic update the deps project that takes ten people and millions of dollars and some availability hits. You keep the principal of change only one thing at a time the first way and get promotions the second way.
Static vs dynamic is just do you want the code owners to be responsible or some centralized ops group to be responsible. Usually ops doesn’t have the insight to do a good job with deps, so it’s better to have dev do it, e.g. statically linked go binaries, statically linked C/c++, frozen package versions in Python, I think mvn has a similar capability, also version pinned base container images etc etc. hence the fact all the packaging systems add new versions and only rarely remove things as that breaks it all worse than a security hole.
As a dev person with a lot of ops experience I like static because I don’t want some janky dependency change to come into the execution path when I am not aware of making a change. On the other hand, people usually just freeze everything and then don’t worry about it, which is wrong. But your security team needs to review source and source deps not just deployed versions. So if code is still owned, the bad deps can be found and fixed either way.
Ooor... you just upgrade your library versions. Backporting fixes is something you do as a distro maintainer to avoid having to update each application separately. It almost never makes sense as a thing to do as an application author. If you find yourself in a position where you don't want to upgrade to the version being maintained by upstream then you're in trouble anyway.
If you are using upstream libxyz 1.3 but they don’t provide security updates for it then you should do something even if you are using dynamic linking. Imagine that redhat provides security updates but Debian uses 1.4, are you going to tell your users that they must use redhat?
I don’t see that dynamic linking makes a lot of difference here.
Typically with dynamic linking, the library is maintained as a separate package by a separate party who takes distribution-wide responsibility for backporting anything.
So you have two alternatives. One is that you make your software compatible with both version 1.3 and version 1.4, so it can be used on both Redhat and Debian with dynamic linking. This is a bit of work up front which is why everybody grumbles and looks for an alternative.
The other is that you vendor in version 1.3 so that you're using version 1.3 even on Debian. But now that library is your responsibility to patch instead of the distribution's. So instead of doing that work up front, you either maintain the library yourself on an ongoing basis (which may be more work than supporting two only slightly different library versions up front), or you neglect to do it and have a security vulnerability.
As a software author, the first alternative doesn't sound very attractive to me. Because as an author, I will want the upstream version of libxyz, and if the distribution provides security updates past the upstream date, this means that I need to use that exact distribution to benefit.
And vendoring in is pretty much equivalent to what happens with static linking.
My parent had suggested that dynamic linking is some great solution, and I don't see it's that great.
> Because as an author, I will want the upstream version of libxyz
Why would that be true in general? It would only be the case if the upstream version contains some feature you need that isn't present in the version packaged by the distribution, and then you would only need to do it until the next version of the distribution packages the newer library version.
> and if the distribution provides security updates past the upstream date, this means that I need to use that exact distribution to benefit.
Or support the multiple versions of the library, which for non-ridiculous libraries is often no more than refraining from using the features not present in the older versions.
>> Because as an author, I will want the upstream version of libxyz
> Why would that be true in general?
Because I want to write software that works on various Unices, not just one Linux distribution. So it seems safer to rely on software that my users (the people who install the software I wrote) will be able to install from source, if necessary.
If I were to require whatever version Debian had at the moment, then I'd leave Fedora, Arch, OpenBSD, FreeBSD, Illumos aficionados in the lurch.
You can also automate the upgrade process with good tooling.
Detect a new version? Update your local copy, run your CI, if it passes, you can automatically merge. Or you could ask for someone of your team to review the changes, knowing your tests passed.
And if it doesn't, you can warn your team, who can then either fix the build manually (hopefully a small incremental change) or open a bug report to the library.
The point is to do that continuously, so that the deltas are smaller and easier to handle. I do that with some pretty critical libraries at work and it's mostly fine, takes a few minutes usually to approve changes every week. Doing an upgrade after 6 months+? That'd require a long rollout.
I'm not sure how dynamic linking even makes this process easier. Either you test every update or you don't. With static linking you could easily just automate having all the dependencies updated. You just risk randomly breaking things which is just the normal case for dynamic linking.
> glibc's hostility towards static linking is arguably a substantial part of why musl has attracted so much interest.
Anothre reason is that glibc is equally bad for dynamic linking. Often a binary compiled on recent glibc doesn't work with older glibc. It is certainly possible but quite challenging to compile portable executables on Linux.
Maybe I hyperbolized that a bit. I think there are pros and cons to each approach. The uses cases for static vs dynamic are also different. There are compile time disadvantages to static libraries, and there are security disadvantages to shared libraries. But calling one method better than the other is inane.
Are you aware of the runtime differences? Both time and space.
Shared libraries (at least with ELF) requires fixups, which involve COW and a considerable number of per-process pages. Compiling with -fPIC reserves one register, which isn't a problem unless you're on a register-starved CPU, which is all of them.
It's not particularly short of registers, but IIRC even aarch64 is short enough that they implemented register renaming. I agree that taking a register away is a much less pressing issue there than on the x86.
The purpose of register renaming in the AArch64 microarchitectures is to support out-of-order. By comparison, you don't find register renaming in the in-order A53.
Suppose you had enough registers, ie. so many that there's no real reason to compilers to save. In that case, you could "support" out-of-order execution by adding a couple sentences to the compiler guidance documentation for your CPU, telling them that they'll get better performance on o-o-o cores if they wait as long as practical with reusing registers.
They've tried variations of this. It takes instruction bits to describe these registers: 32 regs is 5 bits, target + src1 + src2 is 15 bits already. They've tried scratch pads. The Mill CPU's Belt is yet another approach.
https://en.wikipedia.org/wiki/Scratchpad_memory
https://millcomputing.com/
How to Use 1000 Registers
https://caltechconf.library.caltech.edu/200/
1000 registers would be 10 bits. So they tried register windows.
These are all good ideas but they get to compete. So far, I'm a huge AArch64 fan but I now see the purpose of RISC-V. I watched Chris Lattner's ASPLOS talk and it finally clicked.
I know... and it's quite interesting really, but for the purpose of this thread, I think it's safe to say that the CPU-wallahs spend considerable effort on ways to lighten or avoid register pressure, and from that I infer that all of the currently widely used CPUs do have significant pressure.
Much more on the x86 than on sane architectures, of course.
Considering the popularity of vendoring and/or static linking dependencies, I'm not sure where you're getting the "almost no one else agrees with" from.
glibc's hostility towards static linking is arguably a substantial part of why musl has attracted so much interest. Static linking support is explicitly stated as one of musl's core tenets [0].
One can also argue the popularity of containers has in part been due to glibc's dynamic linking requirements, and containers have become the bloated "modern" version of static linked binaries, bundling all the dependencies without eliding all the dynamic linking overhead.
Torvalds is definitely not alone in this particular opinion.
[0] https://www.musl-libc.org/intro.html