The positive headline hides bad news for those of us who run CoreOS. CoreOS will be out of support soon(?). Migration instructions have been announced, but not written yet...
There is a CoreOS fork at https://www.flatcar-linux.org/ promising seamless continuation. I have no experience with or further knowledge about it.
Yes, sorry, unprecise terminology. When I say CoreOS I mean CoreOS Container Linux.
Fedora CoreOS is marketed to be a continuation. But it's not a seamless one, it requires manual porting. Detailed instructions don't exist yet, but the list is not that short https://github.com/coreos/fedora-coreos-tracker/issues/159. While most people won't be affected by many items, a couple of days are easily burned for understanding, coding and testing. Coding probably the smallest part, but if everything goes without "surprises" in real life, I would be surprised.
From what I've seen so far, it is a continuation in name only. Many of the things that made CoreOS nice have disappeared like locksmith, and other stuff has been made incompatible for no good reason, for example dropping docker...
Locksmith[0] implementation is tightly coupled to the specific update daemon, so it can't be directly re-used outside of Container Linux or without update-engine[1].
Its logic has been ported over to Zincati[2], which performs reboot management on top of rpm-ostree[3].
This is generally one of the philosophical differences I see between the RHAT family of distributions and some others (those that I personally favor), that seamless upgrades are generally not well supported from each major version to next.
For example, and I'm sure I'm oversimplifying, and there are caveats; for years, I've maintained Debian and Ubuntu machines which I've used mostly staying inside of the bounds of the apt package manager, and since many years I've never had a need to question whether I could "dist-upgrade" from one release to the next. It just works.
It is a supported operation, and reliably so. At my current job, we use Amazon Linux, a distro which I understand to be mostly derived from CentOS with Amazon addons, (and I'm open to being corrected if anybody knows better, but...) one of the things we're coping with right now is that Amazon Linux 2 does not have an upgrade path from Amazon Linux 1. We are on the hook to rebuild those machines, it's a fact that has given us pause to seriously re-evaluate whether we want to stay on the same train, or get off and switch to another paradigm broadly.
I don't strongly see this as a decision that Amazon made, because I'm fairly sure it's also true that upgrades from eg CentOS 6 to 7 are unsupported. And CentOS apparently got this idea from RedHat Enterprise Linux upstream, which provides this helpful page locked behind paywall[1], but from above the fold you can tell at least one thing without logging in: there are only a limited set of cases where upgrades are supported, even with a paid subscription to RHEL.
Honestly I've had more success migrating between versions of the Fedora family than the Ubuntu family. I don't have enough experience with Debian to judge.
In an case I disagree totally with the assertion that Fedora isn't good at upgrades between versions.
CentOS 7 upgrade would be tricky to do automatically because of the systemd transition and the fact that any Enterprise user would likely have a ton of services they need to migrate safely.
> In an case I disagree totally with the assertion that Fedora isn't good at upgrades between versions.
Same here. I think that changed somewhere around Fedora 15, though. Prior to that it was a minefield. That's probably why it has such a bad reputation among those who've not tried recently.
My current desktop workstation started out at Fedora 14 and now sits at 31. It's gone through multiple motherboard and component upgrades over that time, too.
I said RHAT family, but honestly I did not mean to include Fedora. I've been upgrading my Fedora Core desktop since probably 26, it's not the machine that gets the most use, but it works at least as well through upgrades as the Ubuntu and Debian systems that I maintain.
But none of my servers run Fedora. I still think that the core use case for Linux is servers. (And that's not to say my servers couldn't run Fedora... but it so happens the OS distribution that we use at my $dayjob is usually Centos or Amazon, sometimes RHEL, occasionally Ubuntu, but for some reason absolutely never Fedora.)
I'm not sure that migrating to systemd is as hard as you think, I've done it on my Debian release train, and I've also done it with Ubuntu, after migrating from Sys-V to Upstart, no less even...
It's actually standard policy from what I understand on the distros that are in the RHEL family, that upgrades between major version of releases are officially not supported.
So, Fedora is nice, but if you're using RHEL, be prepared to rebuild everything from scratch every 2-3 years, I guess?
The same is true for Amazon Linux -> Amazon Linux 2.
>We are on the hook to rebuild those machines, it's a fact that has given us pause to seriously re-evaluate whether we want to stay on the same train, or get off and switch to another paradigm broadly.
That's part of the selling point of Kubernetes. Your base OS needs to have just enough to run Kubernetes so upgrading it (ie: nuke and rebuild) is easy. You containers on the other hand can be upgraded piecemeal and are built with code. It's more things to handle but makes each thing smaller in scope.
For Container Linux/Fedora CoreOS, in-place updates don't make much sense. Those are intended as immutable OS images. They get their configuration from a remote source specified on the kernel command line.
The migration work is dealing with changes in the configuration language.
Upgrades between CentOS are not officially supported. Upgrades from RHEL 6 -> 7 and 7 -> 8 are supported, but only for an extremely small subset of packages and systems setups. It’s not really worth it. PXE boot with custom kickstart and post config (e.g. Ansible) runs are a much better solution than the in place upgrades.
> The migration work is dealing with changes in the configuration language.
Yes, that's how all installations work for CoreOS Container Linux. The problem is the day when the autoupdate of the current distro stops (no more new images, and that's supposed to happen soonish, although no date has been communicated) your old configuration language will stop to work (unless you are happy enough not to used anything on the list of incompatibilities, which is pretty unlikely for non-trivial installations). And then you need to port and test.
In my experience coding the configuration takes at least 3 times as long as installing in a traditional way. That is a worthwile investment if you want to be sure what you have installed and want to be able to start identical machines later. But if the investment is broken by commercial/political decisions that's frustrating.
Well, I am not a paying customer, so I cannot really complain. I just have to decide whether I am happy with the flatcar fork or whether I port my configuration to Fedora CoreOS. At the moment they have no complete documentation AFAIK, So it's difficult to make a decision.
We started Flatcar Container Linux to provide the option of continuing as is. We don't like to see perfectly good software be discarded simply because of an acquisition. We understand the hassle with porting configurations and have found a good number of organizations who are supporting our effort through support contracts to continue in the manner they intended when they chose to use CoreOS Container Linux.
We also released the update service as an open source project, Nebraska (https://github.com/kinvolk/nebraska). That was something that was closed source with CoreOS. This allows you to be in control of your updates.
It should be noted that we went a step further than CoreOS by releasing the update service we use to manage updates as an open source project, Nebraska (https://github.com/kinvolk/nebraska).
Is there a firm date for this? The original statement was "at least to the end of 2019". My employer uses CoreOS Container Linux extensively. It's really frustrating not knowing when the hammer will drop.
I'm not sure Flatcar Linux will make it through our security team. I'm "looking forward" to the migration guide to see how much pain i'm in for!
We're happy to talk about what you need on the security front. Some of the folks working on Flatcar Container Linux have a very strong security background; worked on AWS' EC2 security team, do regular pentesting for distributed systems[1], have reported dozens of security issues to upstream projects packaged in Flatcar Container Linux, including the kernel.
We've just now started breaking away from the upstream project, and updating packages. Addressing any open security issues is front and center in our efforts. If you have concerns we'd love to hear them.
We worked with CoreOS team for years (was our founding project) and they trusted us on the security front. We feel that if you trusted CoreOS and know our team + background, you should have just as much trust in Kinvolk.
Thanks for the informed response. To be honest I hope Flatcar does well, I notice the Docker version has been updated on your edge release - long overdue from the upstream project!
I will let you know if our security team has any issues if and when they look at it.
Thanks for giving everyone the option of staying on Container Linux!
The whole point of CoreOS Container Linux was to deliver a steady stream of security/software updates. We've been eager to update packages for Flatcar Container Linux but have wanted to maintain as much compatibility as possible for as long as possible. Fairly soon, however, we'll be introducing an updated kernel and user space (systemd, Docker, etc.) into the alpha channel. For us, this will mark the point where we feel like we're fully taking the reins from CoreOS and carrying forward the original objectives.
There is a CoreOS fork at https://www.flatcar-linux.org/ promising seamless continuation. I have no experience with or further knowledge about it.