There is something epically beautiful about OpenBSD's consistency over the last 10 years. I swear that a 2004 OpenBSD sysadmin would be close to 100% efficient with a circa 2015 OpenBSD system, despite the absolutely massive number of improvements and changes that have been made with the operating system.
Once systemd starts to get traction, it's cognitive overhead is going to become a real problem for sysadmins who want to be able to model the entire boot process and run-state in their head.
For better or worse (I honestly don't know which, btw), you are going to have to start to trust the operating system to manage its boot process/run-state correctly rather than having it be a direct implementation of your mental model. (I would argue other *nix Operating systems, like OS X, have, to some degree, been doing that successfully for a while, so it's certainly possible - just not clear whether you want that type of behavior on your server systems. I guess time will tell.)
I'm not a professional sysadmin, but as a developer with a bunch of virtual servers systemd has only made things far easier.
Writing init scripts used to be a real pain, now I can set up a unit file in two minutes and it works...
Maybe there's something to be said for "sysadmins who want to be able to model the entire boot process and run-state in their head" but with correct information about dependencies and so on, I personally think that's something computers are a lot better at solving.
> Writing init scripts used to be a real pain, now I can set up a unit file in two minutes and it works...
This.
I'm not saying systemd is the best answer, but I really think sysadmins could learn a thing or two from developers. Nearly every custom init script I've ever seen has bugs, and isn't portable between systems. Where are the unit tests? Code reuse? Focus on maintainability/readability?
Shell scripting used to be amazing, but as a language these days it's behind the times.
Sysadmins would say that developers could learn a lot from them. Like, documentation, maintaining backwards compatibility, sane upgrade processes ...
Don't throw stones in glass houses and all that. We all could stand to learn things from other people.
Sure, sysadmins could stand to be better programmers, but software developers tend to forget that software is just a tool, and sysadmins use it like a tool. They just want to get in, get out, and move on to the next problem, so for them, certain best practices in software development aren't all that attractive.
> but software developers tend to forget that software is just a tool, and sysadmins use it like a tool. They just want to get in, get out, and move on to the next problem
Which is exactly why developers don't get to write any documentation other than the bare minimum: software for the client is just a tool, once it is running they want developers to move on to the next problem, not wasting time on any additional documentation (which, arguably, is not strictly a developer task, more a QA one).
> Shell scripting used to be amazing, but as a language these days it's behind the times.
It was never amazing, was given far too much credit, used too widely, and brought up a generation of people thinking that this is the way to manage a complex web of dependencies.
On the other hand, constraints like the above tend to foster creativity, so perhaps that's why people have been so nostalgic for this way of solving the system startup problem.
You should take a look at modern init scripts in FreeBSD. Common functions are sourced in, and then most "scripts" are just a series of variables that define various aspects of a daemon. Scripts are about the same length as systemd scripts, yet you have all the power of shell scripting right there if you need to do something special. In contrast, the systemd method means that if you need to do anything extraordinary, you have to call a shell script, and you've suddenly lost all the functionality given by the init system. So, for all intents and purposes, systemd ends up giving you the worst of both worlds if you need to call out to do housekeeping on startup/shutdown.
>I'm not saying systemd is the best answer, but I really think >sysadmins could learn a thing or two from developers.
>Nearly every custom init script I've ever seen has bugs, and >isn't portable between systems.
Nearly every custom init script is written by a developer for the product you desperately have to run because development wanted it, not bothering to give us sys admins time to look into the: packaging, configurability, operability and if a custom module/cookbook/playbook needs to be written.
>Where are the unit tests? Code reuse? >Focus on maintainability/readability?
If we were given the time, it would be there. My deadline for almost all work I do is "yesterday".
A lot of us sysadmins are or were developers too, it's a lot easier to maintain a system you understand. Plus if it can be automated in code, it will be automated in code, we're lazy like that (well some of us, if I have to do the same thing more than once, I'm not doing it by hand the second time) and end up writing non-trivial amounts of code. Nearly every piece of software I've ever seen has bugs, and what do you mean I can't run the software without xyz framework, abc runtime and libraries a-zzz being installed on the system. Where are the performance specs? Resource reuse? Documentation? Focus on deployment/maintainability? Security audits? (I'm only mostly sarcastic)
Shell scripting is still amazing... when you use it for shell scripting.
Now I do like systemd more, but not because init scripts are bad, I just prefer how it does things, and can always call out to a script if need be anyways. And I suppose making it more friendly to developers is going to make my life easier too, the atrocities I've seen written in every language imaginable to replicate something that would be trivial to do in an init script. I didn't think it was possible (well possible yes, but why, WHY) to essentially rewrite the userland tools and the job of the init system in (very poorly written) php, but I've been proven wrong. Even something like forever for nodejs doesn't really have a need to exist.
But point is developers and sysadmins are both more than capable of writing shit code, lacking tests, code reuse, and all the rest of the best practices. At least we have to deal with the fallout from our code ourselves ;)
The shell has always been awful. It is underpowered and overfeatured with a deeply crappy API. It is more like Windows 3.1 than any sane development environment.
The good news is that you don't have to implement initscripts as shell scripts. I recall at least one GNU/Linux distro that used Python for all its initscripts, for example. Really, as long as the "script" is executable, a SysVInit-based or BSD-rc-based system doesn't really care (unless it's doing something strange like calling a specific interpreter on each script - i.e. calling `sh /etc/rc.d/rc.*` or somesuch).
Actually, a FreeBSD or NetBSD rc system does indeed care, for that very reason. rc.d scripts are sourced by a /bin/sh shell, using the . command. See run_rc_script on the rc.subr(8) manual page for details. Ironically, it's daemontools and daemontools-like systems where what you say holds the most widely. Tools such as execline can be and are in practice used in "run" programs.
It doesn't look like OpenBSD has such a requirement (the start_daemon() routine in /etc/rc just calls each configured daemon's initscript as-is from /etc/rc.d without specifying sh or ksh to execute with); I had (incorrectly) assumed that this was a general BSDism rather than an OpenBSDism.
If your needs are simple, the resulting unit file is trivial and you can basically copy and paste from any example without looking at any documentation.
If your needs are complex, you have better chance by mixing different options in the completely declarative syntax of systemd than mixing shell snippets from different scripts using different styles.
In any case systemd happily launches LSB-compatible init scripts, you literally don't lose anything.
> Complex needs are better solved with source code rather than special blobs of $system.
Arguably, reinventing multiple complex wheels is often a not particularly good choice.
So either you're implementing something really novel in your init script (which does not sound like the best place for it) or it's something that may benefit from a single, shared and tested implementation.
And, again, you can still fallback to LSB shell scripts if you really need something incredibly special: my point was that, bloat notwithstanding, you don't lose any feature but gain many.
In the GNU/Linux world, yep, this is the case. Lots of copypasted boilerplate that becomes a nightmare to debug.
In the BSD world (at least with OpenBSD, where I'm most experienced), this isn't the case at all; OpenBSD ships with /etc/rc.d/rc.subr, which one can (and should) source into one's initscripts to make life much easier than even (in my experience) systemd's unit files. Set the $daemon variable, source in rc.subr, call rc_cmd, done. Sure, you might want/need to read a couple manpages, but this is the case with systemd, too.
Back to the GNU/Linux world, the complexity of SysVInit is not a hard dependency of using shell scripts for daemon management; there's very little stopping a GNU/Linux system from adopting a BSD-style rc system, rc.subr, etc. and making life significantly easier for sysadmins. Alternately, there's very little stopping a GNU/Linux system from implementing shell scripts in something significantly less horrendous to use, such as an interpreted scripting language (like Perl or Python or Ruby). SysVInit and BSD-style rc-based init both have a distinct lack of hard requirements on how initscripts are implemented, so long as said scripts are valid executables.
> Maybe there's something to be said for "sysadmins who want to be able to model the entire boot process and run-state in their head" but with correct information about dependencies and so on, I personally think that's something computers are a lot better at solving.
Would you not want to be able to model the startup process and run-state of the software you write in your head? Why should the underlying operating system be any different? If you have to rely upon external tools just to be able to reason about the execution flow of something - your server, your program, your recipe for lamb vindaloo, whatever - chances are it's way too complicated and needs significant refactoring.
...and that's awesome if you don't need 99.9% uptime and thus need to have absolute faith in your system's stability. However, for mission-critical applications that do it's hard to argue that systemd is a better solution than writing all your scripts by hand. Unless you want to be able to blame systemd if it all goes pear-shaped but I doubt your boss will buy it =)
I would argue that features such as having explicit dependencies are absolutely required for 99.9% uptime. Building systems resilient to failures also means being able to predictably spin up new instances that you can trust.
I'm also a developer and for example one problem I had with my servers on AWS was that my process sometimes happened to start before my volume meant for logs was mounted, which meant that all logs were going to my primary volume, with the server crashing at some point due to out of space errors. Another time I was using a local database meant for local caching and again, sometimes it would fully initialize only after my main process was started.
And because I was using ELB with auto-scaling, it meant that these incorrectly started servers would be connected to the load-balancer, because the main process was suddenly responding to /ping, even though it wasn't ready yet. And then it would receive thousands of requests per second per server, at a moment in time in which the cluster badly needed fresh instances, which led to an unstable cluster. But only sometimes ;-)
Except that there's no reason for systemd's model to "get traction" outside of desktop-focussed Linux distributions, since a "better user experience" is it's current raison d'etre.
So no, I shouldn't need to have to "learn to trust the operating system / systemd" on my servers -- it has no reason to be there!
The model where services do not daemonize and write a pid to a file is ...
... the one that people who moved away from System 5 rc scripts years ago have been pushing for with some degree of success, for years. systemd is not its source at all. systemd has merely been reaping the benefit of all of the programs over the past 18 years that have gained a --no-fork or a --foreground or a -D or a -F or whatever option in response to those who wanted to run services properly without bodging things under daemontools, or runit, or s6, or perp, or freedt, or daemontools-encore ...
The per-unit cgroup and the ability to kill it is rather nice too, without delving in the container-related features where the systemd server-side support really shines.
Many systemd features are explicitly tailored for server workload (eg. resource management), and a particular effort is being put over container-based deployments.
CoreOS has been quite successful in basing their own infrastructure on systemd and with rkt they are moving to use systemd even more.
Once systemd starts to get traction, it's cognitive overhead is going to become a real problem for sysadmins who want to be able to model the entire boot process and run-state in their head.
For better or worse (I honestly don't know which, btw), you are going to have to start to trust the operating system to manage its boot process/run-state correctly rather than having it be a direct implementation of your mental model. (I would argue other *nix Operating systems, like OS X, have, to some degree, been doing that successfully for a while, so it's certainly possible - just not clear whether you want that type of behavior on your server systems. I guess time will tell.)