To all of the sysadmins that look at server access logs, see requests to "news.ycombinator.com", and, instead of adding a block rule, just smile and nod knowingly:
Indeed it does. I salute your ability to endure in that kind of environment. I did that bit for several years before finally saying enough is enough. Working for a small shop is a much better fit for me.
Hey sysadmins, how did you get there? Are there books, articles, blogs, etc. that helped you significantly? (Frisch's classic comes to mind.) Or is the skill 90% about getting your hands dirty? But then again, how did you go from sandboxed sysadmin to your first real, critical job?
What worked for me was to run my own servers and services. You have to learn how to manage things because there's no one else to call on - and if it breaks, you have to fix it.
Hence why I have stories about how to recover a remote machine after PAM has been uninstalled.
A background in customer services can be parlayed into technical support, which - if you can show competence - can get you in the door somewhere and get you commercial experience.
I have no formal training or education, and I don't think I'm an uncommon case. This is a job you can learn by doing.
If you have a development background, look into devops, which is the hybrid/crossover movement. Learn how to deploy your app, how to configure Apache (or the reasons why you might not use Apache at all). For your next weekend project, try getting a cheap VM/VPS and deploying to that instead of Heroku.
The nature of sysadminning is changing, thanks to the flexibility offered by 'cloud' hosting solutions and automated deployment and configuration management tools. Learn how to use chef or puppet. Look up 'chef-box', which uses vagrant and VirtualBox to give you a virtual network that's configured using chef. You can use this to learn how a real network works.
Learn about virtualisation, buy a cheap machine (I like the HP Microserver) and install Xen on it. Increase your swearing vocabulary by trying to use VMWare in production with no support contracts.
I have no idea how you learn about Windows. Never owned a Windows box, never had to do meaningful support for Windows systems. Even if you're a Mac/*nix person, you should probably learn how to manage a Windows network, though.
Learn to hack on the OS of your choice. Make sure it's fun to you first.
Next, get a job as helpdesk. Go through the pain of being the first person people talk to when stuff it broke. Try to fix it. Always try to fix it before you escalate. Preferably you get this job a mid-size company where you can have the opportunity to get to know the staff in general. Single office, walk around it once a day just saying hi to everyone.
Eventually they'll start to wait for you to make your rounds to get help. You're building relationships.
Get to know the maintenance and security guys in your building. These people need to be your friend. Your systems don't run without power, and you need to be able to get to consoles.
Watch for and take opportunities to get into a sysadmin position. Now, working on the trade skill of being lazy. Every task you do, think about how you can automate it. That's one of the real arts to being a sysadmin, if you have a lot of work that keeps you busy, how are you going to have time to respond to problems.
By taking this route though, you've built up one of the 4 skills I think you need to be a good sysadmin. People skills. Remember, by the time someone has to talk to you they've already had to deal with at least one other person who could not fix their problem. The rest of the skills you get from experience as a sysadmin. Scripting, documenting, creating DR scenarios and such... it's a never ending learning opportunity. If you've learned all there is to learn where you're at, then it's time to move on, unless you just really like it there. As often as not the latter is the case and that's not a bad thing as long as it pays the bills.
The one thing I always tell people who ask me to explain being a sysadmin to them is this. It's a service oriented position. The employees are your customers. The business is your customer. Your boss is your customer. You will hit a point where you think there is a technical answer to everything and you know better than them. You don't. It's not your job to make technology changes to change everything and make it easier. It's your job to implement and maintain what the business users feel they need to use to get their job done. Their positions make money for the organization you work for, your department is nothing but cost. Remember that and you can save yourself a lot of frustration.
Oh... and go home and build yourself a nice home network. This is a job where you take mental tasks home with you, having a place to play is helpful. Also helpful for job interviews.
Ah, another tip. Learn the phrase "it is what it is". It's the answer to everything from lieing vendors to incredibly poor implementations. If you can learnt to accept bad software, implement it in a way it's not going to break the bank to support and move on, you'll find yourself avoiding the path to the "grumpy SA".
I'm a programmer, but believe it's important to be well-rounded. To balance my programming skills with sysadmin skills, I've started reading "The Practice of System and Network Administration":
The book was praised by multiple HN commenters ( http://news.ycombinator.com/item?id=1665915 for example), and I'm glad I listened to their advice. I've only read a few chapters so far (it's a big book!), but I already like it. It contains a lot of good insights, and I like that it's not focusing on a specific OS / platform. Also, you don't have to read it in a linear fashion, you can pick and choose chapters depending on your current needs (moving into a new data center? implementing a security policy?). Thumbs up.
My situation is sort of a unique one. To help pay for college, my dad forced me into a summer job doing basic data entry and some light engineering work at a company which did outsourcing for aerospace companies. They had dozens of AIX 4.3 workstations running NFS, NIS, and all of the arcane bullshit everyone used back then.
When the one-woman IT department heard I'd switched majors that spring to CS, she let me take on some of the workload. This mostly consisted of wrangling their VAXstations, handling various requests from users (which I normally passed on to the helpdesk of the company we did most of our work for), and fixing print queues. Eventually, I ended up working on the AIX stuff, too, and then installing and configuring our own servers (AIX and FreeBSD.)
When a branch office closed, I also inherited a Sparc 5 attached to a plotter and a couple of SGI Indigos. (I basically kept the SGIs on my desk because they were useless to our engineers.)
By the end of the summer, I knew more than most 19 year olds should know about AIX, Solaris 2.3, and IRIX 6.2. I stayed there through school during off-semesters and gradually became the go-to person for Unix issues, serving as a mediator between our engineers and the client.
Within a few years of graduation, after the obligatory gig at the failing startup, I ended up as a junior, then senior Unix SA at the aerospace client. It was pretty much the most perfect set of circumstances that could happen, and it was almost as if I had no control over it.
So I guess, for me, being a sysadmin sorta just happened.
That probably wasn't too helpful as a career guide.
As far as the day-to-day side of things goes, though, the only real tools I use these days are google, man pages, and our vendor support, pretty much in that order. And some Hitachi course manuals.
Real SA intuition only comes from experience. Having a degree in CS might seem like overkill, and certainly isn't necessary, but it really doesn't hurt, either. Having a solid fundamental understanding of how computers work at the very lowest level is a huge advantage in the kind of "forensic" work (for lack of a better term) you'll find yourself doing from time to time as an admin.
As far as learning the OS itself goes, and gaining some proficiency at it, my advice is to build, and break, your own systems as often as you can. I basically went through my entire adult life until recently with some sort of Sun, IBM, or SGI machine (usually two or three at the same time) kicking around somewhere in the house: I used them for storage; I learned how to mirror disks; I did tape backups with them; I fucked around with NIS maps and sendmail configurations and DNS. I broke everything dozens of times long before I ever touched an important system at work.
BTW, it took about four years of full time work before I really felt like I knew what the hell I was doing most of the time.
misc: Sites I read every day: HN, reddit, yankees websites, drudgereport (for the lulz.) I really don't keep up with the very latest in tech news or what Oracle or IBM are releasing, unless their sales reps are telling me about it over lunch. There's no need to keep super-current on hardware unless I'm designing systems. (Hardware isn't hard, usually.)
Sadly all of our sysadmins have been moved off site and across the country (actually, most of them decided not to move to Texas and quit instead). I'm thinking of them today but don't ever meet them or know what they look like.
Here's to working 4 days in a row, sleeping behind server racks, bathing in bathroom sinks, and all those bags of potato chips and caffeinated soft drinks!
I thanked the sysadmin crew at the office yesterday and took today off. :) Did the full-bore admin for 10 years before I got burned out and shifted to development. Still am part of the informal team that assist with various scripting, monitoring, etc... Work wants another DBA, so I may have to buy some more caffeine and lube to gain the bigger check.
This one's for you.