Tools are automation.

Rule of thumb: do it manually three times before automating it. That way, you'll have a fair idea of what it is supposed to do (which is the hardest part); you know whether it's predictable enough to be automated; and an how often it will come up.

A programmer is someone who takes 5 hours to automate something that takes 5 minutes

There are three cases for a task that takes a regular human 5 minutes and a computer programmer 5 hours to automate:

1 The task is very complex, this means that there's a lot of room for human error. A well-tested and specified program will reduce this fragility and improve predictability.

2 The programmer is inexperienced, the programmer either is not a very good programmer (yet) or is using tools that he is not familiar with. The 5 hours mostly go into the learning process of the programmer, not in the task itself.

3 The task is simply something computers are not very good at, or hasn't been fully explored. For example speech recognition. The 5 hours go into exploring the domain, creating something reusable that the world could profit from.

In my opinion, it's always worth it, isn't that nice?

You miss the most common issue:

4 The task requires interacting with a system that was not designed with automation in mind.

For instance you have a third party GUI which you don't have source to that you have to push a button on based on what it is showing you. So you wind up having to research this, work around it, etc. Often you can use a GUI test kit to solve it temporarily, but that is going to be fragile - on the next upgrade it will likely break.

In this case automation may not be worthwhile. What you're learning is not connected with what you do. The automated solution is more fragile than the human one. The time saved is not that important.

The solution to that is to reimplement the functionality outside of the GUI. Write the GUI out of your life.

Great if you implemented the thing with the GUI that you have to interact with. Not so great if it comes from a third party.

Even if your people did implement it, depending, getting rid of the GUI might be easier said than done. I remember an Access app that I had to automate stuff around. In the end the best solution to all of the different popups that were buried in their app was to make the machine it was on remotely accessible, and have a watching process page someone if it took too long. A human would then look at the screen, and press the appropriate button.

Yeah, I worked with something like that about 10 years ago. We had an ETL process that involved like 20 Access macros manually triggered in sequence (click, wait, click, wait...over several days per cycle). I converted this entire process to a Perl script that would be pointed at the raw data directories, clean and process the data and throw it into the DB. Access was taken completely out of the equation. It took a few months, but I was an IT guy who had never programmed before and had to learn Perl at the same time.

Even that 5 hours / 5 minutes is worth it if you're going to be doing it enough.

Or a dozen people are doing it dozens of times a day.

Of course it all depends on your actual business, income (i.e. how much time can you afford to spend on internal tools), needs, etc. Since you're talking about client-facing software, I'm going to assume you're in a software development company, which narrows it down a little.

For our company, we don't develop internal tools for any given problem until it's a real pain. You'll know when it's an actual pain (as opposed to just an idea), because you'll keep seeing it crop up in your day-to-day work, and if you talk about it with others on your team, they'll say, "Yeah! I know exactly what you mean, I was thinking about that the other day."

We wait until it's a real pain for a couple reasons:

1) We want to make sure we're not wasting our time; we're mostly engineers, and we all get real, unadulterated enjoyment from building things. It's only too easy for our enjoyment and enthusiasm for building a solution to fool us into thinking there's a problem. But usually, if everyone on the team is instantly excited about the prospect of solving some problem, that's usually a good indicator that it's a real problem.

2) The longer we can put off something, the more information we'll have when we go to solve it. In other words, we give ourselves time to figure out first how we want to work, and then find or build tools around that, rather than trying to fit our system to some tool.

The second reason is really important too, because when we sit down to really think about every facet of a problem, how it affects our work and how we have been putting up with it and working around it, we'll often realize the best solution isn't some new piece of software; it's merely leveraging some tool we already have slightly differently.

For one example, we started losing feedback and updates in email, because some person on the team would get left off the email (e.g. someone hit "reply" instead of "reply-all"), and we wouldn't realize it until some task went un-resolved for several days. We could have built or implemented some sort of support desk system and made it the clients' responsibility to make sure tickets are submitted and then integrated it with email. Or, we could just create a unique email alias for each client, which copies everyone involved. The latter took 2 minutes and improved communication threefold.

If you keep growing, you'll eventually reach a point where anything less than a full-blown software solution just isn't going to cut it. But by that time, you've seen how the team deals with the problem over time, and all the pitfalls of other smaller or less-custom solutions, and you'll be able to build the most awesome solution possible.

Thanks! I'm in agreement with you, I don't like spending a lot of time on tooling if it isn't needed. I like only very simple tools, as I find complex ones inevitably add friction in the workflow and thus end up unused.

I would suggest that it depends a lot on the stage of the company. Early on, only build enough that you can legitimately track the key metrics. Other than that, focus on building the product.

As the product and business matures, it's likely that you will need to spend more time to ensure that the business can keep up with the product.

As a PM, I make sure that we've had to do the same thing manually a number of times before we spend any engineering time on automating it, or setting up the tools so that a non-engineer can do it.

I have actually automated myself out of a job before so be careful! Fortunately the company saw the value of this and contracted me on an adhoc basis.

I wad the unfortunate administrator for a large virtual mail system in the uk which was run off exim and flat files across 50 machines (one per client). This took 3 guys full time to manage as the machines were all over the uk and needed regular maintenance. I centralized everything and moved it to two virtual postfix machines backed with postgresql and wrote a web interface and set up scripts for backup and administrative operations, including integrated dns management.

Now, 12 years later, these guys are one of the larger hosting companies in the UK...

Automation almost always pays off, but not how you anticipate in some cases.

Why they didn't promote you for re-architecting their systems like that is a failure on their part. I hope your contract fee made up for that!

a) they were assholes, b) yes - daily rate was previously my weekly rate :)

My answer at this very moment in time is around 1/3. Which seems a lot. Though these projects have been less about automating drudgery and more about unlocking new workflows and ways of understanding things.

When I first started working at my job I sneakily built a suite of tools for visually working with NL grammars (instead of just buckling down and writing the grammars). Although I don't maintain it anymore, it's now used by an entire team. It was great that my then-managers saw that I was on to something and let me obsess about it for a few weeks.

I recently spent a few weekends writing static code analysis tools to analyze our codebase. There is some pretty exciting stuff that can be done there. It hasn't paid off quite yet, though I know it will.

Another project I've been juggling with my 'official job' is a nice new logging system with all kinds of cool features. There are so many challenges about making a piece of infrastructure work technically and culturally across a large team, however. It's much easier to estimate how long the technical part will take.

I've also written a bunch of tools that just failed to find any real use. In retrospect, these were mostly vanity projects of some kind, whereas the projects that have worked have always been about scratching an itch that I can see other people have (whether or not they articulate it).

I can think of many things that could qualify at "internal tools." The question is broad and the answers depend on what your business is, who works there, and what they do (yourself included!)

Possible "internal tools" that come to mind are: (1) Analytics (2) CRM-y stuff for the sales and support people (3) Deployment and testing stuff for technical people (4) CMS-y stuff to enable non-technical people to experiment with, or even implement, things that are client-visible. And for technical people to do so more efficiently. (5) Backup, redundancy, disaster recovery etc

If none of these seem more important than others, I guess you need to ask: where is the business now? Where will it need to be in <x> months? How are we getting there? What are the pain points? How can the above improve this (versus implementing new features)?

One cherry-picked example: should technical staff time be spent on improving sales? If sales are your company's limiting factor and you don't have enough salespeople, making them more efficient is really important. If you're still in a soft launch-ish place, then it's not. If you have enough salespeople but not enough leads, maybe mining your own data, or public data, could produce some.

You're generalizing too much. 1 and 2 are actually internal products and should be planned and budgeted out by higher-ups as projects. I have no idea what would constitute #4 the way you describe it, but it sounds like a business requirement that should also be dealt with as a project or internal feature. Lastly, #5 is business continuity and a primary function of IT in general.

This leaves #3 as a sector where the manual/automation see-saw can be improved by tooling. The others are more aspects of business that do not happen otherwise. Tooling is more concerned with easing the jobs of people who implement the thing being streamlined with the tool. In a sales context this would be something like switching to a shared address book or whiteboarding an in/out scheduling chart, both of which they can do for themselves. Interdepartmental deliverables are not tooling.

I think he was actually fair spot on, I work on an "Internal Tools" team, and we do #1 - #4, in addition to maintaining the bug tracker, finance and audit tools, and a translation/review product. #5 is definitely DevOps though.

You both may be right in terms of the semantics and divisions of labor in your respective companies/experiences, but they are more general than the historical sense of tooling in a systems context.

How is this different from any other business decision that you'd make? Estimate the costs, estimate the benefits, estimate the rate of return, and proceed if the rate of return exceeds that of the alternatives.

I agree. Do the math. If you don't have the man power and the math makes sense, contract it out.

My team and I spent a lot of time developing internal tools (and/or integrating them with other tools like CampaignMonitor and stuff).

Throughout our massive amounts of failures, we've learned some things (and some things the hard way): there are basics, and there are nice-to-haves. Get the basics right.

Our basics include an entirely zeroMQ based service-oriented architecture (after 8-9 different iterations over the past 4 years). We spent a lot of time writing a ZMQ based DHT, and service wrappers (kinda like zeroRPC[0]). Other basics include a standardized front-end A/B testing based off genetify[1], email service (with AWS' SES), paypal service, our fair matching services and many others.

The one thing to note is that these basics didn't come overnight though. It came through many many iterations of different products that failed. We merely picked up the components that worked best.

There were things that were nice to have like automated testing and deployment options that we considered. Even using tools like Puppet and/or Chef, we determined, were not quite a good use of our time, considering our release schedule isn't all that crazy, and deployments can be managed by one human being.

But our situation does not apply to you. You need to think about what fits your situation, and your styles. Maybe you are the kind that pushes a release to production servers every 5 minutes. Then yes, having an automated test and build system would be extremely useful.

Start with simple stuff like crons and tiny scripts. Let your internal tools evolve on their own. Don't jump in to write massive automation for your systems. Instead, do it in small parts, just enough to help you.

Hope this helps.

[0] http://zerorpc.dotcloud.com/

[1] http://genetify.com/demo/

Very little. A lot of people automate waaaay to early (it's the engineer in you). Of course this is very dependent on your situation (startup vs agency/shop).

I always suggest a digital assistant first. Pay $4-10 an hour and let someone else do it. Wait until that cost becomes a thorn in your side.

First off, such a great question.

I think there is a danger in trying to make your own life too easy at the expense of making your customers lives easier - especially during a startup period.

I think for tools you have to just do the math. If it takes you 10 minutes once a week and the tool you build takes three hours to develop - then the effort will start paying off in 18 weeks. If it takes you 8 hours to build the tool the you don't break even for 48 weeks (almost a year).

There could be other reasons to build a tool, though. For example the process is error-prone and a tool will ensure it is done properly. Or it enables a non-tech staff to take over something that allows them to help customers better.

Lastly one task is always worth automating - that is data backups.

I'd leave the short term to short sighted people. In chess, in wars, in business, in software, the world belong to those who see far away.

As Napoleon said: l'intendance suivra, which means that short term and ancillary problems will be solved anyway.

Another way to say this is that if you keep the general long term direction, it will be ok to use dirty hacks to fix in fee minutes the short term problems, you will know how to do it in a way that we'll not hinder the long term path.

But make sure your tools and automation are really in line with the long term direction.

For example, building a chrome plugin to speed manual testing of a web app would be wrong, architecting it to be fully unit testable would be right.

I think before you start automating you business and processes you first need to know if your product will get any traction. Spending the time on making sure you have the best possible MVP is the most important thing, IMO. You may be launching a lot of tasks manually and modifying the database directly from the console at first but so be it. The fact is that creating internal tools is very expensive and only indirectly provides benefits to your product.

You'll know when you'll need to build better automation to your product; there won't be enough time to get everything done in a day.

That said, there's a downside to following this approach. When you will need to build automation tools you'll be conflicted with spending your time on automation or on improving your product. Improving your product directly provides benefits, tools indirectly provides benefits. It can be sometimes hard to convince yourself or others that you need to spend x% of your time on internal tools.

Where I work we're at a point where automation is very important. As the CTO of the company I constantly have to challenge the CEO and others that spending money on tools will give us some benefit in long term. The rule (or high level design pattern) that we are trying to implement is for every module of our product, there is a tool to understand the data better for debugging purposes (e.g. a UI to see what step in the workflow the process is in and/or what errors happened, etc), and if data is generated, there's an API/Service to get the data out easily.

I think Yishan Wong's thoughts on this really hits the mark: http://algeri-wong.com/yishan/engineering-management-tools-a...

It is incredibly important to get your tools right so you can operate as efficiently as possible. At my company we didn't get around to this till much later, partly due to the dilemma you mentioned abt focusing on customer-facing software vs. internal tools, and partly due to the fact that the people who would benefit the most from automation (e.g. finance folks, inventory managers, etc) don't have the same obsession with automating away mundane tasks like us engineers do.

This lack of tools hurt us quite a bit once we started scaling, in ways that impacted some customer relationships (because occasionally we didn't invoice or ship correctly, etc). We have since invested significantly in tools, but we should've done this to begin with.

To make the customer vs. internal tools compromise easier, you may even want to get some basic tools in place via contract work/offshoring and then revisit these later once you have the time/resources. The hard part really is determining what to automate, and how much, and those are decisions that you should make. The development itself can be done by someone else.

I see I need to add more specifics about my situation.

I work at a university developing research software (primarily - some is also external customer facing). My customers are internal customers, thus profitability and other business measures are not directly my concern.

The biggest issue I face with this is that my customers are non-technical people, so if I were to tell them that our infrastructure is now software defined and I have implemented analytics that will allow me to optimise the caching layer, they will most likely look at me with a puzzled expression and perhaps wonder what I'm actually doing since those things don't provide direct value to them.

The types of internal tools I am thinking of are things like; analytics, automated deployments, CI server etc.

One comment suggested I do a simple calculation like any other business decision. This is something I cannot for the life of me work out how to do. Internal tools will increase my efficiency in the future, so I guess at best I could estimate a payback time for some of these tools (like automation). Even so, how short a payback makes it worthwhile? For other tools, like analytics, how can this type of calculation be made?

Additionally, there are some things that are the "right way" to do things, such as software defined infrastructure vs managing it manually (which I currently do). Investing time in this won't have any immediate payback, but is nice to have in place in the case of a large failure when I need to recover quickly. These types of value calculations for me are not easy to make.

sourthyme, I like the idea of spending one day a week working on this stuff. Perhaps I need to have a conversation with my bosses/customers about this and try to explain to them the value that internal tooling will bring.

Thanks for the answers so far, please keep them coming.

In my opinion, you should not tell people who do not understand how analytics will improve your caching optimization that you are working on analytics. You should tell them you are working on improving the caching layer. If you sound confident that implementing analytics is required for this job to be done properly they will have to trust you in this, _you_ are the programmer.

You are correct in your assertion that when people ask you what you are doing, they want to know what value you are providing for them. So don't answer with technical details, you already know they're not interested in that :) Whenever a business stakeholder asks you what you are doing, trace back how what you are doing provides value for them and inform them of that.

Note that value does not have to be very direct. Caching makes your application faster, its business value is that the customers are happier when using the product. It is for the marketing department to decide wether that is worth it.

I would not bother too much with figuring out the exact numbers, they are not important. You should for yourself decide wether something is useful or not, if you are an experienced programmer I think this intuition will be right. Almost always, tooling will reduce your technical debt.

Tooling is not only automation, it is also process documentation.

I appreciate this reply a lot. This type of situation is a new one for me, I guess I should just bring up the subject as a whole and see how it goes. I hope I don't run into too much resistance since it will result in a short term decrease in deliverables.

It looks like you're trying to do the right thing here, even though there is no boss or peer who understands exactly what you're doing and can appreciate that fact. Kudos for that :)

To echo tinco's comment, do not try to explain the intricacies of what you're doing to your customers. Put yourself in their shoes and think about what is important to them. Explain just the "end results" of your tooling, emphasizing the points that are important to them and leaving out everything else.

For example, you could say that the caching rework will improve the performance of the system, software-defined infrastructure will ensure minimal downtime in case there's a hardware failure, etc. These are all benefits your customers will understand, even though they may not care or understand how exactly you will achieve them.

Everyone on this thread focus on business decisions should something be automated based on perceived monetary value.

However, there is another important aspect in developing internal tools. In a lot of setups, developers rarely have a chance to investigate new technology, new development approach, different architecture, even different programming language on a day to day basis. Work is mostly done using predefined set of technologies.

For tools, the technology requirements are almost always much more relaxed, so the developers themselves gets to choose and try different approaches. While not every tool might end up being success in strictly measured time/money saved sense, in the long run it will help developers stay up to date with the technology, and even apply lessons learned in the real/product development.

As a bonus, employee retention might improve as people won't get bored so soon if they have a chance to try doing something new from time to time.

As a client-oriented business, I am a proud disciple of the Gospel of Tools.

1. Tools are one of the few institutional advantages that are in the domain of the programmer (rather than the MBA or the sales force). There are many shops out there that suffer from a lack of technical leadership and compete on the basis of (absolutely incredible!) strength of the sales force. Software developers don't win at that game.

They can, however, win at having the best tools. You have to have some unfair advantage, and in most developer-run studios, tools are easier to cultivate than sales methodologies.

2. Customer-facing tools provide customer-visible value. If you're writing error-reporting software that helps debug issues on customer systems, that's providing visible value. If you have client dashboards that show the status of their project and predict ship dates, that's visible value. If your network protocol is faster than REST for your vertical, that means faster software, and that's visible value.

It's all about giving your customers something to hang their hat on to select your shop. If they are getting "good vibes" from the other sales guys, but your software automatically builds them a Powerpoint deck of the project status to present at the weekly update meeting, then there's some class of customers who will prefer working with you, and you are better insulated from competition.

3. If you and your competitors don't do tool investment, then all the boats will rise at the same speed. But if you are doing tool investment, your boat will rise faster. At some point you can do the same job and provide better value than anyone else. When people figure that out, your competitors are forced to first, cut prices to remain competitive, and second, raise expenses to try and achieve tool parity. But you've already got a long head start, so they will either run out of money, license the tools from you, or just focus on the more sales/MBA-receptive customer base, and shrink to a niche offering.

4. As I've hinted at, tools can be valuable and can be licensed to other shops. The only way to really get out of the client "rat race" is to become a product company. Tools are an easy way to make that transition; you have a built-in customer base and an intimate knowledge of the problem domain.

Irritation is a factor. If you sense you're going to be irritated by doing something a lot of times, and it's more fun to develop a tool than to do the tedious thing, then develop the tool.

My general rule of thumb is to tell the client you're going to charge 3x for the hours spent on tool development what you will for the tedious work, and let them decide which they think is in their best interests.

Easy! If this is your company, dedicate as many resources as possible to get you to profitability the fastest.

Would adding better analytics allow you to convert more prospects? Sounds pretty important for getting business, despite it being an internal tool.

If you're pre-revenue, or pre-sustainability, all your efforts need to depend on becoming sustainable. Otherwise, you're making a great product for nobody.

When you really need tools, couldn't survive without them and couldn't buy some 3rd-party so then you have to create some internal piece of software. And it's not a matter of time or resources, just accurately calculate will it help or not and decide.

Of course internal software MUST be developed using the same quality standards as your frontend program (VCS/tests/code review/etc).

As a datapoint: when we (https://circleci.com) did our market research, we estimated that a development team spent about 12% of its time on the parts of testing/deployment that we would replace (mostly setting up a CI server w/ continuous deployment).

For new projects or existing projects that desperately need tools, %80-%90 percent is good. The rest can be used for small changes that have visually large results to keep the uppers happy. After you have a basic tools stated %20 or one day of the week to work on tools is good.

This is the wrong question to ask. The questions to ask are about ROI and opportunity cost. If you spend 6 months on a library that lets you do a 6 hour job in 1, then that's a good investment, if you can afford the up front cost.

Depending on how often you have to do that 6 hour job, of course.

Well, I'd say that if you consider a task to be long, repetitive, time-consuming and something you have to do often, work on something that wil make that process a lot shorter and help you save time in the future.

For a game studio, that's significant portion of your time.

Particularly the larger studios.

In those, the answer to the OPs question, "Percent of time spent developing internal tools?", at least on certain teams, is in excess of 90%.

A part of the consideration should be the cost of buying the tools. If you have to build all your own tools, there may be another element to the analysis.

Initially, about 80%, dropping to much less later on.

imho, internal tools form a big part of your competitive advantage

you could spend all your time building these, and still just have scratched the surface.