DevOps, or The Lost Lessons of the Industrial Revolution
What the industry calls a writer of software has changed over the course of my career.
When I started in the early 1990s a writer of software was usually called a “Computer Programmer.” You programmed a specific type of computer hardware, such as the DEC VAX, telling it what you wanted it to do.
By the mid- to late-1990s, this became “Software Developer.” The emphasis shifted away from the computer hardware, to writing software that was more a standalone entity that could run on multiple platforms. This was the dawn of Java, software that ran on a “virtual machine” that theoretically could be realized on many different actual computer hardware types.
By the 2000s, it was “Software Engineer,” which was more of the same, but perhaps sounded like a more serious discipline. After all you needed a Computer Science degree to be a Software Engineer, so you are more or less a scientist, and not some random music major who dabbled in computers.
Since the mid-2010s, I have been known as a “DevOps Engineer,” a rather ugly term. It’s a combination of Developer, plus Operations, plus Engineer, and reflects that we are no longer just developers, we also do operations. But in reality the title DevOps leaves off a number of sub-disciplines for which we are also responsible, such a Quality Assurance, Release Engineering and Technical Writing (these categories are described in more detail below.) Thus the current software industry standard calls for developers to also do five or so additional specialties.
I have been in startups where I was one of two or three developers, and by necessity we all had to do all of the technical specialties. That makes sense for a startup. But why do large companies with hundreds or even thousands of developers choose to abandon specialization, and make all developers more or less interchangeable units? Does this work, or is it a regression from centuries of experience in how to organize workers to produce complex outputs?
The Industrial Revolution, as Told by a 5th-Grader
In the 5th grade I was taught that the key invention of the Industrial Revolution was specialization of workers and uniformity of output of every component of a product. The example I remember was the production of rifles. Prior to the Industrial Revolution a town might have several gunsmiths, each of whom had their own shops where they performed all the tasks in the production of the rifle from start to finish, with the assistance of apprentices. Each rifle was a one-off production, with parts that were generally not interchangeable.
With the Industrial Revolution the multiple gunsmith shops were replaced by a single factory populated by specializing workers, none of whom necessarily had the skill of the gunsmith to create a rifle from end to end by themselves. So one worker might have training as a carpenter and would produces the stocks, another might be a lathe operator and creates the barrels, and others might be responsible for assembly of the parts. Each part is produced to a uniform specification such that parts for the same model are interchangeable. The result was that the factory had a much higher output of rifles than individual gunsmiths, of a more uniform quality and better repairability.
I myself directly witnessed this kind of process. In the summer of 1986, in the twilight of industrialization in America, I worked on an assembly line in an air conditioner factory. My first job on the line was to assemble the blower units, which involved screwing together pieces of sheet metal and was probably the simplest task in the assembly. Other workers specialized in different tasks of increasing complexity, such as those who soldered the electronic circuit boards, or the brazers who combined the internal piping into a single airtight system, or the machinists who milled the most complicated metal parts.
At the end of the line were the quality assurance checkers, a highly skilled job usually held by someone knowledgeable about every part of the machine. They were able to determine where the fault lay in a failed unit and to send it back down the line to the appropriate worker who produced the fault.
At the top of the heap were the supervisors, who had deep knowledge and actual experience of every aspect of the production of the air conditioner. They were able to fill in anywhere on the line if needed, and often did to help keep up the pace.
The executives of the company (few in number) were always around, popping in to watch the line throughout the day.
Software Specialization Mirrors the Industrial Revolution
When I started out in a software company in the early 1990s there were comparable specializations of the workers, based on training, work experience, general skill level and personal predilection. The specialties I saw were:
Programmer – Wrote lines of code to produce a working software component or system.
Build or Release Engineer – Took the output of the Programmer in the form of multiple code files and transformed it into a package that could run on a computer to perform the intended work of the system. This was often an entry position for a more junior person who worked up to Programmer.
Quality Assurance Engineer – Validated the functioning and performance of the built components of the system.
Technical Writer – Documented the system in words and diagrams, for internal users or external customers.
Operations Engineer – Monitored the computers and networks that the company used to produce its software, and also maintained any customer-facing systems. For shipped software they would work with the customers to ensure the software was installed correctly and worked as expected.
Architect – Maintained a big picture of the design of the system. Programmers also did design work, but the Architect ensured that their designs broadly fit in the overall plan.
Technical Lead or Manager – This job most closely aligned with the supervisor role in the air conditioner factory. They guided the work of the Programmers, but importantly were Programmers themselves at one point, and were capable of doing all of the job of those they managed. The difference between the two was that a Technical Lead only managed the work of the Programmers in their group, while the Managers also knew their salaries.
Product Manager – Knew the market and the customers well enough to decide what the software teams should build. Generally a non-technical role, but they needed to understand software well enough to know what was possible to build.
Project Manager – Tracked the tasks of the software teams. Also generally non-technical, but needed to know the lifecycle of software to understand how the tasks hung together.
Those were the broad roles, though the job titles varied from company to company, and sometimes you might not see every category represented. The system worked because it took the best that each person could offer and let them focus on that. So a Tech Writer was mostly a writer, but also knew enough about software to describe it, while a Programmer knew a ton about software, but might not write English so good. Or a Programmer, based on education and experience, would have good general knowledge of computer hardware and networking, but an Operations Engineer knew the nitty-gritty of particular combinations of computer hardware, operating system and networking to a deeper level. As a result they were much more adept at troubleshooting problems with running systems and arrived at solutions more rapidly.
Today’s Software Despecialization
The situation today is that basically all these specializations have been rolled into one: DevOps Engineer.
A few snippets from the Wikipedia page about DevOps:
“The DevOps movement started to coalesce some time between 2007 and 2008, when IT operations and software development communities raised concerns about what they felt was a fatal level of dysfunction in the industry.”
[I myself don’t recall fatal dysfunction in the industry as a software engineer in 2007, but I surely do feel that after 18 years of DevOps.]
“They railed against the traditional software development model, which called for those who write code to be organizationally and functionally apart from those who deploy and support that code.”
[One wonders who the railing “They” are? Could it perhaps be executives trying to save money? More on this later.]
Maybe in its original conception DevOps really did mean just development plus operations, and maybe it was a useful move to make. But what DevOps means in practice is the following: I, and every other DevOps engineer in my organization, basically do all the specialties listed above, i.e.:
Programmer – Yes, we still write lines of code, though this is 20% of the job at best.
Build or Release Engineer – We manage the code files, assemble them into releases, then deploy them to serve online customers
Quality Assurance Engineer – We are responsible for QA’ing our own code, mostly via unit tests (programs whose job is to probe the main program and verify it works as expected). We no longer have dedicated QA staff. This removal of the independent verification of the developers’ work by a separate team of people is one of the biggest disasters to hit the industry, and will receive its own essay in the future.
Technical Writer – We write the documentation describing our own software. The writing ability of programmers is highly variable, as well many do not have English as their first language, with the result that the documentation is frequently not useful to an outsider to understand the system. Documentation is mostly a checklist item that the Agile software methodology requires, and is usually treated as such.
Operations Engineer – The Ops of DevOps. We monitor our software components as they serve customer traffic, using a variety of tools and dashboards. If something goes wrong we troubleshoot and fix it (though at our company we are lucky that we do still have dedicated systems operations teams that do this work for the general infrastructure). Operations is a complicated specialty involving deep knowledge of computer hardware, networking and operating systems. Traditionally software developers did not have the knowledge to the same level as an operations person. Yes, you can force developers to pick up this knowledge, but they won’t learn it to the same level as a person who has spent decades working on these systems. The result is that a developer as Ops is almost always less efficient than an old Ops person.
Architect – There are still architects, but the volume of software that we are tasked with designing is such that we don’t have enough architects to review all the designs in detail. Again, a good developer is quite capable of architecting and designing their own software, but what is lost is the oversight function of the standalone Architect, with the result that different teams will solve the same problem in different ways, making the overall system more difficult to maintain and of generally poorer quality.
Technical Lead or Manager – Senior DevOps engineers often perform the technical guidance function of the traditional software development manager. They often devise work for the more junior developers, then track the work and ensure that it is completed correctly.
Product Manager – We do have many dedicated Product Managers, but we are still asked at times to come up with ideas for things for the team to work on.
Project Manager – We also have many Project Managers and trackers in general, but we DevOps are still heavily involved in project planning and execution tracking. As part of the SAFe process (Scaled Agile Framework, a topic of a future essay) we are forced to participate in a 3-day planning event once a quarter, and that itself is preceded by multiple meetings in the weeks leading up to the event, meetings where we literally plan to plan. Then every day of the quarter we are required to participate in at least one group project tracking meeting.
Despite all these hats we wear, we don’t wear any one of them for long—like, I don’t dedicate one day out of five to QA tasks. Instead all these various tasks are interspersed throughout the day, and every day features nearly all of them in varying degrees.
Maybe the original inventors of DevOps, the ones who railed at the separation of development and operations, would be appalled by how things turned out. Or maybe not. Could the concept of “DevOps” actually be a nice-sounding methodological cover for executives whose only goal is to increase per-worker productivity?
Who Benefits?
Who benefits from DevOps? Certainly not the DevOps Engineer, who is forced to context-switch between much different tasks, from the idiotic to the sublime, from the most routine button-pushing to the most complicated brain work, multiple times a day. And since you don’t have enough time to specialize you do most everything to a “good enough” level. Obviously, work quality suffers as a result.
But to the executive the concept of an interchangeable “Generic Techie,” a worker capable of doing any one of the mysterious technical tasks of the organization, is appealing. At my company we were told explicitly that we needed to become more “T-shaped,” by which they meant that each of us should have a broader (but not necessarily shallower) knowledge of a wide range of specialties. (I am not actually clear on how the form of “T” represents that concept, but that was the jargon used.)
If you are an executive planning to increase—or decrease—staff, having the unit of Techie is quite useful. If you’re planning to hire you don’t need to worry about how many of each old category of tech worker you needed—how many developers, how many QA you needed to cover them, etc.—instead you could just hire X number of Techies—DevOps Engineers—however many your budget allowed.
And on the flip side, imagine how much trickier layoffs were in the 1990s, where the executives had to consider whether each of the specializations would be adequately represented in the post-layoff roster. What if you had only one Technical Writer? That person could never be laid off, unless the company was on the verge of closing.
The generic DevOps Engineer is a boon in this situation. And the transformation of an organization into DevOps somehow results in a lower technical head count, e.g. if you started with a total of 100 people doing development, QA, technical writing, etc., after converting them into DevOps somehow there remains only 60 DevOps engineers afterward, still producing the same amount of work.
The Two Dentists
Picture a dentist’s office staffed by a dentist, a hygienist, a receptionist and a medical biller. Now imagine the same office staffed by two dentists only, who throughout the day trade off the reception duties, the billings, the teeth cleaning and the actual dental work. Could such an office work?
Or course it could. One of the dentists would greet the patient at reception, then take X-rays and clean their teeth, then fill their cavity, interrupted from time to time to run to the front desk to greet the next patient and take down their updated insurance info. The other dentist does the same tasks, performing half of the overall work. If either have an empty block of ten minutes during the day they can interact with insurance companies and send out bills to their patients. If they don’t, they’ll do the billing after hours.
Will they be able to serve all their patients in some capacity? Probably.
Will the quality of care be high, and would you trust your teeth to them? Unlikely.
That’s what DevOps is like.
A return to the specialization that the software industry once understood was necessary would be a big first step toward improving the morale of the workers, and the quality of their output in general.