Perhaps even more important is the ability to give developers, and managers, visibility into things which were only visible to System Admins in the past.

Why is this server running slowly? The graph shows a spike in disk IO usage. What's the process? The database. What changed at the spike? Adam's production push. Why? The explain plan for the new front page query shows a table scan against a 5m row table.

And done. No sysadmin needed, Adam can now identify and fix his own problems. The management can now watch the graphs of application response time go back down in response to Adam's fix, and everyone's happy.

This isn't theory, it's the example I saw demonstrated at a database convention a few weeks ago. Graphite, logship, nagios, and some development which pulls all of this data together can create some virtually magical systems. Casting devops as a side project for "real" developers limits what could be done to facilitate the creation of real agile development environments.

> And done. No sysadmin needed, Adam can now identify and fix his own problems.

> Graphite, logship, nagios, and some development which pulls all of this data together can create some virtually magical systems.

Who do you think pulls all that data together? DevOps/Sysadmins.

I build and wrap together the tools that make your job easier. And I take pride in that. I'm also on call so you don't have to be.

Relevant XKCD: https://xkcd.com/705/

> Who do you think pulls all that data together? DevOps/Sysadmins.

That's what I was trying to point out. A devops engineer is involved in the creation of these systems, however she does not need to be involved at 3am when the problem is discovered, since Adam can identify his own problem and can verify his fix did what was necessary.

> That's what I was trying to point out.

Ouch, my mistake. In that case, I misread your post, and I apologize.

In my personal opinion this is what DevOps is, in fact "OpsDev" might even be a better word for it in most cases.

It's about carrying out what are traditionally "Operations" tasks (Provisioning, Configuration, Orchestration, Monitoring, ect) using tools and workflows with are traditionally used by Developers (Version Control, Automated Testing, ect)

In large companies this work is often carried out by operations people who have expanded their skill set to include things previously limited to developer roles. In startups this work is often carried out by developers, as in the original rant, who are tasked with maintaining infrastructure as well as developing.

In the traditional development/operations divide, developers often have no access to and little insight into the production environment where their code will be run. As such, they cobble together whatever makes it work, and then "throw it over the wall" to operations, where it has to run in a very different environment. This makes for hellish manual deployments, done on multi-day phone calls with ops staff who don't understand how the code works talking to developers who don't understand the rules and procedures of ops.

If you see places where production code has to run in WebSphere or Weblogic because it's "enterprise", but want developers working in Tomcat to save money, run away.

This leads to situations where operations starts providing "developer resources" in terms of servers, dba support, maven repositories, etc, that turn into a tar pit where development gets stuck in incomprehensible bureaucracy, and "hack" around the system in order to actually get anything done. And then security or someone else in ops finds some little skunkworks dev system and has a fit because developers don't understand what they're playing with - or worse, they find out about the skunkworks because they get pulled in to fix what developers screwed up, with their own managers howling about who's gonna pay for it...

So to me, the beauty of DevOps is getting dev and ops working together in concert from the start to create a consistent development/production environment, where the same automation can be used to maintain both. This is a win for everyone.

And this is why DevOps has gained traction. Automated deployments, continuous integration, and software that functions identically whether in development env or production.

You can wax poetic about how it used to be, or complain about people being overworked all you want, but that's an issue of expectations. Personally I never want to go back to two week "integration" processes and multi-day manual deployments.

DevOps to me is an exercise in not making things more complciated than they have to be on the config/deploy side.

hmm, OpsDev sounds like sloppy dev who carelessly breaks stuff and says Oops!

The sysadmin who can not personally automate his job through programming is not a competent sysadmin.

In between putting out fires and occasionally setting up new systems, this is what sysadmins have been doing long before the word "devops" came into being. Anything you had to do twice, write a script for it.

At least that's what I remember from my sysadmin days: automate the shit out of everything so we could spend more time on Usenet, IRC and reading BOFH stories. This of course to counterbalance having to be on call to fix issues at 2am on a Saturday.

This is a really great point.

If you set up your systems right, they pretty much run themselves.

I've been doing systems contracting for a few years and contracting often turns out to be much better suited for this work than full-time employment (unless you're a company at significant scale).

My typical contract goes like this:

    - gain an understanding of their current systems
    - document *everything* that matters in their systems
    - simplify *everything* that matters in their systems
    - organize *everything* that matters in their systems
    - secure *everything* that matters in their systems
    - automate *everything* that matters in their systems
    - monitor *everything* that matters in their systems
    - train their engineering team on how to run the systems

That's it. If I've done my job right, they'll never need me again unless something exceptional happens.

The best sysadmins make themselves redundant as quickly as possible.

The best sysadmins put the power of the systems in the hands of the other engineers.

The best sysadmins are well rested because the systems they set up are so damn stable and boring.

How do you know you have a bad sysadmin?

    - they are a 'hero' that often rescues the company
    - no one understands the systems but them
    - they are a bottleneck for deploys
    - little is documented
    - little is automated
    - they work long hours
    - they're often frazzled and stressed

A big part of DevOps is saving developers from bad sysadmins. Fortunately, the last few years have been a renaissance for usability in systems tools (like Ansible for instance). This newfound usability for managing and automating systems has finally put the power of systems management within the reach of many more developers.

For more, see my posts on these topics:

Boring Systems Build Badass Businesses https://devopsu.com/blog/boring-systems-build-badass-busines...

Metrics that Matter for your Systems https://devopsu.com/blog/metrics-that-matter-for-systems/