Manufacturing Code: Devops & CI/CD

Devops & Docker is something of a buzz word - I see it on almost every backend resume that comes across my desk now. I compare and contrast devops in software engineering to my experience in manufacturing

July 15, 2018

One source of constant anxiety when I was working as a process engineer was the availability of raw materials. When and how we were going to receive them, in what quality they could be delivered and making absolutely sure that once they were on site they were navigated to the correct place. Because of the nature of being in a continuous polymer process any variations or failures can result in days of pain and unhappiness and/or millions of dollars wasted. I have some horror stories: tens of thousands of pounds of wrong material being put in the incorrect silo, titanium dioxide tanks being left unattended and overflowing, 200 gallon tubs of oil being punctured, water pipes not being valved off, an owl knocking out the power, etc.

The experience I gained through this and working with others who had been doing this a long time taught me that one subtle but important condition was desired at the cost of almost any other resource: consistency.

Consistency in all things: people showing up to work, raw materials being delivered, process conditions maintained, product going out the door. Consistency allowed us to come in to work, leave at a reasonable time, sleep at night, and enjoy our weekends.

My first manager told me my first week, "It takes a different kind of person to do this work" and he wasn't joking. Out of the 16 junior engineers who were hired the same time I was, half were gone in the first year. Four more the year after that. Almost all of them cited specific events, usually the result of some inconsistency in the process they were managing, as reasons for quitting. While my reasons for leaving were a little different and I had had been relieved of attending to process emergencies I would be lying if it wasn't in the back of my mind. Especially that damn owl.

So after I migrated to the software world a similar pattern emerged (although in most cases less severe). This time consistency was punctured much less often by seemingly random unfortunate events beyond our control, but mostly due to updates to the systems I worked on. Especially updates that made large changes to systems that had not been changed in a long time. My initial research into this subject showed that this was a common phenomenon and that a proposed solution was the idea of continuous integration (CI) & continuous delivery (CD).

This made intuitive sense to me from my background - we had very large, very expensive systems in place to receive raw material, integrate it into our process, and deliver it to our customer. It makes complete sense to me to approach software from this perspective too.

Back then I set up a Jenkins server to integrate my code and manage deployments. It was fairly crude set up. These days I maintain the CI/CD for several projects, even ones I don't directly contribute code to. To some people it seems like a pain and annoyance, but I get a kick out of it and I know the real pain is when they aren't in place or aren't behaving in a consistent manner. We have full pipelines set up on all of the projects I work on now. An example flow is as follows:

  1. Developer makes final commit to feature branch
  2. Linting & Tests are run over code, coverage is published
  3. Merge request is created to deployment branch (test/stage/prod) after #2 is completed with green lights
  4. When merged into deployment branch the new code is packaged into docker containers, tagged, and pushed to AWS ECR, static assets are transferred to S3, slack notifications are sent out
  5. AWS cloud formation changeset is published, docker containers are retrieved from their repository and uploaded to ECS
  6. Database migrations are run and ECS services update their tasks and perform a red/blue deployment resulting in no downtime from the perspective of the users

It's been a breathe of fresh air working in these conditions. We still have some teams that resist our enthusiasm and haven't automated their processes yet but I think we will convert them by the end of the year!