Last week I received a question about, how to become DevOps expert? I cannot give short answer on that, but I can share my journey, which has some key points needed to be a good DevOps expert. To understand the present, we need to know the history. Why we need DevOps and what it tries to solve.
From Build Server into CI pipeline
At mid 2000 I worked in a company, that had over 30 developers working in the same code base. When we had to deploy our software for customer, we used dedicated build machine to build the source code into working application. We took remote desktop connection into build machine, pull the code from release branch, built the project and copied artifacts into network drive. This was all manual job and took around 30 minutes to complete.
Every now and then we had to do a lot of releases for different customers and it meant, that our build machine had long queue of developers waiting to do their own releases. This could mean, that if I was ~8th in the queue, I would have to wait for 4 hours. Sometimes my turn was around ~18:00, so they were long days.
Around 2015 the automated build pipelines started to appear and we quickly adopted JetBrains TeamCity as our CI system. TeamCity supported multiple build agents, which independently pulled the code from version control, compiled it and pushed the built apps into artifact repository. With this tool we could simultaneously build binaries for multiple releases and our release queue completely disappeared. By working with the build pipeline tools, I understood that the working CI/CD pipeline is critical part of speeding up the release process. With tools like GitHub Actions we can easily create the build machine, that would take us days to configure in the past.
Quality Through Automation
Our release process had few manual steps that needed to be done every time, when new release was created. We had to copy the database scripts from specific folders and put them into release package. We also had to describe the configuration changes in text file. Then the person who installed the new version, run those db scripts manually and applied the configuration changes. This of course lead into situation that the configuration changes were not done correctly (or was missing completely) and the database scripts had lot of issues. The customer database was nothing like expected (on schema wise) or the amount of data was completely different in production than in our test.
After long days (or months…) of fighting with the quality issues, we started to implement automated SQL scripts, which would be run against the test environment every time the deployment was done. Simply by doing that, we reduced SQL script problems near zero. After that all the scripts were tested multiple times, before they were applied into production. Running scripts multiple times also put pressure on developers to optimize their scripts a little bit better.
Lesson here is that we need to automate jobs. When manual work is involved in deployment, it has a huge risk of mistakes over the time. Automate everything.
Correct Tool for the Job
For a while everything run smoothly. We had CI/CD pipeline that compiled apps and pushed them into test environments. We had good punch of build agents that run 247. After few years of running, we started to encounter random problems with our build agents. Disks were corrupting, log files and build binaries filled the hard disks. We had to spend more and more time to manage our own build agents. If you were in a rush to create new deployment and the build agents didn’t work, or most of them were just offline, it started to frustrate.
When the cloud CI/CD pipelines started to become more common. We realized that we need to ditch our on-premise servers and move the CI/CD system into cloud. By using the hosted build agents we could get rid of all the disk errors and simply buy hosted build agents. This again freed more time from our developers. They didn’t have to spend their time tweaking build agents anymore. Not being afraid to evaluate and change the tools saved us a lot of time and pain.
Summary
DevOps expertise is not about specific tool or specific method. Using a tool without understanding why it is needed can be dangerous. As DevOps experts we need to constantly challenge our current way of doing deployments and builds. Is our current process the best way to do it? Does something cause pain and create vasty amount of curse words? Asking questions and taking actions to fix those pain makers makes us DevOps experts.