How we’ve managed the process of moving from DevOps to GitOps for a midsize company

Our team always strives to make the work as transparent as possible for clients. And one of the possible ways to ensure transparency is to support configured CI/CD environments. Though this approach is a standard for the industry, its implementation, like everything else in IT, can be achieved in thousands of ways. Here, I would like to share some insights on how exactly we’ve managed its implementation, what lessons we’ve learned in the process, and why we are fixing everything over now.

First, I would like to expand on the approach that we have been using for the last five years. It’s important to note that the technology stack we work with is usually limited to Laravel, Django, Nest, React, NextJS, and Angular. Also, we seek to work only with cloud solutions, such as AWS, Azure, or GCP. And since we use Google Workspace, we decided to continue with GCP.

Once we’d decided on a cloud solution, the next step was to select tools and technologies for automating CI/CD processes. Here’s what we chose:

• K8s as a container orchestration system for automating software deployment
• Helm as a package manager
• GitLab CI, as we use GitLab for all projects

Now let’s move to the implementation process itself and take a closer look at how exactly everything works.

I would like to refer to Laravel projects as an example. Let’s say our task is to create an API for a client application.

We have a Helm chart for each framework and tool we use, for example: ronasit/laravel.

All these Helm charts are integrated into project template along with .gitlab-ci.yml, and once you do the first commit to our GitLab project, all of the CI/CD magic happens. Within a few minutes, the project becomes available on our subdomain dev.api.project-name.ronasit.com. The charts also deploy all the entities needed for the project. Here, it’s a database and a Redis.

Thus, it took us only 5–7 minutes after the developer had made the first commit to the GitLab project to receive a fully deployed and ready-to-development environment. And since the documentation is generated automatically by default, and the project template already contains a set of basic functionalities, such as login, registration, profile, media, etc., we can focus on other parts of the project like frontend or mobile.

In 2022, this approach is still relevant as it maintains a high level of automation and makes things easy for the whole team.

With this approach, we wanted to achieve maximum automation of the environment deployment and ensure transparency for the client. And, in general, we succeeded, as in our team with more than 70 people working on more than 25 projects at a time, there is only one DevOps engineer.

Without taking into account the stages of research, verifying technologies, and polishing basic Helm charts and Docker images, the approach mentioned above has been working fine for five years.

And then full transition to the cloud happened. It became a milestone for the technological development of our company. And it affected other areas of the working process, even those we did not think of at the very beginning. For example, CI and CD became two separate processes, making it possible to focus on the development of each separately. It led us to create a tool for auto-generating documentation, based on the results of running tests.

In 2022, given the latest trends in the field of technology, it became absolutely clear to us that it was time to move on. And you may wonder in what direction we decided to move. What else can be automated? Well, the answer is simple. Even now the process still requires a DevOps engineer who is responsible for setting up projects, providing access rights to developers, updating existing solutions, and much more. And as it turned out, this part can also be automated. At least, we can try to do it. The transition to a GitOps approach is now in full swing, but even after only four months, we have found many aspects that can be done in a more efficient, fast, and accurate way. For example, we still use both K8s and Helm, but from now on we are also working on our projects operator. This made it possible to store all the charts in one place, set up versioning, build an update system, and reduce the application description to a few lines of code.

Now, once we make any improvements to the chart, all already running applications are automatically updated to the latest version. But what is more important, it helps us manage resources efficiently. We’ve selected those resource quotas that are essential for the efficient operation of all types of applications in our stack.

Another important result is that we improve the security of our projects. First of all, we managed to set up external-secrets for all projects and to move the secrets to the external secure storage. Also, we changed the way of getting access to the database, making it available through BASTION. It means that the database doesn’t have any open ports from the outside that can be used to establish the connection, and all requests go within the private network.

From now on, we use the IaC approach with Terraform to describe our infrastructure. To speed up development, we wrote our own module, that enables deploying an infrastructure similar to ours in a couple of minutes.

We also started using Argo CD, and now, this tool is responsible for deploying the application to the cluster, as well as for setting up access within it.

Once having received access, the developer can track the status of the application at any time, check if something went wrong, or get access for debugging. Moreover, we have completely redesigned the Gitlab CI system, and now it’s true CI. Once you commit to the desired branch, CI runs tests, validates the code, and builds the application, while Argo CD is solely responsible for deploying the application.

Of course, we are at the very beginning of the journey, but even now, it’s already clear that working with the environments has become much easier for the team. Moreover, we don’t have a DevOps engineer role anymore, even though the team has grown, as well as the number of projects. However, we spend no more than 10–20 hours a week working in this direction.

Back in our DevOps days, the level of standardization was significantly lower since DevOps took standard solutions and adapted them for each project. It resulted in a lack of understanding of the final price for the DevOps environment and services, which, in turn, forced us to charge an hourly rate from the client.

Having analyzed more than 300 projects we’ve been working on for the past seven years, we’ve managed to identify the most effective solutions for every project type. And instead of providing support for each project separately, we decided to focus on the development of a single solution that we can provide to the customers. And we managed it.

Now, the advantages of this approach are clear to us. Here they are:

• We can easily estimate the price for the entire period of using the environment. As I already mentioned in Part 2, the use of an operator gives us the ability to set resource quotas for each project, which means that we can calculate the cost and offer our clients the best price for the cloud solution that suits their needs perfectly.
• As it’s out of the box solution, there is no need for the client to worry about things like updates, best practices, scalability, etc. — we take care of everything.
• We work with the same stack of technologies, and the deployment of all projects is based on the same charts with the operator responsible for the process. There is no need for a DevOps engineer role anymore, as all support requests are processed with the rest of the tasks. The client, in turn, receives the most stable and secure environment, as well as support throughout the development.
• Since we are responsible for the solution, we guarantee 99% uptime for any environment.

However, we do not intend to stop here. Our plans for the next iteration are the following:

• Add support for other popular Django and NestJs frameworks
• Cover all charts with tests
• Improve our work with secrets management
• Add support for HPA
• Port the solution for AWS and Azure

Lastly, I want to remind you that all our solutions are open-source and available on GitHub. Both operator and charts can be reused, and the entire cloud solution can be recreated using our terraform-factory.

More content at PlainEnglish.io.

Sign up for our free weekly newsletter. Follow us on Twitter, LinkedIn, YouTube, and Discord.

Interested in scaling your software startup? Check out Circuit.