This year we have started using automation for everything as much as we could, including network automation, server provisioning, and application deployments.<\/p>
Every part has its own limitations and requirements. For network automation, we use Ansible<\/a> because it has a flexible template mechanism, many third party modules, is fast, and really fits our needs. Servers are provisioned with Ansible too, but configurations are pushed using our first-class citizen Chef<\/a> while some parts are handled with Consul<\/a> plus consul-template<\/a>.<\/p> Why not run a single tool for everything? Because every tool has its own limitations:<\/p> We make changes on Github<\/strong> by creating a pull request while Jenkins<\/a> pulls these changes. Before Github introduced Code better with Reviews<\/a>, we would use :+1:,:-1: as a comment to approve or reject changes. Thanks to this new feature, we can do it more transparently.<\/p> Our most maintained Chef repository has enabled As explained above, we exploit Jenkins for every kind of automation. In most cases, we have two or three different builds per Github repository:<\/p> Automation is the way we work at Hostinger. We don’t SSH into the server and do not make any changes. We do changes locally on personal laptops first, then later push the code to Github<\/a> as a Pull Request. SSH to server is necessary only for ad-hoc debugging – <\/span>everything else is dynamically adjusted by Chef. So, there is no point to log in. Most of our user-facing servers are identical (depends on the role), so we simply choose one to verify configuration quickly.<\/p> Before any new feature is deployed into our current stack, we have to think about automation first<\/strong>. We have two environments – <\/span> development and production. At first, changes go to the development environment, where we have more or less identical infrastructure (virtual) and changes are seen quickly after a merge. When everything is in order with the development environment, we are free to use the same versions of cookbooks in the production environment. Just lodge another pull request with increased versions.<\/p> When we provision a new server, it is automatically detected by our monitoring platform Prometheus<\/a>. Clusters are reconfigured according to decent cluster sizes and so on, so nothing else has to be done manually. If someone compromises or alters something in the configuration files, everything is reverted back automatically by Network automation is done using Ansible as the primary tool. We use the Cumulus<\/a> network operating system which allows us to have a fully automated network that we can reconfigure, including BGP neighbors, firewall rules, ports, bridges, and more. Nothing is changed directly inside the switch. Cumulus has a virtual instance called Cumulus VX which allows us to converge all changes before pushing them to production. Jenkins builds Cumulus VX convergences locally, applies Ansible playbook, and does tests. If everything is in order, then we are happy, too.<\/p> For instance, we add a new node, then Ansible will automatically see changes in the Chef inventory by looking at LLDP attributes and will regenerate network configuration for that particular switch. If we want to add a new BGP upstream or firewall rule, we simply create a pull request on our Github repo and everything is done automatically, including checking syntax and deploying changes in production. You can find more information about our network stack in the previous blog post<\/a>.<\/p>\n
Require pull request reviews before merging<\/code> protection, which means code review is mandatory. After pulling changes from Github, Jenkins starts building and does the rest. Some builds have multiple Jenkins slaves for executing specific builds, like building LXC<\/a> containers or Docker<\/a> containers, which requires different Linux distributions.<\/p>\n
chef-client<\/code> which runs every 7 minutes in the background. However, some services need to respond faster than every 7 minutes, so consul-template helps here. We have consul clusters per region and consul-template runs as client where needed. For example, we use consul-template for regenerating upstreams for Openresty<\/a>, as it requires nearly real-time operation.<\/p>Other Automation Processes<\/h2>