From a Few Hours to a Reasonable Convergence Time
access_time
hourglass_empty
person_outline

From a Few Hours to a Reasonable Convergence Time

Last week we tackled a huge upgrade where we made our current Hostinger infrastructure fully automated.

Hence, after a few days of discussions about what tools to use, we decided to continue with Ansible because it is the simplest solution requiring the least amount of time and effort. What’s more, almost everyone has had at least some experience with it. This took only a few weeks.

In short, we converted our old infrastructure (custom compiled PHP, Nginx, Apache, and more) to a more sustainable approach (CloudLinux CageFS, PHP selector, Openresty, and so on). Since the upgrade, we have been able to perform proactive monitoring which we weren’t able to do before. It allows us to spot the biggest problems sooner.

The early days of the implementation were difficult. Ansible was not horrible, but it’s really not as flexible as Chef which we already use for our internal infrastructure and the 000webhost.com project. We got stuck on almost every second task and didn’t know how the best workarounds. That said, it’s performing well enough for now.

As we’ve written in the previous posts, we use Jenkins for all sorts of deployments. This was also not an exception. Once we had tested the Ansible convergence in a development environment, everything was smooth and fast, and deployments took 3-5 minutes on average. Unfortunately, during the production, deployments took hours.

Performance Optimizations for Ansible

Pipelining + ControlPersist

This was the first thing to enable. After it was turned on, we achieved 2x improvements of convergence time. With some great luck, we reduced the time from 3 h 28 min to 1 h 48 min per build. This was good for the first optimization, but there was still room for improvement. We could reduce this time by running a single Ansible play, but we have a knob as a pre-flight check to deploy changes to the development environment first. If it succeeds, we can continue in the production environment.

[ssh_connection]
ssh_args = -C -o ControlMaster=auto -o ControlPersist=600s
control_path = ~/.ssh/ansible-ssh-%%h-%%p-%%r
pipelining = True

Make Sure to Minimize ‘Changed’ Tasks

Before making any improvements, I suggest turning on profiling for Ansible tasks:

[defaults]
callback_whitelist = profile_tasks

This will let you see the most critical paths such as:

10:56:39 setup ------------------------------------------------------------------ 28.48s
10:56:39 hostinger-cloudlinux : Upgrade alt-python-pip -------------------------- 12.96s
10:56:39 hostinger-cagefs : Remove some default commands from CageFS ------------- 3.60s
10:56:39 hostinger-init : Remove unnecessary packages ---------------------------- 2.49s

Avoid using state=latest if you do not keep your system up to date. This will return almost immediately instead of looking for the newest versions.

- name: Install Apache and dependencies 
  yum: 
    name: "{{ item }}"
    state: latest
  with_items:
    - httpd

If you are using the yum module with external sources like:

- name: Enable remi-repo
  yum:
    name: http://rpms.famillecollet.com/enterprise/remi-release-{{ ansible_distribution_major_version }}.rpm
  when: remi.stat.exists == false

Make sure to check if the repository file exists, because yum downloads it every time:

- name: Check if remi-repo is installed
  stat: 
    path: /etc/yum.repos.d/remi.repo 
  register: remi

Another trick we did involved disabling update for the git module so that it does not check the latest version every time. We also added the creates option for almost all unarchive, uri, shell modules and so on.

After these changes, our build time dropped to 1 h.

ansible jenkins deploy

Conclusion

  • Always use profiling tools if performance matters.
  • Agentless automation tools are slower than agent-based by their nature.

The author

Author

Donatas Abraitis / @ton31337

Related stories

Leave a reply

Please fill the required fields.Please accept the privacy checkbox.Please fill the required fields and accept the privacy checkbox.

Comment

Name*

Email*

Thank you! Your comment has been successfully submitted. It will be approved within the next 24 hours.

Become a part of Hostinger now!