Moving Towards Full Stack Automation

In 2015 we launched Awex, the first IPv6-only project powered by Hostinger. It’s fully automated starting from application deployments to network gear. Since that moment we have been automating almost everything what’s possible to avoid manual work, and increase efficiency.

As we wrote in previous posts we use Chef and Ansible for most of the tasks.

Due to huge demand for premium shared hosting we have been shipping an increased amount of new servers in the last few months.

Before having it automated, we had to manually setup servers. This means creating hardware RAIDs, setting IP addresses, installing OS, etc. Fortunately, we decided to stop here, and create a streamline process. We employed PXE for this kind of job.

Why PXE?

• It allows us to fully converge our infrastructure with minimal costs;
• We use phpIPAM to store our inventory information like Chef run lists, environments, IP addresses, MAC addresses, etc. Hence we ended up with a simple solution: single PXE virtual machine per data center serving kickstart configuration files from phpIPAM.

The process itself is trivial:

• Put all the necessary information about the new server into phpIPAM (IP/MAC addresses/hostname/etc.);
• Generate kickstart config (which is also stored in phpIPAM along with the server as additional metadata field);
• Install OS (custom microservice (written with Ruby/Sinatra) downloads kickstart configuration from phpIPAM dynamically for given MAC address);
• After OS is installed, we run full Ansible playbook to converge all roles;
• Later we do not run full converges because it takes much time to complete.

We use incremental converges where we just involved versioning control for roles to speed up the process 10x.

We also considered using FAI, Foreman but this tool looked oover-engineeredfor our simple needs.

Why Network?

This year we are delivering more and more L3 networks across all of our data centers (DC).

We started the new data center in Netherlands which will bring new opportunities for clients sourcing from Europe. This DC is fully L3-only. This means no fucking stretched VLANs between racks, rooms and so on. On that occasion, I want to say that we switched Singapore’s data center over L3 as well. It’s just the right architecture in place. Yeah, I worked in three ISPs over my career and already found that most interesting networking stuff lies in data centers architectures.

We use Cumulus Linux and switched to 40G between leaf and spines. Everything is automated with Ansible. We connect every commodity server to a 10G network using fiber, somewhere copper. The project without issues cannot be successful in the first phase:

• We dealt with some of the hardware issues bootstrapping new data center;
• 40G adapters were not recognized properly – one end expects copper, while other – fiber;
• 10G over QSFP+ using breakout cable;
• Quagga –> FRRouting;
• Migration from L2 to L3 without any downtime.

40G is Something New

root@nl-srv-spine1:~# cat /cumulus/switchd/config/interface/swp1/interface_mode
CR4

root@nl-srv-leaf1:~# cat /cumulus/switchd/config/interface/swp50/interface_mode
XLAUI

It is really weird how it works while both sides have different interface XLAUI (fiber) and CR4 (copper). After few minutes the link is down. Why is the hell CR4 used at all?

I tried to override cable settings with /etc/cumulus/portwd.conf by setting in both cable sides:

[ports]
[cables]
00:17:6a,QSFP-BD-40G=40G-CR4

But no joy.

I did not find any documentation about portwd daemon at all, thus decided to take a look at the source code.

@classmethod
    def decode_ethcodes_base(self, ethcodes, hacks_dict, quirks_dict, eth_map_dict, ext):
        ...
        if ethcodes in quirks_dict.keys():
            quirk = quirks_dict[ethcodes]
            log_throttle('enabling ethcode quirk: 0x%02x -> %s' % (ethcodes, quirk), level='debug')
            # ethcodes with 0 belongs to this test, hence returning 40g-cr4 as default
            return (quirk,)

    @classmethod
    def decode_ethcodes(self, eeprom):
        ...
        # this returns 0
        ethcodes = struct.unpack('B', data)[0]

        if ethcodes == 0:
            ouipn = '%s,%s' % (self.decode_vendor_oui(eeprom),
                               self.decode_vendor_pn(eeprom))

            # skipping this test because certain ouipn is not listed
            if ouipn in self.ETH_CODE_ZERO_OVERRIDES.keys():
                override = self.ETH_CODE_ZERO_OVERRIDES[ouipn]
                log_throttle('enabling zero ethcode override: %s -> %s'
                             % (ouipn, override),
                             once=True, level='info')
                return (override,)

        base = self.decode_ethcodes_base(ethcodes, self.ETH_CODE_SANCTIONED_HACKS,
                                       self.ETH_CODE_QUIRKS, self.ETH_CODE_MAP, 0)
        ...

    @property
    def ethcodes(self):
        if self._ethcode_override_map is not None:
            if self._ethcode_override_map.has_key('port'):
                override = self._ethcode_override_map['port']
                log_throttle('enabling port ethcode override: %s'
                             % (override[0]), once=True, level='info')
                return override
            # we tried to override by vendorpn which is nothing else but '00:17:6a,QSFP-BD-40G'
            elif self._ethcode_override_map.has_key(self.vendorpn):
                override = self._ethcode_override_map[self.vendorpn]
                log_throttle('enabling vendorpn ethcode override: %s -> %s'
                             % (self.vendorpn, override[0]), once=True,
                                level='info')
                return override

The default interface is 40G-CR4 (cannot decode interface information from ASIC), hence we have it. We ordered another kind of transceivers recommended by Cumulus – and it works! CR4 becomes SR4.

Another one interesting problem faced with QSFP+ was using an adapter to connect both ends with 10G. This was needed because our DC provider didn’t have 40G ports to connect to. We use EdgeCore 6712-32X-O-AC-F in our spines layer which doesn’t have 10G ports, hence we moved to 10G over QSFP+ adapter (breakout cable).

2 –> 3

Most interesting part was migrating existing L2 network with almost full three racks in Singapore to L3. I picked using private AS per rack numbering from 65031, 65032, … This architecture allows keeping eBGP rules regardless if it’s an upstream neighbor or just a ToR switch.

Existing infrastructure was built using separate sessions for IPv4 and IPv6 thus this option allowed us to migrate smoothly without any downtime by testing one protocol before switching to another. Before that, we had single exit upstream where we announced our prefixes. After setting up another one (for migration only) we needed to make sure traffic flows as expected to avoid asymmetric traffic using next-hop manipulation, AS prepends and so on.

This is an example of Ansible’s YAML hash to configure peers:

upstreams:
  nme:
    port: swp48
    ip: 10.13.128.46/30
    ipv6: <ipv6>/126
    peers:
      nme4:
        as: 38001
        local_as: 47583
        as_prepend: 47583 47583
        ip: 10.13.128.45
        announce:
          - 156.67.208.0/20
          - 31.220.110.0/24
      nme6:
        as: 38001
        local_as: 47583
        as_prepend: 47583 47583
        ip: <upstream-ipv6>
        announce:
          - 2a02:4780:3::/48
  hostinger:
    port: swp1
    ip: 10.0.31.1/30
    ipv6: <ipv6>/64
    peers:
      hostinger6:
        default_route: true
        as: 65032
        ip: <ipv6>
  hostinger-r1:
    port: swp2
    ip: 10.0.32.1/30
    ipv6: <ipv6>/64
    peers:
      hostinger6-r1:
        as: 65033
        ip: <ipv6>
  hostinger-r2:
    port: swp3
    speed: 1000
    ip: 10.0.33.1/30
    ipv6: <ipv6>/64
    peers:
      hostinger6-r2:
        as: 65034
        ip: <ipv6>
...

This snippet is taken from one of the Singapore’s spine switches. You should notice that we use IPv6-only BGP sessions between internal peers for both protocols and different AS numbers for different racks.

For new deployments, we started using the newest version of Cumulus which replaced Quagga with FRRouting daemon for routing protocols. In general, FRRouting is just a community fork of Quagga. Hence the configuration wasn’t so much different, basically, only paths had to be changed for configuration files. Before provisioning new version in production we test it in the kitchen with the same playbooks and Cumulus-VX. If you push stuff that’s untested you are just blocking your fire escape.

Final Cuts

• Just like a power company shouldn’t change its voltage every year, we believe that our clients should be able to trust our service to be stable;
• Cumulus Linux still remains the best OS for networking;
• Do NOT design network topology according to application needs. Today applications are smart enough to be network agnostic and adaptive to any infrastructure. If you care about your infrastructure do not turn it into unfrastructure.

Hence we are trying to do less manual work while spending our time on improving the quality of the service.