July 18, 2016
4 min Read
July 18, 2016
4 min Read
This article is about how we built the new high scalable cloud hosting solution using IPv6-only communication between commodity servers, what problems we faced with IPv6 protocol and how we tackled them for handling more than ten millions active users.
At Hostinger we care much about innovation technologies, thus we decided to run a new project named Awex that is based on this protocol. If we can, so why not start since today? Only frontend (user facing) services are running in dual-stack environment, everything else is IPv6-only for west-east traffic.
I don’t want to dive into details with this post but I will describe crucial components needed for building this architecture.
We are using pods. Pod is a cluster which shares the same VIPs (Virtual IPs) addresses as anycast and can handle HTTP/HTTPS requests in parallel. Hundreds nodes per pod can handle user’s request simultaneously without saturating the single one. Parallelization is done using BGP and ECMP with resilient hashing to avoid traffic scattering. Hence every edge node is running BGP daemon for announcing VIPs to ToR switch. As BGP daemon we are running ExaBGP and using single IPv6 session for announcing both protocols (IPv4/IPv6). BGP session is configured automatically during server bootstrap step. Announcements are different depending on server’s role, including /64 prefix per node plus many of VIPs for north-south traffic. /64 prefix is specially delegated for containers. Every edge node runs plenty of containers and they communicate each other between other nodes and internal services.
Every edge node uses Redis as slave replica to get upstream for particular application, hence every upstream has thousands of containers (IPv6) as list spanning between nodes in pod. These huge lists are generated in real-time using consul-template. Edge node has many public IPv4 (512) and global IPv6 (512) addresses. Wondering why? To tackle with DDoS attacks. We use DNS to randomize A/AAAA for client’s response. Client points his domain to our CNAME record named
route, which in turn is randomized by our custom service named Razor. _We will talk about Razor on the further posts.
At first, for ToR switches we decided to use OpenSwitch, which is quite young but interesting and promising community project. We tested this OS in our lab for few months, even contributed some changes to OpenSwitch, like this patch. Issued a number of bugs, most of them were finally fixed, but not as fast as needed, hence we postponed experimenting with OpenSwitch for a while and gave Cumulus a try. By the way, we are still testing OpenSwitch in our lab because we are planning to use it in the near future.
Cumulus allows us to have fully automated network, where we reconfigure network including BGP neighbors, upstreams, firewall, bridges, etc. on changes. For instance, we add a new node, then Ansible will automatically see changes in Chef inventory by looking at LLDP attributes and regenerate network configuration for particular switch. If we want to add a new BGP upstream or firewall rule, we just create pull request to our Github repo and everything is done automatically including checking syntax and deploying changes in production. Every node is connected with single 10GE interface using Clos topology.
Here are a few examples of pull requests:
[2001:dead:beef::1]), others do not (
2001:dead:beef::1), or the best are (
# of pkts dropped due to large hdrs:126.
vmxnet3driver and check
vmxnet3_rx_error()to see what buffer length is hitting the queues. That was really disappointing because buffer size was 54 bytes and it wasn’t even IPv4 or IPv6 packet. Just some VMWare underlying headers. Finally, by adjusting MTU for nodes running on ESXi we were able to handle all packets without dropping them.