1. The background

IPv6, short for Internet Protocol Version 6, is a next-generation IP Protocol designed by the Internet Engineering Task Force (IETF) to replace IPv4. The shortage of IPv4 address resources seriously restricts the application and development of the Internet. IPv6 can not only solve the problem of the number of network address resources, but also solve the obstacles of a variety of access devices to the Internet.

At the end of 2017, the General Offices of the CPC Central Committee and The State Council issued the Action Plan for Large-scale Deployment of Internet Protocol Version 6 (IPv6), which calls for accelerating the large-scale deployment of ipv6-based next-generation Internet. The number of active IPv6 users in China will reach 200 million by the end of 2018, 500 million by the end of 2020, and China will be the world’s largest bY the end of 2025, according to the plan.

Under the impetus of technological progress and market trend in the new Internet era, IQiyi actively responds and fully carries out IPv6 special deployment, promotes the evolution of basic network, background service and front-end application to IPv4/IPv6 dual stack support, and contributes to the actual number of users and traffic. In June 2019, it was rated as an excellent case by the expert Committee on IPv6 Scale Deployment.

Iqiyi scheduling system (including Anycast DNS scheduling and HTTP video scheduling) is one of the core CDN systems of IQiyi. With the efforts of iQiyi’s technical team, it has fully equipped with IPv6 service capabilities, supports efficient interconnection of all services with IPv6 basic resources, and speeds up the IPv6 scale deployment process of IQiyi.

2. Scheduling system design

2.1 System Architecture

The whole process architecture of iQiyi CDN scheduling system is shown in the figure below.

Figure 2-1 IPv4/IPv6 scheduling architecture diagram

The following is a description of the roles of each system in the architecture diagram:

  • Client: The playback client provided by IQiyi covers multiple platforms such as mobile phones, tablets, computers, TVS, and web pages. The client first accesses THE DNS to resolve the HTTP video scheduling domain name, then requests the HTTP video scheduling to obtain the CDN download address, and finally downloads video data from the CDN.

  • Probe server: A group of HTTP servers with dual-stack enabled to provide probe services for clients on the entire network and help clients determine local IPv4 and IPv6 availability.

  • Local DNS: DNS server address used by the client, which is provided by the carrier of the user. The client initiates a domain name resolution request to the Local DNS. The Local DNS recursively queries Anycast DNS and returns the result to the client.

  • Anycast DNS: A high-performance DNS system developed by IQiyi, with a single QPS of nearly one million. Deployed based on Anycast technology, it provides highly available DNS services for global users. After supporting IPv6, it can provide IPv6 AAAA record query and resolution.

  • HTTP video scheduling: one of the core CDN systems, it selects the CDN with the best service quality for users according to the operator and region of the source address requested by the client, so as to ensure that users can enjoy the ultimate playback experience.

  • Iqiyi hybrid CDN: including self-established CDN and commercial CDN, iQiyi provides hundreds of millions of users with nearby download services;

  • Resource management platform (Tiangong) : a unified resource management platform, realizing the unified management of resource demand collection, business development, resource procurement, resource deployment, resource management, basic operation and cost calculation, aiming to build an automatic operation and maintenance platform based on process management;

  • Scheduling management platform (Fuxi) : scheduling management platform, managing DNS scheduling and HTTP video scheduling various configurations and policies. The scheduling management platform automatically synchronizes basic information about equipment rooms and servers from the resource management platform, and implements rapid delivery and effectiveness of configurations through the configuration management platform. In addition, the server can be dialed by domain name and IP address to detect the health status of servers on the whole network in real time and automatically offline faulty servers.

  • Configuration management Platform (Fast) : As the management platform of CDN server, it links multiple systems such as resource management platform and scheduling management platform to synchronize basic information such as assets and services. Proxy clusters deployed in multiple carriers + regions ensure the connectivity of CDN nodes in the whole network and provide stable and reliable configuration management services.

  • SDN management platform: a network configuration management platform that supports template-based customized service scenario configuration, batch customized service scenario configuration, and netconf, SSH, and SNMP management modes. Through the generation of configuration architecture and full configuration of network elements, realize the automation construction of core IDC construction.

2.2 System Design ****

Iqiyi’s technical team has done a lot of design work on the client, scheduling and background management systems to support IPv6.

The following is a brief description of the design work of each system module for IPv6 functions:

  • Client: IQiyi provides mobile phones, tablets, computers, TV, web and other broadcast clients, covering hundreds of millions of users. Most clients already support IPv6. When the client starts and detects network changes, it accesses the iQiyi IPv6 detection server and sends HTTP requests through IPv4 and IPv6 to determine the local IPv4/IPv6 network connectivity. If IPv6 is supported, the client preferentially resolves the IPv6 address for video scheduling. When the client requests the scheduler through IPv4 or IPv6, it carries the NET-stack header to inform the video scheduler of the local protocol Stack support.
  • Anycast DNS: Enables dual-stack and supports the query of A records and AAAA records using IPv4/IPv6 DNS. Also provides always-on IPv4&IPv6 authoritative DNS services;
  • HTTP video scheduling: Enable dual-stack and query the carrier and region of the client based on the IPv6 address. Based on the net-stack parameters in the request header and the source IP address, determine whether the client supports IPv4 or IPv6, and return the IPv4 or IPv6 download address to the user in combination with the scheduling policy switch and IPv6 resource load. For ipv6-only clients, return the IPv6 download address. For clients that support dual-stack, the system returns the IPv6 download address if IPv6 resources are available. For some dual-stack clients, when IPv6 resources are limited, the IPv4 download address is returned. For ipv4-only clients, return the IPv4 download address. The process is as follows:

Figure 2-2 IPv4/IPv6 scheduling process

The HTTP request header net-stack is described as follows:

= X: indicates that the client supports only IPv4.

= Y: Indicates that the client supports only IPv6.

= Z: indicates that the client supports double stack;

  • Iqiyi hybrid CDN: including iQiyi self-built CDN and commercial CDN, both need to enable dual-stack and support IPv4 and IPv6. Combined with the scheduling management platform, it can support IPv6 resource mode for some nodes and some commercial CDN to ensure that resources can go online smoothly in gray scale.
  • Resource management platform: To support IPv6, the resource management platform implements IPv6 address segment management, including: Based on the functions of IPv6 address segment entry, automatic allocation, dial-up and retrieval of IDC, in order to complete the allocation of IPv6 addresses on the whole network, an automatic allocation tool based on the command line is developed, and the IPv6 configuration of the entire network is completed in combination with the configuration management platform. During server online (server installation process), IPv6 addresses can be automatically assigned based on the IDC IPv6 status.
  • Scheduling management platform: automatically synchronize server IPv6 configurations from the resource management platform, add DNS scheduling and HTTP video scheduling IPv6 address libraries, and dial the system compatible with IPv6. Add IPv6 switches based on the area and room dimensions to control the pace of IPv6 coming online, and gradually increase the proportion of IPv6 traffic while ensuring user experience.
  • Configuration management platform: After receiving the IPv6 configuration task, each CDN node uses its own asset and service information to perform secondary rendering of the task template, obtain the final configuration command and execute it, and report the modification status after execution.
  • SDN management platform: fully supports IPv6. Through the generation of configuration architecture and full configuration of nes, the core machine room IPv6 configuration can be completed with one click, effectively improving the efficiency of IPv6 machine room resource delivery.

2.3 Preparing Network Resources ****

In order to guarantee the basic network resource requirements for the online IPv6 of all services, the transformation of data center and network double stack is the earliest work link, and the IPv6 work of data center and network capability is basically completed by the end of 2018.

Here are the key infrastructure network capabilities readiness:

  • Data center IPv6 ready: IQiyi data centers include Core, Storage, and Cache. The application, deployment, assignment, and registration of IPv6 addresses for all self-built equipment rooms have been completed, ensuring the security and availability of IPv6 network egress, and fully supporting dual-stack online services.
  • Network capability IPv6 ready: IQiyi has completed the IPv6 upgrade of its own global backbone network QNET, realizing IPv6 interconnection between three major carriers and several basic telecom providers such as education Network, and IPv6 upgrade of QNET-based Anycast system, providing highly available IPv6 service for DNS core business.
  • IPv6 network quality monitoring: Traffic monitoring and network monitoring systems are upgraded to dual-stack, and the differences between IPv6 and IPv4 network quality and IPv6 network quality in different areas are compared and analyzed to guide and ensure the online delivery of IPv6 services.

2.4 IPv6 Node Transformation ****

The key of IPv6 node transformation is the transformation of machine room infrastructure and service, which involves CDN node and core platform.

The following is a brief introduction to the process of node IPv6 transformation:

(1) Manage the IPv6 address segment of the node through the resource management platform, and automatically generate IPv6 addresses for each server and network device of the node;

(2) The SDN management platform obtains IPv6 information of network devices in nodes from the resource management platform. By generating configuration architecture and full configuration of nes, the SDN management platform realizes the delivery and implementation of IPv6 related configurations in the core machine room with one click;

(3) Create a task template on the configuration management platform, automatically deliver IPv6 configurations and software configurations to the server, restart the network, upgrade software configurations smoothly, and support IPv6 services;

(4) If the service supports IPv6, register with the scheduling management platform to inform it of IPv6 service capability;

(5) The scheduling management platform automatically adjusts the scheduling policy. When IPv6 is enabled in the area, IPv6 traffic is automatically entered;

The overall flow chart is as follows:

Figure 2-3 Node transformation process

By iQIYI CDN perfect supporting platform, whether in the hardware resources, software or services management, support for IPv6 is fully automated, particularly in basic network equipment configuration, operating system support, IPv6 support IPv6 resources management, service, service registration and automatic scheduling, etc have been automated management and online.

2.5 Client Scheduling process ****

The following is the sequence diagram of IPv4 and dual-stack clients’ access to DNS, video scheduling, and video CDN. It can be seen from the diagram that IQiyi DNS, HTTP video scheduling service, and video service all support IPv6 and can return IPv4 or IPv6 resources according to different client request modes.

Figure 2-4 IPv4/IPv6 playback process

3. Smooth the line

The previous part introduces the architecture and flow of IPv6 service scheduling in detail. This part introduces the key steps for the system to go online, including the IPv6 configuration support at the operating system and software level and how to smoothly go online on the scheduling management platform.

3.1 System Configuration ****

System-level configuration includes kernel adjustment and IPv6 configuration. The IPv6 configuration can be enabled as follows:

(1) ENABLE IPv6 in GRUB configuration

Delete ipv6.disable=1 from the /etc/default/grub configuration file

(2) Re-generate GRUB and configure it

Run the grub2-mkconfig –output /boot/grub2/grub. CFG command to generate a new GRUB configuration file

(3) IPv6 global switch configuration

Added NETWORKING_IPV6=yes in the /etc/sysconfig/network configuration file

(4) Configure an IPv6 address for the NIC

Update the /etc/sysconfig/network-scripts/ifcfg-ethx configuration file to add the following options

IPV6INIT=”yes”IPV6ADDR=” Allocated IPv6 address “IPV6_DEFAULTGW=” Allocated IPv6 gateway”

(5) The IPv6 address takes effect

In CentOS 7, you need to restart the system to ensure that the IPv6 address takes effect. In CentOS 6, you only need to run the service network restart command to make the IPv6 address take effect

(6) Test network connectivity

You can run the ping6 command to test the connectivity between the IPv6 address and the external IPv6 network

A complete command line tool has been developed to automatically obtain and configure IPv6 information.

The following kernel parameters also need to be adjusted during system initialization:

The kernel parameters Parameters of the function
net.ipv6.conf.bond0.accept_dad IPv6 DAD (Duplicate Address Detection) is used to Detect whether other nodes use this Address. For the NIC interface that is assigned a static unicast address, set interface parameters to 0 to improve system initialization speed.
net.ipv6.route.max_size The kernel parameter net.ipv6.route.max_size is the maximum number of ipv6 route cache entries. In CentOS 7, the preset value of this parameter is too small. This parameter can be set based on the actual situation of the production system or adjusted to the preset value of net.ipv4.route.max_size.
net.ipv6.conf.all.disable_ipv6net.ipv6.conf.eth0.disable_ipv6 IPv6 is enabled by default at the global system level. You can also adjust the kernel parameters of network interfaces by setting them to 0 to ensure that IPv6 is enabled.

For the above o&M work related to system kernel parameters, network configuration, and automatic IP address assignment, we use our self-developed configuration management platform and have realized automatic task templates. For existing devices, the system automatically obtains IP addresses from the resource management platform and automatically configures them. For new devices, IPv6 addresses are automatically assigned and related system configurations are automatically adjusted.

3.2 Service Software Configuration

(1) Enable IPv6 port listening for the scheduler and video Cache server

The scheduler and video Cache are based on the development of Nginx, online environment in the server{} module to add IPv6 listening configuration example as follows:

server{listen [::]:80 backlog=8192; … }

(2) Enable IPv6 port listening on the DNS server

DNS developed by IQiyi is an open source FRAMEWORK based on ACL [1]. In order to support IPv6, ACL framework has made a lot of optimization in basic modules and configuration. IPv6 related configurations are enabled as follows:

service hpdns{master_disable = no; Master_service = | # 53 to monitor all 53 ports network address… }

3.3 Service cuts ****

With iQiyi scheduling management platform, IPv6 traffic can be flexibly adjusted and controlled. For managed video IDC nodes, IPv6 service support of IDC can be enabled by setting the option to enable IPv6 access of video services at IDC node level:

The scheduling management platform supports IPv6 by user area and carrier:

In the actual process of promotion, gray scale cutting is also achieved by configuring the two dimensions of user area and scheduling resource (IDC) to ensure that user experience is not affected and there is no perception to users.

4. Summary and outlook

After more than a year’s efforts, iQiyi scheduling system has been fully equipped with IPv6 service capabilities, and is gradually increasing the coverage of IPv6 users by region and carrier. By the end of June 2019, the number of IPv6 monthly live users of IQiyi reached 100 million, and the peak bandwidth reached hundreds of gigabytes.

In the future, we will continue to promote the deployment of IPv6, steadily increase the proportion of IPv6 bandwidth, make full use of the advantages of IPv6 technology, and bring users a better broadcasting experience. At the same time, we will promote IPv6 network deployment and system architecture transformation to further improve user experience and reduce costs.