By Cao Duc Nguyen

Source: medium.com/swlh/a-desi…

An overview of the

For years, Netflix has been the world’s best-experienced online subscription video streaming service, accounting for more than 15% of the world’s Internet bandwidth. In 2019, Netflix had 167 million subscribers, adding an average of 5 million subscribers per quarter, and its service was available in more than 200 countries.

Netflix subscribers spend more than 165 million hours a day watching more than 4,000 movies and 47,000 episodes of TV shows. From an engineering perspective, these statistics show how Netflix’s technology team has designed a great video streaming system that is highly usable and scalable to serve customers around the world.

In fact, IT took Netflix’s technology team more than eight years to build the powerful IT system IT is today.

Netflix’s infrastructure transformation began in August 2008, when the company’s data center suffered a Web service outage that shut down its entire DVD rental service for a costly three days. Netflix’s tech team woke up to the need for a more reliable infrastructure without single points of failure. Therefore, the technical team management made two important decisions to move the IT infrastructure from its own IDC to the public cloud, by transforming IT into a microservices architecture, replacing monolithic applications with smaller, easily managed software components.

These two decisions laid a solid foundation for Netflix’s success today.

Netflix chose the AWS cloud to migrate its IT infrastructure because AWS can provide highly reliable databases, massive cloud storage, and numerous data centers worldwide. By leveraging the cloud infrastructure built and maintained by AWS, Netflix avoids the heavy rework of building its own data centers and focuses more on its core business of providing high-quality video streaming experiences. While Netflix had to rebuild its entire technology infrastructure to run smoothly on the AWS cloud, in return, the scalability and service availability of the system improved significantly.

architecture

Netflix operates on Amazon cloud Computing services (AWS cloud) and the company’s internal content delivery network, Open Connect. The two systems must work seamlessly to provide high-quality video streaming services around the world.

From a software architecture perspective, Netflix consists of three major parts: the client side, the back end, and the Content delivery network (CDN).

The client is any supported browser on the user’s laptop or desktop, or the Netflix app on the smartphone/smart TV. Netflix has developed its own iOS and Android apps to try to provide the best viewing experience for every client and every device. Netflix can control its own apps and other devices through its SDK, allowing it to transparently adjust its streaming service under certain scenarios, such as slow network speeds or server overloads.

The back end includes services, databases, and storage that run entirely on the AWS cloud. The back end can handle basically anything that doesn’t involve streaming video. Some of the back-end components and their corresponding AWS services are listed below:

  • Scalable Compute Instances (AWS EC2)
  • Scalable Storage (AWS S3)
  • Business Logic Microservices (Netflix-specific framework)
  • Scalable distributed database (AWS DynamoDB, Cassandra)
  • Big data processing and analysis jobs (AWS EMR, Hadoop, Spark, Flink, Kafka, and some Netflix-specific tools)
  • Video processing and transcoding (Netflix dedicated tools)

The Open Connect CDN is a network of servers called Open Connect Appliances (OCAs) that has been optimized for storing and streaming large size video. These OCA servers are placed within Internet Service Provider (ISP) and Internet Switching Place (IXP) networks around the world. OCA is responsible for streaming video directly to the client.

Playback architecture

When the subscriber clicks the Play button on the app or device, the client talks to the back end on AWS and the OCA on the Netflix CDN to stream the video. The following figure illustrates how the PLAYBACK process works:

Playback Architecture for streaming video:

  • OCA continuously sends health reports on its load status, routability, and available video to the Cache Control service running in AWS EC2 so that Playback applications can update clients with the latest health OCA.
  • The Play request is sent from the client device to the Netflix Play App service running on AWS EC2 to get the URL of the streaming video.
  • Playback application service must determine that the Playback request is valid before a particular video can be viewed. The validation process here will check the user’s subscription plans, video licenses in different countries, etc.
  • The Playback application service talks to the Steering service running in AWS EC2 to get a list of the appropriate OCA servers for the requested video. The boot service uses the client’s IP address and ISP information to determine the best set of OCAs for that client.
  • The client tests the network connection quality of these OCAs from a list of 10 OCA servers returned from Playback application Service and selects the fastest and most reliable OCAS to request video files for streaming.
  • The selected OCA server accepts the request from the client and begins streaming the video.
  • In the figure above, Playback application service, boot service, and cache control service are running entirely in the AWS cloud based on the microservices architecture. In the next section, I’ll introduce the Netflix back-end microservices architecture that improves the availability and scalability of the current service.

The backend architecture

As mentioned earlier, the back end handles almost everything, from registration, login, and billing to more complex processing tasks such as video transcoding and personalized recommendations. To support both light and heavy loads running on the same underlying infrastructure, Netflix chose a microservices architecture for its cloud-based system. Figure 2 shows the microservice architectures that Netflix might use, and I’ve summarized these architectural forms from some online sources:

The client sends a playback request to the back end running on AWS. This request is processed by the AWS Load balancer (ELB).

  • The AWS ELB forwards the request to an API gateway service running on an AWS EC2 instance. The component, called Zuul, was built by the Netflix team to provide dynamic routing, traffic monitoring and security, as well as the ability to recover from cloud deployment edge failures. The request is applied to some predefined filters corresponding to the business logic and then forwarded to the Application API for further processing.
  • The application API component is the core business logic behind Netflix’s operations. There are several types of apis that correspond to different user activities, such as a registration API and a recommendation API for retrieving video recommendations. Here, forward requests from the API gateway service are handled by the playback API.
  • The playback API invokes one or a series of microservices to fulfill the request. The playback application service, bootstrap service, and cache control service in Figure 1 can be considered microservices.
  • Microservices are primarily small stateless programs that can also call each other. To control its cascading failures and gain resilience, Hystrix isolates each microservice from the caller process. The results can be cached in a memory-based cache for faster access to those critical low-latency requests.
  • Microservices can save to or retrieve data from a data store in a process.
  • Microservices can send events to track user activity or other data to Stream Processing pipelines to process personalized recommendation tasks in real time, or batch business intelligence tasks.
  • Data from the stream processing pipeline can be persisted to other data stores, such as AWS S3, Hadoop HDFS, Cassandra, etc.

The architecture above provides an overview of how the various parts of the system organize and work together to stream video. But to analyze the usability and scalability of this architecture, we need to delve into each of the important components to understand how they perform under different loads. More on this in the next section.

component

This section takes a closer look at the components defined in Section 2 to analyze their availability and extensibility. As I introduce each component, I’ll also explain how they meet these design goals. A more in-depth design analysis of the entire system will be carried out in later chapters.

The client

Netflix’s technology team has put a lot of effort into developing client-side applications that run faster and smarter on laptops, desktops, or mobile devices. Even on some smart TVS without a dedicated Netflix client, Netflix can still control the device’s performance through its own SDK. Virtually any Device environment requires the Netflix Ready Device Platform (NRDP) to be installed for the best viewing experience. Figure 3 shows a typical client architecture component.

  • The Client App will divide its connection to the back end into two types: Content Discovery and Content playback. The client uses the NTBA protocol for play requests to ensure higher security at its OCA server location and eliminate delays caused by SSL/TLS handshakes for new connections.
  • If the network connection is overloaded or faulty while streaming the video, the client application intelligently degrades the video quality or switches to another OCA server. Even if the connected OCA is overloaded or fails, the client application can easily switch to another OCA server for a better viewing experience. The client can achieve all of these goals because the Netflix Platform SDK on it keeps track of the latest list of health OCAs retrieved from the Play app service.

The back-end

API Gateway Service

The API Gateway Service component communicates with the AWS Load Balancer to resolve all requests from the client. This component can be deployed to multiple AWS EC2 instances in different regions to improve the availability of Netflix services. Figure 4 shows the open source Zuul, which is an implementation of the API gateway created by the Netflix team.

  • Inbound filters can be used for authentication, routing, and decorating requests.
  • Endpoint filters can be used to return static resources or route requests to the appropriate Origin or application API for further processing.
  • Outbound filters can be used to track metrics, decorate responses to users, or add custom headers.
  • Zuul integrates with the service discovery component Eureka to enable discovery of new application apis.
  • Zuul is widely used for traffic routing tasks for a variety of purposes, such as enabling new application apis, load testing, routing to different service endpoints under heavy load, and so on.
Application API

The application API acts as the business process layer (also known as the orchestration layer) for Netflix Microservices. This API provides the logic to assemble calls to the underlying microservices in the desired order, with additional data from other data stores to construct the appropriate response. The Netflix team spent a lot of time designing the application API component because it corresponds to Netflix’s core business functions. It also needs to be scalable and highly available under high request volumes. Currently, application apis are divided into three categories: Signup API for non-member requests (such as registration, ordering, and free trials), Discovery API for searching and discovering requests, and play API for streaming video and viewing license requests. Figure 5 provides a detailed structural component diagram of the application API.

  • In the latest update to the playback API implementation, the network protocol between the playback API and microservices is gRPC/HTTP2, which “allows RPC methods and entities to be defined through protocol buffers and client libraries/SDKS to be automatically generated in multiple languages”. This change enables application apis to be properly integrated with automatically generated clients through two-way communication and “minimize code reuse across service boundaries.”
  • The application API also provides a common elasticity mechanism based on Hystrix commands to protect its underlying microservices.

Because the application API must handle a large number of requests and construct appropriate responses, its internal processing requires a high degree of parallelism. The Netflix team found that the right approach was a combination of synchronous execution and asynchronous I/O.

  • Each request from the API Gateway service is processed in the application API’s Network Event Loop;
  • Each request will be blocked by a dedicated thread handler that puts Hystrix commands (such as getCustomerInfo and getDeviceInfo) into the Outgoing Event Loop. The outgoing event loop is set up for each client and runs as non-blocking I/O. Once the invoked microservice completes or times out, the dedicated thread above constructs the corresponding response.
Micro service

According to Martin Fowler, “a microservice is a group of small services, each running in its own process and communicating using a lightweight mechanism…” . These small programs can be deployed or upgraded independently and have their own encapsulated data.

The microservice component implementation on Netflix is shown in Figure 7.

  • Microservices can work independently or invoke other microservices through REST or gRPC.
  • The implementation of the microservice can be similar to the implementation of the application API described in Figure 6: requests are put into a network event loop, and results from other invoked microservices are put into a result queue in asynchronous non-blocking I/O.
  • Each microservice can have its own data store and some in-memory cache store for recent results. EVCache is Netflix’s primary choice for microservice caching.
Data is stored

When Netflix migrated its infrastructure to the AWS cloud, it used different data stores (Figure 8) for different purposes, including SQL and NoSQL.

  • MySQL database is used for movie title management and trading/ordering purposes.
  • Hadoop is used for big data processing based on user logs.
  • ElasticSearch provides title search capabilities for Netflix apps.
  • Cassandra is a column-based distributed NoSQL data store that can handle a large number of read requests without a single point of failure. To optimize latency for large-scale write requests, Netflix uses Cassandra because of its ultimate consistency capabilities.
Flow processing pipeline

Stream Processing Data Pipeline has become the Data backbone of Netflix business analysis and personalized recommendation tasks. It is responsible for generating, collecting, processing, and summarizing all microservice events in real time and moving them to other data handlers. Figure 9 shows the various parts of the platform.

  • This state-of-the-art processing platform processes trillions of events and petabytes of data every day. As the number of subscribers increases, it also automatically expands.
  • The Router module supports routing to different data sinks or applications, while Kafka takes care of routing messages and buffering downstream systems.
  • Flow process-as-a-Service (SPaaS) enables data engineers to build and monitor their own custom manageable flow processing applications, while the platform takes care of scalability and operations.

Open Connect

Open Connect is a global content delivery network (CDN) that stores Netflix shows and movies and delivers them to subscribers around the world. Netflix built and operates Open Connect, an efficient network, to get what people want to watch as close as possible to where they want it. To direct Netflix viewing traffic to customers’ local networks, Netflix has partnered with Internet service providers (ISPs) and Internet exchange points (IX or IXP) around the world, To deploy dedicated equipment called Open Connect Appliances (OCA) within the network of these partners.

OCA is an optimized server for storing large video files from IX or ISP sites and streaming them directly to subscribers’ homes. These servers regularly report their health metrics to the Open Connect Control Plane service on AWS, including information such as the best paths they learned from IXP/ISP networks and what videos are stored on their SSDS. In turn, the control plane service automatically directs the client device to the optimal OCA based on the metrics such as file availability, server health, and network distance from the client reflected in this data.

The Control plane service also controls the behavior of filling new files or updating files on THE OCA each night. The fill behavior is shown in Figure 11.

  • When new video files have been successfully transcoded and stored on AWS S3, the control plane service on AWS transfers these files to the OCA server on the IXP site. These OCA servers apply cache fill to transfer these files to OCA servers at ISP sites under the subnetwork.
  • When the OCA server has successfully stored the video files, it will be able to enable peer fill to copy the files to other OCA servers within the same site as needed.
  • OCA can apply a Tier fill process between two different sites that can see each other’s IP addresses instead of the normal cache fill.

Design goals

In the previous section, I detailed the cloud architecture and its components that power Netflix’s video streaming business. In this and subsequent sections, I want to examine this design architecture in more depth. I’ll start with a list of the most important design goals, which look like this:

  • Ensure high availability of streaming services worldwide.
  • Elastic handling of network faults and system interruptions.
  • Minimize stream transmission latency on each supported device under various network conditions.
  • Scalability for high volume of requests.

In the following sections, I examine the availability of the flow service and its corresponding optimal latency. Section 6 is a more in-depth analysis of elastic mechanisms, such as chaos engineering, while Section 7 covers the extensibility of streaming services.

High availability

By definition, the availability of a system is measured by how many times a request is responded to over time, but there is no guarantee that the response contains the latest version of the information. In our system design, the availability of the streaming service is determined by the availability of both the back-end service and the OCA server that holds the streaming video files.

  • The goal of the back-end service is to get the list of healthy OCAs closest to a particular client through caching or the execution of some microservice. Thus, its availability depends on the many components involved in the playback request: load balancer (AWS ELB) _ proxy server (API Gateway Service), playback API, execution of microservices, cache store (EVCache) and data store (Cassandra) :
  • Load balancers can route traffic to different proxy servers to help prevent load overloads and improve availability.
  • The playback API controls the execution of timeout microservices through the Hystrix command to prevent cascading faults from affecting other services.
  • If a microservice’s call to an external service or data store takes longer than expected, it can respond with the data in the cache to play the AI.

When the client receives a list of OCA servers from the back end, it probes these OCAs on the network and selects the best OCAS to connect to. If this OCA is overloaded or fails during the flow processing, the client will switch to another OCA in good condition, otherwise the Platform SDK will request another OCA. Therefore, its availability is highly correlated with the availability of all available OCAs in the ISP or IXP.

The high availability of Netflix streaming service comes at the expense of complex multi-region AWS operations and peacekeeping services, as well as redundancy of OCA servers.

Low latency

The wait time for the streaming service depends primarily on how quickly the playback API can parse the list of healthy OCAs and the health of the client’s connection to the selected OCA server.

As I described in the application API components section, the play API does not wait forever for the microservice to execute because it uses the Hystrix command to control how long it has to wait before getting the results, and when it times out it gets non-current data from the cache. Doing so keeps latency to an acceptable level and prevents cascading failures from affecting more services. If the currently selected OCA server suffers a network failure or overload, the client will immediately switch to another OCA server with the most reliable network connection. It can also degrade the video quality to match that of the network if it sees a drop in connection quality.

Weigh the

After careful consideration, two important tradeoffs have been made in the above system design:

  • Trade consistency for low latency
  • Trade consistency for high availability

The architecture of the system’s back-end services chooses to trade consistency for low latency. The playback API can retrieve obsolete data from the EVCache store or from an ultimately consistent data store, such as Cassandra.

Similarly, the trade-off between consistency and high availability is that the system wants to initiate a response with an acceptable delay without microservicing the latest data in a data store like Cassandra.

There are trade-offs between scalability and performance that are not completely correlated. Under this tradeoff, increasing scalability by increasing the number of instances to handle more load can result in a system not achieving the desired level of performance improvement. This can be a problem for design architectures that do not balance the load well between available workers. However, Netflix solved this contradiction with AWS automatic scaling. We’ll discuss this solution in detail in Section 7.

The elastic

Designing a cloud system that can recover from failures or outages has been Netflix’s long-term goal since the day it moved to the AWS cloud. Some of the common failures that the system has resolved are as follows:

  • Failed to resolve the service dependency.
  • Failure in microservice execution, resulting in cascading failure affecting other services.
  • Unable to connect to an API due to overload.
  • Failed to connect to the instance or server (such as OCA).

To detect and resolve these failures, the API gateway service Zuul provides built-in capabilities such as adaptive retry and limiting concurrent calls to application apis. Conversely, the application API uses the Hystrix command to timeout calls to microservices to stop cascading failures and isolate the point of failure from other services.

The Netflix technology team is also known for its practice in chaos engineering. The idea is to inject pseudo-random errors into the production environment and build solutions to automatically detect, isolate, and recover from such failures. These errors can increase delays in executing microservice responses, kill services, stop servers or instances, or even bring down the infrastructure of an entire region. By purposefully using tools to detect and resolve such failures and introducing real-world production failures into a monitored environment, Netflix can catch such defects early before they cause major problems.

scalability

In this section, I’ll cover the scalability elements of Netflix’s streaming service, horizontal scaling, parallel execution, and database partitioning. Elements such as caching and load balancing also contribute to scalability, as discussed in Section 4.

First, the AWS Auto Scaling service provides the horizontal Scaling capability of EC2 instances on Netflix. As the volume of requests increases, the AWS service will automatically start more elastic instances and close unused instances. More specifically, based on tens of thousands of such instances, Netflix has built Titus, an open source container management platform that runs about 3 million containers per week. Similarly, any component in the Figure 2 architecture can be deployed within a container. In addition, Titus allows containers to operate in multiple regions on all continents around the world.

Second, the implementation of the application API or microservice in Section 3.2.2 also improves scalability by allowing parallel execution of tasks over network event loops and asynchronous outgoing event loops.

Finally, wide column stores (such as Cassandra) and key-value object stores (such as ElasticSearch) also provide high availability and scalability without a single point of failure.

conclusion

This paper describes the overall cloud architecture picture of Netflix streaming service, and analyzes the corresponding design objectives from the aspects of availability, latency, scalability and adaptability to network failure or system outage.

Overall, Netflix’s cloud architecture is proven by its production systems to serve millions of subscribers running on thousands of virtual servers; The architecture also provides high availability, optimal latency, strong scalability, and resilience to network and system failures on a global scale through integration with AWS cloud services.

Most of the architectures and components mentioned in this article were learned from trusted online resources on the Internet. Although there are not many resources on the web that directly describe the internal implementation of these microservices and the tools and systems that monitor their performance, the results of this study can serve as a reference implementation for building typical production systems.

recommended

  • Ten minutes to RocketMQ
  • Spring Boot Core Technical Guide for building a multi-tenant SaaS platform
  • Redis cache and MySQL data consistency solution
  • Nginx traffic limiting configuration
  • Deep dive into Netty, Kafka zero copy technology!

Learning Materials Sharing

12 sets of core technology materials of micro services, Spring Boot and Spring Cloud. This is part of the data catalog:

  • Spring Security authentication and authorization

  • Spring Boot Project practice (background service architecture and operation and maintenance architecture of small and medium-sized Internet companies)

  • Spring Boot Project (Enterprise rights Management project)

  • Spring Cloud Micro-service Architecture Project (Distributed transaction solution)

  • .

Public account background replyarch028Access to information: