This article is published by netease Cloud
1.Flink architecture and characteristic analysis
Flink is a fairly old project, started in 2008, but has only recently gained attention. Flink is a native stream processing system with a high level API. Flink also provides apis to do batch processing like Spark, but the fundamentals of the two processes are completely different. Flink treats batch processing as a special case of stream processing. In Flink, all data is considered a stream, which is a good abstraction because it is closer to the real world.
1.1 Basic Architecture
The following describes the basic architecture of Flink. The architecture of Flink is similar to that of Spark, which is based on the master-slave style.
When the Flink cluster starts, a JobManger and one or more TaskManagers are first started. The Client submits a task to the JobManager, which then dispatches the task to each TaskManager for execution. The TaskManager then reports heartbeat and statistics to the JobManager. Data is transferred between taskManagers in the form of streams. All three are independent JVM processes.
Client is the Client that submits jobs. It can run on any machine (connected to the JobManager environment). After the Job is submitted, the Client can end the process (the Streaming task) or wait for the result to return.
JobManager is responsible for scheduling jobs and coordinating tasks to checkpoint, similar to Storm’s Nimbus. After receiving resources such as jobs and JAR packages from clients, the system generates optimized execution plans and allocates them to each TaskManager based on Task units.
TaskManager starts with a number of slots. Each Slot can start a Task, which is a thread. The system receives tasks to be deployed from the JobManager. After the deployment starts, the system establishes a Netty connection with the upstream device to receive data and process the data.
JobManager
JobManager is the coordinator of the Flink system. It receives Flink jobs and schedules the execution of multiple tasks that comprise the jobs. In addition, JobManager collects Job status information and manages the TaskManager in the Flink cluster. JobManager is responsible for various management functions. The events it receives and processes mainly include:
RegisterTaskManager
6. Initials Geregistration The TaskManager will register with the JobManager at Flink cluster startup and, if successful, the JobManager will reply to the TaskManager with the message.
SubmitJob
The Flink program internally submits the Flink Job to the JobManager through the Client. In the message SubmitJob, the basic information of the Job is described in the form of JobGraph.
CancelJob
CancelJob CancelJob CancelJob CancelJob CancelJob CancellationSuccess CancellationFailure CancellationFailure
UpdateTaskExecutionState
TaskManager asks JobManager to update the ExecutionVertex state in ExecutionGraph, and returns true if the update succeeds.
RequestNextInputSplit
A Task running on TaskManager requests the NextInputSplit to process, and returns NextInputSplit on success.
JobStatusChanged
ExecutionGraph sends this message to the JobManager to indicate that the state of the Flink Job has changed. For example, RUNNING, CANCELING, and FINISHED are all involved.
TaskManager
TaskManager is also an Actor, which is the group of tasks that perform Flink jobs on the Worker that is actually responsible for performing the computation. Each TaskManager manages resource information on its node, such as memory, disks, and networks, and reports the resource status to JobManager upon startup. TaskManager can be divided into two phases:
Stage of registration
The TaskManager will register the initials with the JobManager, send the RegisterTaskManager message, wait for the JobManager to return the initials geregistration, and then the TaskManager can initiate the process.
Operational stage
In this phase, the TaskManager can receive and process task-related messages, such as SubmitTask, CancelTask, and FailTask. If the TaskManager cannot connect to The JobManager, the TaskManager loses contact with the JobManager and automatically enters the “registration phase”. After the registration, the TaskManager can continue processing task-related messages.
Client
When a user submits a Flink program, a Client will be created first. The Client will preprocess the Flink program submitted by the user and submit it to the Flink cluster for processing. Therefore, the Client needs to obtain the address of JobManager from the configuration of the Flink program submitted by the user. Establish a connection to JobManager and submit the Flink Job to JobManager. The Client will assemble a JobGraph from the Flink application submitted by the user and submit it as a JobGraph. A JobGraph is a Flink Dataflow that consists of multiple JobVertex daGs. A JobGraph contains the following information about a Flink program: JobID, Job name, collocation information, a set of JobVertex, and so on.
1.2 Yarn Layer Architecture
The yarn-level architecture is similar to spark on YARN mode. In both scenarios, the Client submits the App to RM for running. RM allocates the first Container to run AM. Note that the YARN mode of Flink is similar to the cluster mode of Spark on YARN. In cluster mode, the dirver runs as a thread in AM. In Flink on Yarn mode, JobManager is also started in a Container to schedule and allocate tasks similar to the driver. Yarn AM and Flink JobManager reside in the same Container. In this way, AM can know the address of Flink JobManager so that AM can apply for a Container to start Flink TaskManager. After Flink successfully runs on the YARN cluster, the Flink YARN Client can submit Flink jobs to Flink JobManager and perform subsequent mapping, scheduling, and calculation.
1.3 components stack
Flink is a system with a layered architecture, with each layer containing components that provide specific abstractions to serve the components at the top.
Deployment layer
This layer mainly involves the deployment mode of Flink, which supports a variety of deployment modes: local, cluster (Standalone/YARN), cloud (GCE/EC2). The Standalone deployment mode is similar to that of Spark. Here, we take a look at the deployment mode of Flink on YARN
The Runtime layer
The Runtime layer provides all the core implementations that support Flink computing, such as distributed Stream processing, JobGraph to ExecutionGraph mapping, scheduling, and so on, and provides basic services for the upper API layer.
The API layer
The API layer mainly implements Stream processing oriented to unbounded Stream and batch-oriented Batch processing API, in which stream-oriented processing corresponds to DataStream API and batch-oriented processing corresponds to DataSet API.
Libraries layer
This layer can also be called Flink application framework layer. According to the division of API layer, the implementation computing framework built on the API layer to meet the specific application also corresponds to stream-oriented processing and batch-oriented processing respectively. Stream-oriented processing support: CEP (complex event processing), SQL-like operations (Table based relational operations); Batch-oriented support: FlinkML (Machine learning library), Gelly (Graph processing).
One of the most important features of Flink is that Batch and Streaming use the same processing engine, allowing Batch applications to run efficiently as a special type of Streaming application.
One problem is how Batch and Streaming are processed using the same processing engine.
1.4 How Batch and Streaming use the same processing engine.
Below is a code explanation of how Batch and Streaming use the same processing engine. First, the Flink test case is used to distinguish between the two.
Batch WordCount Examples
Streaming WordCount Examples
Batch and Streaming use different ExecutionEnviroment, For ExecutionEnviroment read the source data is a DataSet, and StreamExecutionEnviroment source data is a DataStream.
The Batch process from Optimzer to JobGgraph is used to construct the LocalPlanExecutor, and the Remote executor is RemotePlanExecutor
The first step in the Run method is to get the jobGraph, which is the client-side operation, and the client submits the jobGraph to the JobManager to be turned into ExecutionGraph. The difference between Batch and streaming is in getting JobGraph.
If we initialize FlinkPlan StreamingPlan, first of all, constructing Streaming StreamingJobGraphGenerator to will optPlan into JobGraph, Batch directly adopt another way of transformation.
In short, Batch and Streaming will have two different executionEnvironments that will translate different apis into different JobGgrah, In addition to StreamGraph, JobGraph also has OptimizedPlan. OptimizedPlan is converted from the Batch API. StreamGraph is a translation of the Stream API, and JobGraph’s responsibility is to unify the Batch and Stream diagrams.
1.5 Feature Analysis
High throughput & low latency
Flink’s stream processing engine requires minimal configuration to achieve high throughput and low latency. The following figure shows the performance of a distributed counting task, including the streaming data shuffle process.
Event Time and out-of-order events are supported
Flink supports windowing mechanisms for stream processing and Event Time semantics.
Event Time makes it easier to calculate events that arrive out of order or that may arrive late.
The exact-once semantics of state computation
Streams can maintain custom state during computation.
Flink’s Checkpointing mechanism guarantees the exactly once semantics of the state even in the event of a malfunction.
Highly flexible streaming window
Flink supports time Windows, statistics Windows, session Windows, and data-driven Windows
Windows can be customized with flexible trigger conditions to support complex streaming computing patterns.
Continuous flow model with back pressure
Data flow applications perform continuous (resident) Operators.
Flink streaming has natural flow control at run time: slow data sink nodes will backpressure fast data sources.
Fault tolerance
Flink’s fault tolerance mechanism is based on chandy-Lamport Distributed Snapshots.
This mechanism is very lightweight, allowing the system to have high throughput rates while still providing strong consistency.
Batch and Streaming a system uses the same engine for both stream and Batch processing
Flink shares a common engine for both stream and batch applications. Batch applications can run efficiently with a special stream processing application.
Memory management
Flink implements its own memory management within the JVM.
Applications can exceed the main memory size limit and incur less garbage collection overhead.
Iteration and incremental iteration
Flink has specialized support for iterative computation (such as in machine learning and graph computation).
Incremental iterations can take advantage of dependency computations for faster convergence.
Application tuning
Batching automatically optimizes some scenarios, such as avoiding expensive operations (such as Shuffles and sorts) and caching intermediate data.
API and libraries
Flow processing application
The DataStream API supports functional transformations over data flows, with custom state and flexible Windows.
The example on the right shows how to count word occurrences in a text data stream in a sliding window.
val texts:DataStream[String] = …
val counts = text .flatMap { line => line.split(“\\W+”) } .map { token => Word(token, 1) } .keyBy(“word”) .timeWindow(Time.seconds(5), Time.seconds(1)) .sum(“freq”)
Batch application
Flink’s DataSet API lets you write beautiful, type-safe, maintainable code in Java or Scala. It supports a wide range of data types, not just key/ Value pairs, as well as a wealth of Operators.
The example on the right shows one of the core loops of the PageRank algorithm in graph computation.
case class Page( pageId: Long, rank:Double) case class Adjacency( id: Long, neighbors:Array[Long])
val result = initialRanks.iterate(30) { pages => pages.join(adjacency).where(“pageId”).equalTo(“pageId”) { (page, adj, out : Collector[Page]) => {out.collect(Page(page.id, 0.15 / numPages)) for (n < -adj. neighbors) {out.collect(Page(n, 0.85 * page rank/adj. Neighbors. Length)}}}. GroupBy (” pageId “). The sum (” rank “)}
The class library ecological
The Flink stack provides a number of advanced apis and libraries for different scenarios: machine learning, graph analysis, and relational data processing. Currently, the library is still in beta and is undergoing a lot of development.
Extensive integration
Flink has integration with many projects in the open source big data processing ecosystem.
Flink can run on YARN, work with HDFS, read stream data from Kafka, execute Hadoop program code, and connect to multiple data storage systems.
The deployment of
Flink can be deployed independently of Hadoop, relying only on the Java environment, which is relatively simple.
Apache Streaming Framework Flink, Spark Streaming, Storm
Netease has
Enterprise big data visualization analysis platform. The self-service and agile analysis platform for business personnel adopts PPT mode to make reports, which is easier to learn and use. It has powerful exploration and analysis functions, and truly helps users to gain insight into data and discover value.
Click here – free trial.
Understand netease Cloud:
The official website of netease Cloud is www.163yun.com/
New user package: www.163yun.com/gift
Netease Cloud community: sq.163yun.com/