An overview of the
The year 2019 is the most extraordinary year in the field of real-time computing of big data. In January 2019, Alibaba Blink (internal Flink branch version) opened the open source, and the field of big data went from Spark to the era of two-power rivalry overnight. Flink has become a hot framework for big data processing because of its natural streaming computing characteristics and powerful processing performance.
Up to now, Flink has developed to version 1.9. In the field of big data development, the investigation of Flink in the interview has become a must for job seekers of big data development. This paper summarizes nearly 50 interview research points about Flink based on my experience as an interviewer.
Chat in this field is divided into the following parts:
The first part: The core concept and foundation of Flink, including the overall introduction of Flink, core concepts, operators and other research points.
The second part: The advanced part of Flink, including Flink data transmission, fault tolerance mechanism, serialization, data hot spot, back pressure and other problems encountered in the actual production environment.
The third part: Flink source code, including the implementation of Flink core code, Job submission process, data exchange, distributed snapshot mechanism, Flink SQL principle and other research points.
Part ONE: The core concepts and basic investigation in Flink
A brief introduction to Flink
Flink is a framework and distributed processing engine for stateful computation of unbounded and bounded data streams. Flink also provides core functions such as data distribution, fault tolerance mechanism and resource management.
Flink provides a number of apis with a high level of abstraction to allow users to write distributed tasks:
- DataSet API performs batch processing operations on static data and abstracts static data into distributed data sets. Users can easily use various operators provided by Flink to process distributed data sets. It supports Java, Scala and Python.
- DataStream SUPPORTS Java and Scala. DataStream abstracts streaming data into distributed data streams.
- Table API, query operations on structured data, abstraction of structured data into relational tables, and through SQL-like DSL query operations on relational tables, support Java and Scala.
In addition, Flink also provides domain libraries for specific application domains, e.g. Flink ML, Flink’s machine learning library, provides machine learning Pipelines API and implements various machine learning algorithms. Gelly, Flink’s graph computing library, provides the related API of graph computing and various graph computing algorithm implementation.
According to the website, Flink’s features include:
Support high throughput, low latency, high performance stream processing support Window operation with event time support Exactly-once semantics support stateful computing support highly flexible Window operation, Support window operations based on time, count, session, and data-driven support Continuous flow model with Backpressure support fault tolerance based on lightweight distributed Snapshot implementation a runtime simultaneously supports Batch on Streaming processing and Streaming processing Flink implements its own memory management within the JVM support for iterative computation support for program automatic optimization: avoid certain cases of Shuffle, sorting and other expensive operations, intermediate results are necessary to cacheCopy the codeIi. What is the difference between Flink and Spark Streaming?
This is a very macro problem because there are so many differences between the two frameworks. But there is one important point to make when interviewing: Flink is a standard real-time processing engine, event-driven. Spark Streaming is a micro-batch model.
Here are some key differences between the two frameworks:
1. Architectural model
Spark Streaming includes Master, Worker, Driver, and Executor while Flink includes Jobmanager, Taskmanager, and Slot.
2. Task scheduling
Spark Streaming continuously generates tiny data batches to build directed acyclic graph DAG. Spark Streaming creates DStreamGraph, JobGenerator and JobScheduler in sequence.
Flink generates StreamGraph from user-submitted code, optimizes it to generate JobGraph, and then submits it to JobManager for processing. JobManager generates ExecutionGraph from JobGraph. ExecutionGraph is the core data structure of Flink scheduling. JobManager schedules jobs based on ExecutionGraph.
3. Time mechanism
Spark Streaming supports a limited time mechanism, only processing time. Flink supports three definitions of time for stream handlers: processing time, event time, and injection time. It also supports the watermark mechanism to process lagging data.
4. Fault tolerance mechanism
For Spark Streaming tasks, we can set a checkpoint, and then if there is a failure and restart, we can recover from the last checkpoint, but this behavior only keeps the data from being lost and may be processed repeatedly, rather than processing the semantics exactly at once.
Flink uses the two-phase commit protocol to solve this problem.
What are the components of Flink stack?
According to the Flink website, Flink is a system with a layered architecture. Each layer contains components that provide specific abstractions to serve the components at the top.
From bottom to bottom, each layer represents the following: Deploy layer: This layer mainly involves the deployment mode of Flink. As shown in the figure above, Flink supports a variety of deployment modes including Local, Standalone, Cluster and Cloud. Runtime layer: The Runtime layer provides the core implementation that supports Flink computing, such as distributed Stream processing, JobGraph-to-ExecutionGraph mapping, scheduling, and so on, and provides basic services for the upper API layer. The API layer: The API layer mainly implements stream-oriented processing and Batch processing apis, in which stream-oriented processing corresponds to DataStream API, and batch-oriented processing corresponds to DataSet API. Later versions, Flink has plans to unify DataStream and DataSet apis. Libraries layer: This layer is called the Flink application framework layer. According to the division of the API layer, the implementation computing framework built on the API layer to meet the specific application also corresponds to stream-oriented processing and batch-oriented processing respectively. Stream-oriented processing support: CEP (complex event processing), SQL-like operations (Table based relational operations); Batch-oriented support: FlinkML (Machine learning library), Gelly (Graph processing).
Does Flink have to rely on Hadoop components to run?
Flink can be completely independent of Hadoop and run independently of Hadoop components. However, as a big data infrastructure, Hadoop system is beyond any big data framework. Flink can integrate many Hadooop components, such as Yarn, Hbase, HDFS, and so on. For example, Flink can be integrated with Yarn to schedule resources, read and write HDFS, or use HDFS to perform checkpoints.
5. How big is your Flink cluster?
Note that this question seems to ask you about the size of the Flink cluster in your application, but it also hides another question: How many cluster sizes can Flink support?
To answer this question, you can describe the cluster scale, nodes, memory, and deployment mode (Flink on Yarn in general) in your production process. In addition, you can run Flink tasks on a small cluster (less than five nodes) or thousands of TB nodes at the same time.
Do you know the basic programming model of Flink?
What are the roles of Flink cluster? What is the role of each?
Flink program has three roles: TaskManager, JobManager and Client. JobManager plays the role of the manager Master in the cluster. It is the coordinator of the whole cluster, responsible for receiving Flink jobs, coordinating checkpoints, Failover recovery, etc., and managing TaskManager of slave nodes in the Flink cluster.
A TaskManager is a group of tasks that perform Flink jobs on the Worker that is actually responsible for computing. Each TaskManager manages the resource information on its node, such as memory, disk, and network, and reports the resource status to The JobManager when it is started.
Client is the Client for Flink program submission. When a user submits a Flink program, a Client will be created first. The Client will preprocess the Flink program submitted by the user and submit it to the Flink cluster for processing. Therefore, the Client needs to obtain the JobManager address from the Flink program configuration submitted by the user, establish a connection to the JobManager, and submit the Flink Job to the JobManager.
Task Slot in Flink resource Management
As mentioned in the Flink architecture role, TaskManager is the Worker that is actually responsible for performing the computation. TaskManager is a JVM process that executes one task or more subtasks in separate threads. To control how many tasks a TaskManager can accept, Flink introduced the concept of Task Slot.
In simple terms, TaskManager divides the resources managed on its nodes into different slots: a fixed-size subset of resources. This prevents tasks of different jobs from competing with each other for memory resources, but the main requirement is that Slot only performs memory isolation. CPU isolation is not done.
Nine, say Flink common operators?
The most common operators of Flink include: Map: DataStream → DataStream. A parameter is input to generate a parameter. The function of Map is to convert the input parameter. Filter: Filters out the specified data. KeyBy: Groups groups according to the specified key. Reduce: merges results. Window: a Window function that groups the data for each key according to certain properties (for example, the data that arrived within 5s)
What do you know about Flink’s zoning strategy?
Understand what partitioning policies are. Partitioning policies are used to determine how data is sent downstream. Currently Flink supports the implementation of partitioning policies in 8.
Here is the partitioning strategy inheritance diagram for the entire Flink implementation:
The GlobalPartitioner data is distributed to the first instance of the downstream operator for processing.
ShufflePartitioner data is randomly distributed to each instance of the downstream operator for processing.
The RebalancePartitioner data is looped through to each instance downstream for processing.
The RescalePartitioner circulates to each instance of the downstream operator based on the parallelism of the upstream and downstream operators. It’s A little hard to understand here, but let’s say upstream parallelism is 2, and the numbers are A and B. The downstream parallelism is 4, and the number is 1,2,3,4. So A circulates data to 1 and 2, and B circulates data to 3 and 4. Assume that the upstream parallelism is 4 and the numbers are A, B, C, D. The parallelism of the downstream is 2, and the number is 1,2. So A and B send data to 1, and C and D send data to 2.
The BroadcastPartitioner outputs upstream data to each instance of the downstream operator. Suitable for large data sets and small data sets to do Jion scenarios.
ForwardPartitionerForwardPartitioner instance is used to record the output to the downstream local operator. It requires the same degree of parallelism of upstream and downstream operators. In short, the ForwardPartitioner is used to do console printing of data.
KeyGroupStreamPartitionerHash partition. The Hash value of the Key is output to the downstream operator instance.
CustomPartitionerWrapper User-defined divider. Users need to implement the Partitioner interface themselves to define their own partitioning logic. Such as:
static class CustomPartitioner implements Partitioner<String> { @Override public int partition(String key, int numPartitions) { switch (key){ case "1": return 1; case "2": return 2; case "3": return 3; default: return 4; }}}Copy the code11. Is the parallelism of Flink understood? What is Flink’s parallelism setting?
Tasks in Flink are divided into parallel tasks for execution, where each parallel instance processes a portion of the data. The number of these parallel instances is called parallelism.
We can set parallelism in a real production environment at four different levels:
- Operator Level
- Execution Environment Level
- Client Level
- System Level
Note the following priorities: Operator layer > Environment layer > Client layer > system layer.
What is the difference between Flink Slot and Parallelism?
The official website is very classic picture:
Slot refers to the concurrent execution capability of the TaskManager, Suppose we will taskmanager. NumberOfTaskSlots configured for 3 so every taskmanager allocated in 3 TaskSlot, 3 a total of nine TaskSlot taskmanager.
Parallelism refers to the concurrency capabilities that TaskManager actually uses. If we set parallelism. Default to 1, then only one of the nine taskslots can be used and eight of them are free.
Does Flink have a restart strategy? What are some of them?
Flink implements a variety of restart strategies.
- Fixed Delay Restart Strategy
- Failure Rate Restart Strategy
- No Restart Strategy
- Fallback Restart Strategy
Have you used distributed cache in Flink? How to use it?
Flink implements distributed caching in a similar way to Hadoop. The purpose is to read the file locally and place it in the TaskManager node to prevent the task from pulling it repeatedly.
val env = ExecutionEnvironment.getExecutionEnvironment
// register a file from HDFS
env.registerCachedFile("hdfs:///path/to/your/file", "hdfsFile")
// register a local executable file (script, executable, ...)
env.registerCachedFile("file:///path/to/exec/file", "localExecFile", true)
// define your program and execute
...
val input: DataSet[String] = ...
val result: DataSet[Integer] = input.map(new MyMapper())
...
env.execute()
Copy the code15. What should we pay attention to when using the broadcast variable in Flink?
We know that Flink is parallel, and the calculation may not be in the same Slot, so there is a situation where we need to access the same data. So the broadcast variable in Flink is designed to solve this situation.
We can understand the broadcast variable as a common shared variable. We can broadcast a dataset, and then different tasks can obtain the data on the node. There is only one copy of the data on each node.
Tell me about Windows in Flink.
Here’s a classic picture from the official website:
Flink supports two ways of dividing Windows, by time and count. If you divide a window by time, it’s a time-window and if you divide a window by data, it’s a count-window.
Flink supports two important properties of Windows (size and interval)
If the size = interval, then can form tumbling – window (no overlapping data) if the size > interval, then can form sliding – window (overlap data) if the size interval, then the window will be lost data. For example, every five seconds, if you count the number of cars that have passed an intersection in the past three seconds, you will miss two seconds of data.
Four basic Windows can be obtained by combination:
- * timeWindow(time.seconds (5)) * timeWindow(time.seconds (5))
- Time-sliding -window Specifies the time window with overlapping data. Example: timeWindow(time.seconds (5), time.seconds (3))
- The count – tumbling – the number of overlapping Windows without data window, set mode, for example: countWindow (5)
- Count-sliding -window specifies the number of overlapping data Windows. Example: countWindow(5,3)
State storage in Flink?
Flink often needs to store intermediate states during calculation to avoid data loss and state recovery. Different state storage policies can affect how state persistence interacts with checkpoint.
Flink provides three state storage modes: MemoryStateBackend, FsStateBackend, and RocksDBStateBackend.
What are the types of time in Flink
Like other streaming computing systems, Flink time is divided into three categories: event time, ingestion time, and processing time.
If the time window is defined against EventTime, an EventTimeWindow is formed, requiring that the message itself should carry EventTime. If the time window is defined based on IngesingtTime, it will be IngestingTimeWindow, based on source systemTime. If the time window is defined against the ProcessingTime baseline, the ProcessingTimeWindow will be formed using operator’s systemTime.
What is the concept of watermark in Flink and what role does it play?
Watermark is a mechanism proposed by Apache Flink to handle EventTime window calculations, essentially a timestamp. Typically, Watermark is used with Windows to handle out-of-order events.
Are you familiar with Flink Table & SQL? What does the TableEnvironment class do
TableEnvironment is the core concept of Table API and SQL integration.
This class is used to:
- Registry in the internal catalog
- Register an external Catalog
- Execute SQL query
- Register user-defined (scalar, table, or aggregate) functions
- Convert DataStream or DataSet to a table
- Hold on ExecutionEnvironment or StreamExecutionEnvironment references
What is Flink SQL implementation principle? How is SQL parsing implemented?
First, you should know that Flink’s SQL parsing is based on Apache Calcite, an open source framework.
Based on this, a complete SQL parsing process is as follows:
- Users develop business applications using the syntax that provides Stream SQL externally
- StreamSQL is syntactic checked by Calcite. After passing the syntactic check, it is converted to calCite logical tree node. The final logical plan for Calcite
- Flink’s self-defined optimization rules, Calcite volcano model and heuristic model were used to optimize the logical tree and generate the optimal Flink physical plan
- For the physical plan, janino CodeGen was used to generate the code, and the stream application described by the low-level API DataStream was generated and submitted to the Flink platform for execution
Part TWO: The Flink interview advanced part
How does Flink support batch integration?
Flink’s developers consider batch processing to be a special case of streaming. Batch processing is limited stream processing. Flink uses one engine to support the DataSet API and DataStream API.
Ii. How does Flink achieve efficient data exchange?
In a Flink Job, data needs to be exchanged between different tasks, and the entire data exchange is handled by the TaskManager, whose network component first collects records from the buffer and then sends them. Records are not sent one by one. Second, a batch is accumulated and then sent. The batch technology can make more efficient use of network resources.
Three, how does Flink do fault tolerance?
Flink fault tolerance mainly relies on the strong CheckPoint mechanism and State mechanism. Checkpoint periodically creates distributed snapshots and backs up application status. State is used to store intermediate states during computation.
Four, what is the principle of Flink distributed snapshot?
Flink’s distributed snapshot is tailored to the Chandy-Lamport algorithm. In simple terms, this means continuously creating a consistent snapshot of the distributed data stream and its state.
The core idea is to insert barriers into the input source side and control the synchronization of barriers to achieve snapshot backup and exactly-once semantics.
5. How does Flink ensure Exactly-once semantics?
Flink implements end-to-end consistency semantics by implementing two-phase commit and state saving. It is divided into the following steps:
- BeginTransaction creates a temporary folder to write data to
- PreCommit writes cached data in memory to a file and closes it
- Commit The temporary files that have been written before into the target directory. This represents some delay in the final data
- Abort Discards temporary files
If the failure occurs after the pre-submission is successful and before the formal submission. Pre-committed data can be submitted or deleted based on status.
What’s so special about Flink’s Kafka connector?
Flink has a single connector module that all other connectors rely on. The new Kafka connector released in version 1.9 eliminates the need to rely on different versions of kafka clusters. You only need to rely on one connector.
Flink memory management is how to do?
Instead of storing a large number of objects on the heap, Flink serializes them all into a pre-allocated block of memory. In addition, Flink uses a lot of out-of-heap memory. If the data to be processed exceeds the memory limit, some data is stored on hard disks. Flink implemented its own serialization framework for directly manipulating binary data.
Theoretically, Flink’s memory management is divided into three parts:
- Network Buffers: this is allocation in TaskManager start-up time, this is a set of Network used to cache data memory, each block is 32 k, 2048 default distribution, can be modified by “taskmanager.net work. NumberOfBuffers”
- Memory Manage pool: A large number of Memory Segment blocks for run-time algorithms (Sort/Join/Shuffle, etc.) that are allocated at startup. The following code calculates the allocation of memory based on various parameters in the configuration file. (heap or off-heap, discussed in the next section), memory allocation supports pre-allocation and lazy load, the default lazy load mode.
- User Code, which is the data structure for the User Code and TaskManager itself in addition to Memory Manager.
How to serialize Flink?
Java has its own serialization and deserialization functions, but auxiliary information occupies a large space, and too much class information is recorded when serializing objects.
Apache Flink abandons Java’s native serialization approach and handles data types and serialization in a unique way, including its own type descriptors, generic type extraction, and type serialization frameworks.
TypeInformation is the base class for all type descriptors. It reveals some of the basic properties of the type and can generate serializers. TypeInformation supports the following types:
- BasicTypeInfo: Any Java basic type or String type
- BasicArrayTypeInfo: Any Java primitive type array or String array
- WritableTypeInfo: Implementation class for any Hadoop Writable interface
- TupleTypeInfo: Any Flink Tuple type (Tuple1 to Tuple25 supported). Flink tuples are Java tuples of fixed length and type
- CaseClassTypeInfo: Any Scala CaseClass(including Scala tuples)
- PojoTypeInfo: Any POJO (Java or Scala), for example, all member variables of a Java object that are either defined by public modifiers or have getter/setter methods
- GenericTypeInfo: Any class that cannot match any of the previous types
Flink automatically generates typeserializers for each of the first six types of datasets, making it very efficient for serializing and deserializing datasets.
9. Data skew appears in The Window of Flink. Do you have any solution?
Window-generated data skew refers to the fact that the amount of data stacked in different Windows differs too much. Essentially, this happens because the data source sends different amounts of data at different speeds. This situation is generally solved in two ways:
- Preaggregate data before it enters the window
- Redesign the window aggregation key
X. How to solve data hot spots in Flink when using aggregation functions such as GroupBy, Distinct and KeyBy?
Data skew and data hotspot are problems that all big data frameworks cannot avoid. There are three aspects to deal with such problems:
- Avoid such problems in your business
For example, in a hypothetical order scenario, the order volume of Beijing and Shanghai increases by dozens of times, while the data volume of other cities remains unchanged. At this time, when we are aggregating, there will be data accumulation in Beijing and Shanghai. We can collect data of Beijing and Shanghai separately.
- Key design
By splitting hot key, such as Beijing and Shanghai in the previous example, Beijing and Shanghai can be split and aggregated by region.
- Parameter Settings
An important improvement in the performance optimization of Flink 1.9.0 SQL(Blink Planner) is the upgrade of the microbatch model, known as MiniBatch. The principle is to cache certain data and then trigger processing to reduce the access to State, thus improving throughput and reducing the amount of data output.
11. Flink has a high task delay. If you want to solve this problem, how do you start?
In the background task management of Flink, we can see which operator and task of Flink have backpressure. The most important means are resource tuning and operator tuning. Resource tuning refers to tuning the number of concurrent operators (parallelism), CPU (core), heap memory (HEAP_memory), and other parameters in a job. Job parameter tuning includes: parallelism setting, State setting, checkpoint setting.
How does Flink deal with back pressure?
Flink is based on the producer-consumer model for message delivery, and Flink’s backpressure design is also based on this model. Flink uses efficient bounded distributed blocking queues, like Java’s generic BlockingQueue. As downstream consumers slow down, upstream gets clogged.
What are the differences between Flink’s backpressure and Strom’s?
Storm monitors the load of the receiving queue in Bolt. If the load exceeds the high water level, it will write the backpressure information to Zookeeper, and the Watch on Zookeeper will notify all workers in the topology to enter the backpressure state. Finally, Spout stops sending tuples.
Backpressure in Flink uses an efficient bounded distributed blocking queue where slow downstream consumption causes the sender to block.
The biggest difference between the two is that Flink is backpressure stage by stage, while Storm slows down speed directly from source.
Do you know the concept Operator Chains?
For more efficient distributed execution, Flink links operator subtasks together to form tasks as much as possible. Each task is executed in one thread. Linking operators to a task is a very effective optimization: it reduces switching between threads, reduces serialization/deserialization of messages, reduces exchanging of data between buffers, and improves overall throughput while reducing latency. This is what we call a chain of operators.
15. When will Flink combine Operator chain to form Operator chain?
Conditions where two operator chains are together:
- The parallelism of upstream and downstream is consistent
- The downstream node has an input of 1 (that is, the downstream node has no input from other nodes)
- Both upstream and downstream nodes are in the same slot group.
- The chain policy of downstream nodes is ALWAYS (links with upstream and downstream nodes can be set to ALWAYS by default, such as map, flatMap, and filter).
- The chain policy of the upstream node is ALWAYS or HEAD (the upstream node can only be linked to the downstream node, but not to the upstream node. The default Source is HEAD).
- Data is partitioned forward between two nodes (see Partitioning for Understanding Data Flow)
- The user does not disable chain. Procedure
What are the new features of Flink1.9?
- Hive read and write supports UDF
- Flink SQL TopN and GroupBy optimization
- Checkpoint and SavePoint are optimized for real-world business scenarios
- Flink state query
Kafka data consumption, how to deal with dirty data?
A fliter operator can be added before processing to filter out the data that does not conform to the rules.
Part THREE: Flink interview source code
Flink Job Submission Process Flink Job submitted by users is converted into a DAG task, as follows: StreamGraph, JobGraph, ExecutionGraph, Flink JobManager and TaskManager, JobManager and Client interaction is based on Akka toolkit, is message driven. The whole Flink Job submission also includes the creation of ActorSystem, the startup of JobManager, and the startup and registration of TaskManager.
Ii. Which “graphs” are Flink’s so-called “three-layer graph” structure?
The DAG generation of a Flink task generally goes through the following three processes:
- StreamGraph closest to code expressed by some logical calculation topology structure, according to the user code execution order to add StreamTransformation StreamExecutionEnvironment constitute a flow chart.
- JobGraph, generated from StreamGraph, merges nodes that can be cascaded together, sets edges between nodes, arranges resource sharing slot slots and places associated nodes, uploads files needed for tasks, sets checkpoint configurations, and more. Equivalent to a task graph that has been partially initialized and optimized.
- ExecutionGraph, a translation of JobGraph, contains the content needed to execute a task and is the closest ExecutionGraph to the underlying implementation.
What role does JobManger play in the cluster?
JobManager is responsible for task scheduling and resource management of the entire Flink cluster, obtains the submitted application from the client, and then according to the TaskSlot usage of the TaskManager in the cluster, Assign TaskSlot resources to the submitted application and command TaskManager to launch the application obtained from the client.
The JobManager is the Master node of the entire cluster, and only one active JobManager is responsible for task management and resource management of the entire cluster.
JobManager and TaskManager communicate with each other through Actor System to obtain the task execution status and send the task execution status to the client.
During task execution, Flink JobManager triggers the Checkpoint operation. After each TaskManager receives the Checkpoint command, the TaskManager completes the Checkpoint operation. All Checkpoint coordination is done in Fink JobManager.
When the task is complete, Flink reports the task execution information back to the client and frees up resources in The TaskManager for the next submission of the task.
4. What role does JobManger play in the cluster startup process?
JobManager is responsible for receiving Flink jobs, scheduling tasks, collecting job status and managing TaskManager. It contains an Actor and does the following:
- RegisterTaskManager: This is sent by the TaskManager that wants to register with JobManager. 6. Successful registration is Ack via the initials Geregistration message.
- SubmitJob: sent by the Client that submits the job to the system. The submitted information is job description information in the form of JobGraph.
- CancelJob: cancels the job with the specified ID. CancellationSuccess is returned, and CancellationFailure is returned otherwise.
- UpdateTaskExecutionState: the TaskManager sent to update the execution node (ExecutionVertex) state. Return true on success, false otherwise.
- RequestNextInputSplit: the Task on the TaskManager requests the NextInputSplit, and returns NextInputSplit on success, or null otherwise.
- JobStatusChanged: It means that the state of the job (RUNNING, CANCELING, FINISHED, etc.) has changed. This message is sent by ExecutionGraph.
What role does TaskManager play in the cluster?
TaskManager functions as a Slave node in the entire cluster and is responsible for executing specific tasks and applying for and managing resources on each node for corresponding tasks.
The client compiles and packages the written Flink application and submits it to JobManager. Then JobManager assigns tasks to TaskManager nodes with resources according to the resources of TaskManager registered in JobManager. Then start and run the task.
The TaskManager receives the tasks to be deployed from JobManager, starts the Task using Slot resources, establishes a network connection for data access, receives the data, and begins data processing. The taskManagers interact with each other through data flows.
It can be seen that Flink tasks are run in a multi-threaded way, which is very different from MapReduce multi-JVM. Flink can greatly improve the efficiency of CPU usage, and share system resources between multiple tasks and tasks by way of TaskSlot. Each TaskManager manages multiple TaskSlot resource pools to effectively manage resources.
What role does TaskManager play in the cluster startup process?
TaskManager the start process is relatively simple: start classes: org. Apache. Flink. Runtime. The TaskManager. Start TaskManager core method: SelectNetworkInterfaceAndRunTaskManager starts directly to the JobManager registered yourself, after completion of registration, for part of the module’s initialization.
How is Flink computing resource scheduling implemented?
The most fine-grained resource in a TaskManager is Task Slot, which represents a fixed-size subset of resources, and each TaskManager allocates the resources it occupies to its slot.
By adjusting the number of Task slots, users can define how tasks are isolated from each other. Each TaskManager has a slot, which means that each task runs in a separate JVM. With multiple slots per TaskManager, that means multiple tasks running in the same JVM.
Tasks in the same JVM process can share TCP connections (based on multiplexing) and heartbeat messages, reducing network transmission and sharing data structures, reducing the cost of each task to some extent. Each slot can accept a single task or a pipeline consisting of multiple consecutive tasks, as shown in the following figure. The FlatMap function occupies a taskslot, and the key Agg function and sink function share a taskslot:
8. Brief introduction of Flink’s data abstraction and data exchange process?
Flink implements self-managed memory to avoid inherent drawbacks of the JVM, such as low Java object storage density and FGC impact on throughput and response. MemorySegment is a memory abstraction of Flink. By default, a MemorySegment can be thought of as an abstraction of a 32KB chunk of memory. This memory can be either a byte[] in the JVM or a DirectByteBuffer.
On top of the MemorySegment abstraction, Flink uses the abstract, or memory object, Buffer to transfer data from the operator data object to the TaskManager in preparation for delivery to the next node.
Interconnecting intermediate objects converted from Java objects to buffers is another abstract StreamRecord.
How is the distributed snapshot mechanism implemented in Flink?
A core part of Flink’s fault tolerance mechanism is to make consistent snapshots of distributed data flows and operator states. These snapshots act as consistency checkpoints that the system can roll back in the event of a failure. The mechanism Flink uses to make these snapshots is described in “Lightweight Asynchronous Snapshots for Distributed Data Flows.” It was inspired by the standard Chandy-Lamport algorithm for distributed snapshots and customized to Flink’s execution model.
Barriers are injected into the parallel data stream at the data stream source. The location at which snapshot N is inserted (we call it Sn) is the maximum location in the data source of the data contained in the snapshot. For example, in Apache Kafka, this location will be the offset of the last record in the partition. Report the location Sn to the Checkpoint coordinator (Flink’s JobManager).
Then barriers flow downstream. When an intermediate operator receives snapshot N’s barriers from all of its input streams, it issues barriers for snapshot N into all of its output streams. Once the sink operator (the end of the streaming DAG) receives Barriers n from all its input streams, it confirms to the Checkpoint coordinator that snapshot N is complete. After sink confirms all snapshots, the snapshots are completed.
Once snapshot N is completed, the job will never ask the data source for records before Sn, because these records (and subsequent records) will have been processed through the entire data flow topology.
How is FlinkSQL implemented?
Flink hands over SQL validation, SQL parsing, and SQL optimization to Apache Calcite. Calcite is used in many other open source projects such as Apache Hive, Apache Drill, Apache Kylin, Cascading. Calcite is at the heart of the new architecture, as shown below.
The task of building the abstract syntax tree was left to Calcite. The SQL Query is turned into a tree of SQL nodes by the Calcite parser, which is then validated to build Calcite’s abstract syntax tree (the Logical Plan in the diagram). On the other hand, calls to the Table API are built into the abstract syntax tree of the Table API, which is converted into Calcite’s abstract syntax tree through the RelBuilder provided by Calcite. This is then translated into logical execution plans and physical execution plans in turn.
After submitting the task, it is distributed to each TaskManager for execution, and at runtime, the Janino compiler is used to compile the code and run it.
Welcome to follow the author other Chat:
Big data development interview Guide Flink’s sharpest weapon: Flink SQL primer and actual combat
Big data Development interview Guide
Number of real-time warehouse | you need is a powerful OLAP engines
Statement: all articles in this number are original, except for special notes, public readers have the right to read first, shall not be reproduced without the permission of the author, or tort liability.
Pay attention to my public number, background reply [JAVAPDF] get 200 pages of questions! 50000 people pay attention to the big data into the way of god, don’t you want to know? Fifty thousand people pay attention to the big data into the road of god, really not to understand it? Fifty thousand people pay attention to the big data into the way of god, sure really not to understand it?
