Moment For Technology

How to solve TCP Sticky Packet problem with NetTY based on RPC Framework communication

Posted on May 28, 2023, 1:36 a.m. by Sienna Hart
Category: The back-end Tag: The back-end TCP/IP

This is the 19th day of my participation in the First Challenge 2022

Cause 0.

Because they build a RPC frame wheel, the need to solve the TCP sticky packet problem, hereby record, hope to facilitate others. This is the GitHub address I wrote for the RPC framework Welcome star, Fork. Several articles have been written explaining how this framework works. Those who are interested in the principle are welcome to talk.

1. What is sticky bag

1.1 What is TCP Sticky Packet

TCP glue bag is in the process of TCP data transmission, for some reason, the receiver receive the read data is not a data, but to save multiple packet stream of bytes assembly together instead of leading to multiple data together, the receiver when read don't know how to divide the data into the desired multiple sets of data, this is the package.

1.2 Formation Reasons

TCP sticky packets are caused by the buffer and TCP data flow at the sender and receiver.

For example, Nagle algorithm will assemble many small packets of data into one large data to improve bandwidth utilization. (Specific Nagle algorithm will not expand).

But sticky packets still exist even when Nagle algorithm is turned off. This is not the root cause of TCP sticky packets. Because of the existence of the buffer, the buffer is not sent until the cache is full, and the receiver also uses the cache to receive data, and then reads the received data parsing from the cache. At this time, someone asked, if the amount of data is very small, always does not hit the full buffer what to do, this depends on the sending and receiving end of the timer, they will be timed to process data, or it is not a bug.

It is because of the existence of buffer and the form of TCP data flow, resulting in the splicing of multiple groups of data, the formation of sticky packet, half packet problem.

1.3 How to Solve the problem

At present, the commonly used method is to define the start edge character + data length to tell the receiver the specific length of a packet.

However, there are fixed length definitions, but this may result in the waste of blank bytes and beyond the length of the way is not easy to expand. The method of pure boundary character prevents the collision between the actual message body and the boundary character, resulting in false truncation of the message.

2.NET TY How to solve this problem

Netty encapsulates NIO mode TCP communication perfectly. Can let a person quickly write available TCP communication server and client, and very simple to solve the sticky packet problem.

Netty provides delimiters and length based codecs for developers to use. DelimiterBaseFrameDecoder can be user-defined data, to split the delimiter LineBaseFrameDecoder is that by the end of each line (or \ n \ r \ n) segmentation, speed than the former. And based on the length of the FixedLengthFrameDecoder fixed decoder, LengthFieldBasedFrameDecoder dynamic length decoder. Each of the four methods has a corresponding codec.

At the same time for the data type boundary, Netty also support byte, string, protobuf, etc., we can go to see MessageToMessageDecoder subclass, you can find netty provides a lot of decoding rules.

3. Practice - Client and server implementation of RPC framework

However, in the first version, I used synchronous IO to write the client, but found it was so bad during the pressure test that I decided to use NIO to rewrite it. At first, I felt it was inconvenient to write the client using Netty (I didn't know how to write it at that time), so I decided to use Java native NIO to write the client. In the end, I found that it was very difficult to deal with sticky packets. I needed to define the special boundary symbol and set the length, which was very complicated to parse on the client and server side. So I tried to use Netty to write, and found it very simple., workerGroup).channel(NioServerSocketChannel.class)
						.childHandler(new ChannelInitializerSocketChannel() {

							protected void initChannel(SocketChannel ch) throws Exception {
								ChannelPipeline pipeline = ch.pipeline();
								/** * uses variable length to solve sticky packet * the first 4 bits of data are stored, so the maximum length supported by a call is 65535 bytes, while filtering out the first 4 bits */
								pipeline.addLast("frameDecoder".new LengthFieldBasedFrameDecoder(Integer.MAX_VALUE,;
								/** * Calculates the length of the data and reverses the */ in the transmitted data
								pipeline.addLast("frameEncoder".new LengthFieldPrepender(4));
								pipeline.addLast("decoder".new ByteArrayDecoder());
								pipeline.addLast("encoder".new ByteArrayEncoder());
								pipeline.addLast(new ServerHandler());
						}).option(ChannelOption.SO_BACKLOG, 128)
						.childOption(ChannelOption.SO_KEEPALIVE, true)
						.childOption(ChannelOption.RCVBUF_ALLOCATOR, new AdaptiveRecvByteBufAllocator(64,Global.getInstance().getMaxBuf(),Global.getInstance().getMaxBuf()));

Copy the code

This is the server side of the code, is not particularly simple, because TCP will transfer data serialized from compressed data into a ByteArray, so the use of the ByteArray codec, using a dynamic length baseframe to solve the sticky packet problem. In this way, the sticky bag problem was solved.

If you want to learn more about the code, you can download it from Github, which contains all the KRPC code. Welcome to exchange and study.

About (Moment For Technology) is a global community with thousands techies from across the global hang out!Passionate technologists, be it gadget freaks, tech enthusiasts, coders, technopreneurs, or CIOs, you would find them all here.