The Spring Festival is coming soon, I thought I could finish the task at hand to prepare for the Spring Festival. Unexpectedly Netty server was attacked again, when received the server alarm (CPU surge alarm) message, we know the other party attacked again.

In the past, it had been left to the brothers below to solve the problem, but this time, in order to have a good year, I decided to take matters into my own hands.

Story prelude

Netty service is a relatively marginal service of the company, only one device is in use, and the code was written by the technical Leader (who has left) before, and the deadline is always tight, so I didn’t take time to solve the problem thoroughly.

At the beginning of the attack did not check the code, saw the crazy requests, CPU running full, log full, still thought it was a DDoS attack.

Several temporary measures have been taken:

  • Detach servers to ensure that an attack on one service does not bring down other services;
  • Changed IP and port;
  • Add a blacklist for attacking IP addresses.
  • At the code level, an illegal request is found forcing the connection down;
  • Add logs to trace attack packets and their source.
  • Report the IP of the attack service (Shanghai Ali Cloud);

But before long, the hacker came again, ten days and half a month to attack, as if to know the service IP and background code, haunting.

This is not, was caught today, and before adding log printing, also got the message content of the attack, repeated the attack operation.

/ / the attacker first try 8000002872 fe1d130000000000000002000186a00001977c0000000000000000000000000000000000000000 / message/message of the attacker a second attempt 8000002872FE1D130000000000000002000186A00001977C00000000000000000000000000000000Copy the code

In the preceding packets, the first packet triggers the attack, but the second packet does not affect the attack (the format is the same as that of normal service packets).

The following takes you to analyze the vulnerability in the attack logic and code.

Knowledge reserves

To understand how attacks work, you need to have some knowledge of Netty technology. Netty how to implement the client and server side of the code is not expanded here, you can take a look at the implementation example: github.com/secbr/netty…

Let’s focus on custom decoders and io.net ty.buffer.bytebuf. The customized decoder is used to parse the packet, and the packet content is cached through ByteBuf.

The format of the attack message above indicates that the hacker had “guessed” that we were transferring content in hexadecimal Btye format (and even knew it).

Custom decoder

To customize the decoder, inherit the MessageToMessageDecoder class and implement the decode method. Here is the sample code:

public class MyDecoder extends MessageToMessageDecoder<ByteBuf> {

    @Override
    protected void decode(ChannelHandlerContext ctx, ByteBuf in, List<Object> out) {
    }
}
Copy the code

The logic for parsing packets is to process them in decode method. ByteBuf in is the container for receiving incoming packets, and List<Object> out is used to output parsed results.

Here’s a look at the buggy code (desensitized) :

protected void decode(ChannelHandlerContext ctx, ByteBuf in, List<Object> out) { int readableBytes = in.readableBytes(); while (readableBytes > 3) { in.skipBytes(2); int pkgLength = in.readUnsignedShort(); in.readerIndex(in.readerIndex() - 4); if (in.readableBytes() < pkgLength) { return; } out.add(in.readBytes(pkgLength)); readableBytes = in.readableBytes(); }}Copy the code

The code above is fine when running normal business, but when attacked, it goes into an endless loop. As a result, even though the operation to close the connection is added during the business process, it is invalid.

Before we look at the code above, we need to take a closer look at how ByteBuf works.

The principle of ByteBuf

ByteBuf maintains two indexes: a readIndex for reading and a writeIndex for writing.

When reading from ByteBuf, readIndex is incremented by the number of bytes that have been read, and when writing to ByteBuf, writeIndex is incremented.

The preceding figure shows the attack packet as an example. The attacker uses a 44-byte packet to attack. Since the hexadecimal system is used, two characters take up 1 byte.

Both readIndex and writeIndex start at index position 0. When ByteBuf’s readXXX or writeXXX methods are executed, the corresponding index is advanced. Not when performing operations on setXXX or getXXX methods.

After understanding the basic processing principle of ByteBuf, we will compare the attacker’s message and source code to restore the attack process.

Attack the reduction

The following is a step by step analysis directly through the source code, mainly involving the methods of the ByteBuf class. The effective attack packet is the first packet mentioned above.

/ / the attacker message 8000002872 fe1d130000000000000002000186a00001977c0000000000000000000000000000000000000000 first attemptCopy the code

Here’s the code:

int readableBytes = in.readableBytes();
Copy the code

This line of code uses the readableBytes method to get the number of bytes that can be read in the current ByteBuf. The above attack message is 88 characters long, so this gives 44 bytes.

Specific parsing takes place when readableBytes are greater than 3:

in.skipBytes(2);
Copy the code

Obviously, two bytes have been skipped through the skipBytes method.

int pkgLength = in.readUnsignedShort();
Copy the code

Using the readUnsignedShort method, we get two bytes of content that correspond to a hexadecimal value of 0028 and a decimal value of 40. The meaning of these two bytes in a message is the length of (part or whole) the message.

There are two algorithms for the length of a packet. First, the length represents the length of the entire packet (the meaning used in services). Second, length represents the length of the message after the first four bytes (meaning used by an attacker).

In fact, normal services can be executed because of the definition of this length, but attack packets enter an infinite loop.

Continue sharing the code below:

in.readerIndex(in.readerIndex() - 4);
Copy the code

With the skipBytes and readUnsignedShort calls above, the read index of ByteBuf has reached the fourth byte. So in.readerIndex() returns a value of 4, and in.readerIndex(4-4) resets the read index to 0, i.e. reading from the beginning.

if (in.readableBytes() < pkgLength) {
    return;
}
Copy the code

After the read index moves to 0, check whether the number of bytes readable in the packet is smaller than the number specified in the packet content. Clearly, in.readableBytes() corresponds to a value of 44 bytes, while pkgLength is 40 bytes and does not return.

out.add(in.readBytes(pkgLength));
Copy the code

Read 40 bytes and output. There are four bytes left, and readIndex points to the 40th byte.

readableBytes = in.readableBytes();
Copy the code

Since readIndex already points to the 40th byte, the number of bytes readable at this point is 4.

Then, enter the second cycle. This is where the magic happens. We can see that the packet values of the last four bytes of the attack are all 0.

in.skipBytes(2);
int pkgLength = in.readUnsignedShort();
Copy the code

So after skipping 2 bytes, readIndex is 42, and pkgLength gets the values of bytes 43 and 44:0.

in.readerIndex(in.readerIndex() - 4);
Copy the code

The code sets readIndex to the 40th byte.

if (in.readableBytes() < pkgLength) {
    return;
}
Copy the code

ReadableBytes returns 4, but pkgLength has changed to 0 and will not return.

The next time you read the content, something happens:

out.add(in.readBytes(pkgLength)); ReadableBytes = in.readableBytes();Copy the code

The above readBytes read 0 bytes, while readableBytes is always 4. At this point, the entire while loop goes into an infinite loop, consuming CPU resources.

This is not the end of the story. At most, the CPU runs up to 100%, but when the null characters are written to the buffer that receives the data, the buffer frantically calls the Handler that processes the business, further intruding into the business processing logic.

Although the business logic layer makes a judgment and closes the connection, it has nothing to do with the connection at this point. The while loop has entered an infinite loop and closing the connection has no effect. At the same time, logs are generated at the service layer, and a large number of logs are generated to disks, causing disks to be overwritten.

CPU monitoring and disk monitoring alarms are generated. At first glance, it looked like another DDoS attack…

summary

To sum up, the length of the packet transmitted by the attacker is inconsistent with the specified length in the packet. As a result, the packet is parsed in an infinite loop.

Once the problem is found, it is easy to solve. In fact, through this incident also got some inspiration. First, when you encounter a problem, it is often the best solution to tackle it head-on. Avoiding can only delay the problem, but it cannot be solved. Second, as long as calm down to analyze, step by step analysis, there are few problems that can not be solved.


Program new horizon

\

The public account “program new vision”, a platform for simultaneous improvement of soft power and hard technology, provides massive information

\