In this article you can thoroughly understand the Redis serialization protocol. you can also write Redis clients

The premise

Recently, WHEN I was learning Netty, I wanted to make a coding and decoding module based on Redis service protocol. In the process, I read RESP service serialization protocol, translated the document based on my own understanding and simply realized RESP based on Java language. The JDK version used for this article is [8+].

Introduction of RESP

The Redis client communicates with the Redis server based on a Protocol called RESP, which stands for Redis Serialization Protocol. Although RESP was designed for Redis, it can also be used in other client-server software projects. In the design of RESP, the following points are taken into account:

Easy to implement.
Quick parsing.
High readability.

RESP can serialize different data types, such as integers, strings, arrays, and a special Error type. The Redis command that needs to be executed is encapsulated as a request like an array of strings and sent to the Redis server via the Redis client. Redis server replies with Redis replies selecting the corresponding data type with a command-specific data type.

RESP is binary-safe and there is no need to process bulk data transferred from one process to another under RESP because it uses prefixed-length, which is the number of data blocks defined in the prefix of each data block. Similar to Netty’s fixed-length encoding and decoding) to transfer bulk data.

Note: The protocols outlined here are only used for client-server communication, Redis Cluster uses different binary protocols to exchange messages between multiple nodes (i.e., nodes in the Redis Cluster do not use RESP communication).

The network layer

The Redis client connects to the Redis server by creating a TCP connection on port 6379.

Although RESP is non-TCP specific in terms of underlying communication protocol technology, in the context of Redis, RESP is only used for TCP connections (or similar stream-oriented connections, such as Unix sockets).

Request-response model

The Redis server receives a command consisting of different parameters, receives the command, processes it, and sends a reply back to the Redis client. This is the simplest model, but there are two exceptions:

RedisSupport pipeline (PipeliningAssembly line, more commonly known as pipeline) operation. In the case of pipes,RedisA client can send multiple commands at once and then wait for a one-time replyreplies, understood asRedisThe server will return a batch reply result at once.
whenRedisClient subscriptionPub/SubWhen channeling, the protocol changes the semantics and becomes a push protocol (push protocol), that is, the client no longer needs to send commands becauseRedisThe server will automatically send a new message to the client (which subscribes to the channel)RedisThe server actively pushes to subscribers to a particular channelRedisClient).

Redis protocol is a simple request-response protocol, except for the two exceptions mentioned above.

Data types supported by RESP

RESP was introduced in Redis 1.2, and in Redis 2.0, RESP officially became the standard solution for communicating with Redis servers. If you need to write a Redis client, you must implement this protocol in the client.

RESP is essentially a serialization protocol that supports the following data types: single-line strings, error messages, integer numbers, fixed-length strings, and RESP arrays.

RESP is used as a request-response protocol in Redis in the following ways:

RedisThe client encapsulates the command asRESPArray type (Array elements are fixed-length stringsIt is important to note thatRedisThe server.
RedisThe server selects the corresponding command based on the command implementationRESPOne of the data types to reply to.

In RESP, the data type depends on the first byte of the datagram:

The first byte of a single-line string is+.
The first byte of the error message is-.
The first byte of an integer number is:.
The first byte of a fixed-length string is$.
RESPThe first byte of the array is*.

In addition, you can use a fixed length string or a special variant of an array to represent Null values in RESP, as described below. In RESP, different parts of the agreement are always terminated with \ R \ N (CRLF).

The current five data types in RESP are summarized as follows:

The data type	Translation of this article	The basic characteristics of	example
`Simple String`	One-line string	The first byte is`+`, the last two bytes are`\r\n`The other bytes are the string contents	`+OK\r\n`
`Error`	The error message	The first byte is`-`, the last two bytes are`\r\n`The other bytes are the text content of the exception message	`-ERR\r\n`
`Integer`	integer	The first byte is`:`, the last two bytes are`\r\n`The other bytes are the text content of the number	`:100\r\n`
`Bulk String`	Fixed length string	The first byte is`$`, and the next byte is`Content string length \r\n`, the last two bytes are`\r\n`The other bytes are the string contents	`$4\r\ndoge\r\n`
`Array`	`RESP`An array of	The first byte is`*`, and the next byte is`Number of elements \r\n`, the last two bytes are`\r\n`Other bytes are the contents of individual elements, each of which can be of any data type	`*2\r\n:100\r\n$4\r\ndoge\r\n`

The following sections provide a more detailed analysis of each data type.

RESP Simple String -Simple String

Simple strings are encoded as follows:

(1) The first byte is+.
(2) What follows is a cannot containCRorLFCharacter string.
(3)CRLFTermination.

Simple strings guarantee the transfer of non-binary safe strings with minimal overhead. For example, after many Redis commands are executed successfully, the server needs to reply with the OK string. In this case, the data packet encoded into 5 bytes is as follows:

+OK\r\n
Copy the code

If you want to send binary-safe strings, you need to use fixed-length strings.

When the Redis server responds with a simple string, the Redis client library should return a string to the caller that consists of characters from + to the end of the string content (the content of part (2) mentioned above), excluding the last CRLF byte.

RESP Error message -Error

Error message types are RESP-specific data types. In fact, the error message type is basically the same as the simple string type, except that the first byte is -. The main difference between the error message type and the simple string type is that the error message should be perceived by the client as an exception when the Redis server responds, whereas the string content of the error message should be perceived as an error message returned by the Redis server. Error messages are encoded as follows:

(1) The first byte is-.
(2) What follows is a cannot containCRorLFCharacter string.
(3)CRLFTermination.

A simple example is as follows:

-Error message\r\n
Copy the code

The Redis server responds to error messages only when an actual error has occurred or a perceived error has occurred, such as an attempt to perform an operation on the wrong data type or a command that does not exist. When the Redis client receives an error message, it should raise an exception (usually throwing an exception directly, which can be classified according to the content of the error message). Here are some examples of error message responses:

-ERR unknown command 'foobar'
-WRONGTYPE Operation against a key holding the wrong kind of value
Copy the code

– The content between the first word after it and the first space or newline character, representing the type of error returned. This is just the convention Redis uses, not part of the RESP error message format.

For example, ERR is a general error, and WRONGTYPE is a more specific error, indicating that the client tries to perform an operation against the wrong data type. This definition, called an error prefix, is a way for clients to understand the type of error returned by the server without relying on the exact message definition given, which may change over time.

The client implementation can return different kinds of exceptions for different error types, or it can provide a generic way to catch the error by providing the name of the error type directly to the caller as a string.

However, the ability to categorize error message handling should not be considered critical because it is not very useful, and some client implementations may simply return specific values to mask error messages as generic exception handling, such as simply returning false.

RESP Integer -Integer

Integer numbers are encoded as follows:

(1) The first byte is:.
(2) What follows is a cannot containCRorLFThe string of characters, that is, the number, is first converted to a sequence of characters and finally printed as bytes.
(3)CRLFTermination.

Such as:

:0\r\n
:1000\r\n
Copy the code

Many Redis commands return integer numbers, such as INCR, LLEN, and LASTSAVE commands.

The returned integer numbers have no special meaning, like INCR, which returns the total increment, and LASTSAVE, which is a UNIX timestamp. But the Redis server ensures that the integer number returned is within the range of a signed 64-bit integer.

In some cases, the returned integer number may refer to true or false. For example, the EXISTS or SISMEMBER command returns 1 for true and 0 for false.

In some cases, the returned integer number will indicate whether the command actually had an effect. For example, if the SADD, SREM, and SETNX commands are executed, 1 indicates that the command takes effect, and 0 indicates that the command does not take effect (the command is not executed).

The following commands return integer numbers: SETNX, DEL EXISTS, INCR, INCRBY, DECR, DECRBY, DBSIZE, LASTSAVE, RENAMENX, MOVE, LLEN, SADD, SREM, SISMEMBER, SCARD.

RESP Fixed length String -Bulk String

A fixed-length string is used to represent a binary-safe string (Bulk) with a maximum length of 512MB. Fixed-length strings are encoded as follows:

(1) The first byte is$.
(2) Immediately following is the length of the number of bytes that make up the string (calledprefixed length, that is, the prefix length), the prefix length is divided into blocksCRLFTermination.
(3) And then a cannot containCRorLFThe string of characters, that is, the number, is first converted to a sequence of characters and finally printed as bytes.
(4)CRLFTermination.

For example, doge uses fixed-length string encoding as follows:

First byte	The prefix length	`CRLF`	String content	`CRLF`		Fixed length string
`$`	`4`	`\r\n`	`doge`	`\r\n`	`= = = >`	`$4\r\ndoge\r\n`

Foobar uses a fixed-length string encoding as follows:

First byte	The prefix length	`CRLF`	String content	`CRLF`		Fixed length string
`$`	`6`	`\r\n`	`foobar`	`\r\n`	`= = = >`	`$6\r\nfoobar\r\n`

When representing an Empty String (“” in Java), the fixed-length String is encoded as follows:

First byte	The prefix length	`CRLF`	`CRLF`		Fixed length string
`$`	`0`	`\r\n`	`\r\n`	`= = = >`	`$0\r\n\r\n`

Fixed-length strings can also use a special format to represent Null values, indicating that they do not exist. In this special format, the prefix length is -1 and there is no data, so Null values are encoded using a fixed-length string as follows:

First byte	The prefix length	`CRLF`		Fixed length string
`$`	`- 1`	`\r\n`	`= = = >`	`$-1\r\n`

When the Redis server returns a fixed-length string encoding Null, the client should not return an empty string, but a Null object in the corresponding programming language. Nil in Ruby, NULL in C, NULL in Java, and so on.

RESP Array – Array

The Redis client sends commands to the Redis server using the RESP array. Similarly, some Redis commands require the server to use the RESP array type to return the collection of elements to the client, such as the LRANGE command that returns a list of elements. The RESP array is not exactly the same as the array we know, and its encoding format is as follows:

(1) The first byte is*.
(2) Next comes compositionRESPThe number of elements in the array (decimal number, but eventually needs to be converted to a sequence of bytes, such as 101and0Two adjacent bytes), the number of elements is divided into blocksCRLFTermination.
(3)RESPThe contents of each element of the array, and each element can be arbitraryRESPData type.

An empty RESP array is encoded as follows:

*0\r\n
Copy the code

An RESP array containing two fixed-length string elements foo and bar is encoded as follows:

*2\r\n$3\r\nfoo\r\n$3\r\nbar\r\n
Copy the code

The general format is *

CRLF as the prefix part of the RESP array, while the elements of the other data types that make up the RESP array are simply concatenated one after the other. For example, an RESP array containing three elements of type integer is encoded as follows:

*3\r\n:1\r\n:2\r\n:3\r\n
Copy the code

The elements of RESP arrays do not have to be of the same data type and can contain elements of mixed types. For example, here is the encoding of an RESP array containing four integer elements and one fixed-length string element (a total of five elements) (this can’t actually be done in multiple lines for clarity) :

#Number of elements
*5\r\n
#Element of the first integer type
:1\r\n
#Element of the second integer type
:2\r\n
#Element of the third integer type
:3\r\n
#Element of the fourth integer type
:4\r\n
#A fixed-length string element
$6\r\n
foobar\r\n
Copy the code

The first line *5\r\n of the Redis server response defines 5 Reply data to be followed by 5 Reply data, each of which is then used as an element item to form the Multi Bulk Reply for transport.

This is analogous to an ArrayList (generic erase) in Java, somewhat similar to the following pseudocode:

List encode = new ArrayList();
// Add the number of elements
encode.add(elementCount);
encode.add(CRLF);
// Add the first element of type -1
encode.add(':');
encode.add(1);
encode.add(CRLF);
// Add element 2 of type 2
encode.add(':');
encode.add(2);
encode.add(CRLF);
// Add a third element of integer type - 3
encode.add(':');
encode.add(3);
encode.add(CRLF);
// Add element 4 of type 4
encode.add(':');
encode.add(4);
encode.add(CRLF);
// Add a fixed-length string element
encode.add('$');
// Prefix length
encode.add(6);
// The content of the string
encode.add("foobar");
encode.add(CRLF);
Copy the code

The concept of Null values also exists in RESP arrays, which is called RESP Null Array. For historical reasons, a special encoding is used to define Null values in RESP arrays, as distinct from null-valued strings in fixed-length strings. For example, when the BLPOP command times out, a RESP Null Array response is returned. RESP Null Array codes are as follows:

*-1\r\n
Copy the code

When the Redis server returns a RESP Null Array, the client should return a Null object, not an empty Array or list. This is important, as it is the difference between an empty array reply (that is, the command executed correctly and returned normal) and something else (such as a BLPOP timeout).

The elements of the RESP array can also be RESP arrays. Here is an RESP array containing two elements of the RESP array type, encoded as follows:

#Number of elements
*2\r\n
#The first RESP array element
*3\r\n
:1\r\n
:2\r\n
:3\r\n
#The second RESP array element
*2\r\n
+Foo\r\n
-Bar\r\n
Copy the code

The above RESP array contains two elements of RESP array type. The first RESP array element contains three elements of integer type, and the second RESP array element contains one element of simple string type and one element of error message type.

RESPA Null element in an array

Individual elements in RESP arrays also have the concept of Null values, hereinafter referred to as Null elements. If the array type is RESP and there is a Null element in the RESP array, the element is missing and must not be replaced by an empty string. This can happen with the SORT command when used with the GET mode option in the absence of a specified key.

Here is an example of an RESP array that contains Null elements (you can’t really do this by encoding it in multiple lines for clarity) :

*3\r\n
$3\r\n
foo\r\n
$-1\r\n
$3\r\n
bar\r\n
Copy the code

The second element in the RESP array is the Null element, and the client API should eventually return:

# Ruby
["foo",nil,"bar"]
# Java
["foo",null,"bar"]
Copy the code

Write a high-performance parser based on RESP

The java.nio.byteBuffer provided by JDK cannot be automatically expanded and needs to be switched to read/write mode. Therefore, Netty is directly introduced here and ByteBuf provided by Netty is used for RESP data type resolution. At the time of writing, the latest version of Netty is 4.1.42.final. Introducing dependencies:

<dependency>
    <groupId>io.netty</groupId>
    <artifactId>netty-buffer</artifactId>
    <version>4.1.42. The Final</version>
</dependency>
Copy the code

Define the decoder interface:

public interface RespDecoder<V>{
    
    V decode(ByteBuf buffer);
}
Copy the code

Define constants:

public class RespConstants {

    public static final Charset ASCII = StandardCharsets.US_ASCII;
    public static final Charset UTF_8 = StandardCharsets.UTF_8;

    public static final byte DOLLAR_BYTE = '$';
    public static final byte ASTERISK_BYTE = The '*';
    public static final byte PLUS_BYTE = '+';
    public static final byte MINUS_BYTE = The '-';
    public static final byte COLON_BYTE = ':';

    public static final String EMPTY_STRING = "";
    public static final Long ZERO = 0L;
    public static final Long NEGATIVE_ONE = -1L;
    public static final byte CR = (byte) '\r';
    public static final byte LF = (byte) '\n';
    public static final byte[] CRLF = "\r\n".getBytes(ASCII);

    public enum ReplyType {

        SIMPLE_STRING,

        ERROR,

        INTEGER,

        BULK_STRING,

        RESP_ARRAY
    }
}
Copy the code

The implementation of the parsing module in the following sections already ignores the parsing of the first byte, because the first byte determines the specific data type.

Parsing simple strings

A simple String type is a single-line String that is parsed to the Java String type. The decoder implementation is as follows:

// Parse single-line strings
public class LineStringDecoder implements RespDecoder<String> {

    @Override
    public String decode(ByteBuf buffer) {
        returnCodecUtils.X.readLine(buffer); }}public enum CodecUtils {

    X;

    public int findLineEndIndex(ByteBuf buffer) {
        int index = buffer.forEachByte(ByteProcessor.FIND_LF);
        return (index > 0 && buffer.getByte(index - 1) = ='\r')? index : -1;
    }

    public String readLine(ByteBuf buffer) {
        int lineEndIndex = findLineEndIndex(buffer);
        if (lineEndIndex > -1) {
            int lineStartIndex = buffer.readerIndex();
            // Calculate the length of bytes
            int size = lineEndIndex - lineStartIndex - 1;
            byte[] bytes = new byte[size];
            buffer.readBytes(bytes);
            // Reset the read cursor to the first byte after \r\n
            buffer.readerIndex(lineEndIndex + 1);
            buffer.markReaderIndex();
            return new String(bytes, RespConstants.UTF_8);
        }
        return null; }}public class RespSimpleStringDecoder extends LineStringDecoder {}Copy the code

A class LineStringDecoder is extracted to parse single-line strings so that an inheritance can be done while parsing error messages. Test it out:

public static void main(String[] args) throws Exception {
    ByteBuf buffer = ByteBufAllocator.DEFAULT.buffer();
    // +OK\r\n
    buffer.writeBytes("+OK".getBytes(RespConstants.UTF_8));
    buffer.writeBytes(RespConstants.CRLF);
    String value = RespCodec.X.decode(buffer);
    log.info("Decode result:{}", value);
}
// Decode result:OK
Copy the code

Parsing error messages

The nature of the error message is also a single-line string, so its decoding implementation can be the same as that of a simple string. The decoder for the error message data type is as follows:

public class RespErrorDecoder extends LineStringDecoder {}Copy the code

Test it out:

public static void main(String[] args) throws Exception {
    ByteBuf buffer = ByteBufAllocator.DEFAULT.buffer();
    // -ERR unknown command 'foobar'\r\n
    buffer.writeBytes("-ERR unknown command 'foobar'".getBytes(RespConstants.UTF_8));
    buffer.writeBytes(RespConstants.CRLF);
    String value = RespCodec.X.decode(buffer);
    log.info("Decode result:{}", value);
}
// Decode result:ERR unknown command 'foobar'
Copy the code

Parse integer numbers

Integer numeric types are essentially long integers that need to be restored from a byte sequence in a signed 64bit, because they are signed. The type identifies the bit: the first byte after the byte needs to be checked for negative characters – because it is parsed from left to right, and the current numeric value is multiplied by 10 for each new bit parsed. The implementation of its decoder is as follows:

public class RespIntegerDecoder implements RespDecoder<Long> {

    @Override
    public Long decode(ByteBuf buffer) {
        int lineEndIndex = CodecUtils.X.findLineEndIndex(buffer);
        // No line end, exception
        if (-1 == lineEndIndex) {
            return null;
        }
        long result = 0L;
        int lineStartIndex = buffer.readerIndex();
        boolean negative = false;
        byte firstByte = buffer.getByte(lineStartIndex);
        / / negative
        if (RespConstants.MINUS_BYTE == firstByte) {
            negative = true;
        } else {
            int digit = firstByte - '0';
            result = result * 10 + digit;
        }
        for (int i = lineStartIndex + 1; i < (lineEndIndex - 1); i++) {
            byte value = buffer.getByte(i);
            int digit = value - '0';
            result = result * 10 + digit;
        }
        if (negative) {
            result = -result;
        }
        // Reset the read cursor to the first byte after \r\n
        buffer.readerIndex(lineEndIndex + 1);
        returnresult; }}Copy the code

The analysis of integer numeric types is relatively complicated, so we must pay attention to negative numbers. Test it out:

public static void main(String[] args) throws Exception {
    ByteBuf buffer = ByteBufAllocator.DEFAULT.buffer();
    // :-1000\r\n
    buffer.writeBytes(": - 1000".getBytes(RespConstants.UTF_8));
    buffer.writeBytes(RespConstants.CRLF);
    Long value = RespCodec.X.decode(buffer);
    log.info("Decode result:{}", value);
}
// Decode result:-1000
Copy the code

Parse a fixed-length string

The key of fixed-length string type parsing is to read the first byte sequence after the type identifier $into a 64-bit signed integer to determine the length of the string content to be parsed in bytes, and then read the following bytes according to that length. Its decoder implementation is as follows:

public class RespBulkStringDecoder implements RespDecoder<String> {

    @Override
    public String decode(ByteBuf buffer) {
        int lineEndIndex = CodecUtils.X.findLineEndIndex(buffer);
        if (-1 == lineEndIndex) {
            return null;
        }
        // Use RespIntegerDecoder to read length
        Long length = (Long) DefaultRespCodec.DECODERS.get(ReplyType.INTEGER).decode(buffer);
        if (null == length) {
            return null;
        }
        // Bulk Null String
        if (RespConstants.NEGATIVE_ONE.equals(length)) {
            return null;
        }
        // Bulk Empty String
        if (RespConstants.ZERO.equals(length)) {
            return RespConstants.EMPTY_STRING;
        }
        // The length of the actual byte content
        int readLength = (int) length.longValue();
        if (buffer.readableBytes() > readLength) {
            byte[] bytes = new byte[readLength];
            buffer.readBytes(bytes);
            // Reset the read cursor to the first byte after \r\n
            buffer.readerIndex(buffer.readerIndex() + 2);
            return new String(bytes, RespConstants.UTF_8);
        }
        return null; }}Copy the code

Test it out:

public static void main(String[] args) throws Exception{
    ByteBuf buffer = ByteBufAllocator.DEFAULT.buffer();
    // $6\r\nthrowable\r\n
    buffer = ByteBufAllocator.DEFAULT.buffer();
    buffer.writeBytes("$9".getBytes(RespConstants.UTF_8));
    buffer.writeBytes(RespConstants.CRLF);
    buffer.writeBytes("throwable".getBytes(RespConstants.UTF_8));
    buffer.writeBytes(RespConstants.CRLF);
    String value = RespCodec.X.decode(buffer);
    log.info("Decode result:{}", value);
}
// Decode result:throwable
Copy the code

Parse the RESP array

Key to RESP array type resolution:

The type identifier is read first*The last first byte sequence is partitioned to 64bitA signed integer that determines the number of elements in the array.
Each element is resolved recursively.

I have referred to a lot of Redis protocol parsing framework, many of which are implemented by stack or state machine. Here is a simple recursive implementation, and the decoder code is as follows:

public class RespArrayDecoder implements RespDecoder {

    @Override
    public Object decode(ByteBuf buffer) {
        int lineEndIndex = CodecUtils.X.findLineEndIndex(buffer);
        if (-1 == lineEndIndex) {
            return null;
        }
        // Parse the number of elements
        Long length = (Long) DefaultRespCodec.DECODERS.get(ReplyType.INTEGER).decode(buffer);
        if (null == length) {
            return null;
        }
        // Null Array
        if (RespConstants.NEGATIVE_ONE.equals(length)) {
            return null;
        }
        // Array Empty List
        if (RespConstants.ZERO.equals(length)) {
            return Lists.newArrayList();
        }
        List<Object> result = Lists.newArrayListWithCapacity((int) length.longValue());
        / / recursion
        for (int i = 0; i < length; i++) {
            result.add(DefaultRespCodec.X.decode(buffer));
        }
        returnresult; }}Copy the code

Test it out:

public static void main(String[] args) throws Exception {
    ByteBuf buffer = ByteBufAllocator.DEFAULT.buffer();
    //*2\r\n$3\r\nfoo\r\n$3\r\nbar\r\n
    buffer = ByteBufAllocator.DEFAULT.buffer();
    buffer.writeBytes("* 2".getBytes(RespConstants.UTF_8));
    buffer.writeBytes(RespConstants.CRLF);
    buffer.writeBytes("$3".getBytes(RespConstants.UTF_8));
    buffer.writeBytes(RespConstants.CRLF);
    buffer.writeBytes("foo".getBytes(RespConstants.UTF_8));
    buffer.writeBytes(RespConstants.CRLF);
    buffer.writeBytes("$3".getBytes(RespConstants.UTF_8));
    buffer.writeBytes(RespConstants.CRLF);
    buffer.writeBytes("bar".getBytes(RespConstants.UTF_8));
    buffer.writeBytes(RespConstants.CRLF);
    List value = RespCodec.X.decode(buffer);
    log.info("Decode result:{}", value);
}
// Decode result:[foo, bar]
Copy the code

summary

After a relatively deep understanding of RESP content and its encoding and decoding process, Redis service encoding and decoding module can be written based on Netty, as a very meaningful example of Netty entry. The last section of this article only demonstrates the decoding part of RESP, the encoding module and more details will be shown in another article implementing Redis clients with Netty.

References:

Redis Protocol specification

link

Hope you can read this and find me:

Making Page: www.throwable.club/2019/10/09/…
Coding Page: throwable. Coding. Me / 2019/10/09 /…

The appendix

All the code covered in this article:

public class RespConstants {

    public static final Charset ASCII = StandardCharsets.US_ASCII;
    public static final Charset UTF_8 = StandardCharsets.UTF_8;

    public static final byte DOLLAR_BYTE = '$';
    public static final byte ASTERISK_BYTE = The '*';
    public static final byte PLUS_BYTE = '+';
    public static final byte MINUS_BYTE = The '-';
    public static final byte COLON_BYTE = ':';

    public static final String EMPTY_STRING = "";
    public static final Long ZERO = 0L;
    public static final Long NEGATIVE_ONE = -1L;
    public static final byte CR = (byte) '\r';
    public static final byte LF = (byte) '\n';
    public static final byte[] CRLF = "\r\n".getBytes(ASCII);

    public enum ReplyType {

        SIMPLE_STRING,

        ERROR,

        INTEGER,

        BULK_STRING,

        RESP_ARRAY
    }
}

public enum CodecUtils {

    X;

    public int findLineEndIndex(ByteBuf buffer) {
        int index = buffer.forEachByte(ByteProcessor.FIND_LF);
        return (index > 0 && buffer.getByte(index - 1) = ='\r')? index : -1;
    }

    public String readLine(ByteBuf buffer) {
        int lineEndIndex = findLineEndIndex(buffer);
        if (lineEndIndex > -1) {
            int lineStartIndex = buffer.readerIndex();
            // Calculate the length of bytes
            int size = lineEndIndex - lineStartIndex - 1;
            byte[] bytes = new byte[size];
            buffer.readBytes(bytes);
            // Reset the read cursor to the first byte after \r\n
            buffer.readerIndex(lineEndIndex + 1);
            buffer.markReaderIndex();
            return new String(bytes, RespConstants.UTF_8);
        }
        return null; }}public interface RespCodec {

    RespCodec X = DefaultRespCodec.X;

    <IN, OUT> OUT decode(ByteBuf buffer);

    <IN, OUT> ByteBuf encode(IN in);
}

public enum DefaultRespCodec implements RespCodec {

    X;

    static final Map<ReplyType, RespDecoder> DECODERS = Maps.newConcurrentMap();
    private static final RespDecoder DEFAULT_DECODER = new DefaultRespDecoder();

    static {
        DECODERS.put(ReplyType.SIMPLE_STRING, new RespSimpleStringDecoder());
        DECODERS.put(ReplyType.ERROR, new RespErrorDecoder());
        DECODERS.put(ReplyType.INTEGER, new RespIntegerDecoder());
        DECODERS.put(ReplyType.BULK_STRING, new RespBulkStringDecoder());
        DECODERS.put(ReplyType.RESP_ARRAY, new RespArrayDecoder());
    }

    @SuppressWarnings("unchecked")
    @Override
    public <IN, OUT> OUT decode(ByteBuf buffer) {
        return (OUT) DECODERS.getOrDefault(determineReplyType(buffer), DEFAULT_DECODER).decode(buffer);
    }

    private ReplyType determineReplyType(ByteBuf buffer) {
        byte firstByte = buffer.readByte();
        ReplyType replyType;
        switch (firstByte) {
            case RespConstants.PLUS_BYTE:
                replyType = ReplyType.SIMPLE_STRING;
                break;
            case RespConstants.MINUS_BYTE:
                replyType = ReplyType.ERROR;
                break;
            case RespConstants.COLON_BYTE:
                replyType = ReplyType.INTEGER;
                break;
            case RespConstants.DOLLAR_BYTE:
                replyType = ReplyType.BULK_STRING;
                break;
            case RespConstants.ASTERISK_BYTE:
                replyType = ReplyType.RESP_ARRAY;
                break;
            default: {
                throw new IllegalArgumentException("first byte:"+ firstByte); }}return replyType;
    }

    @Override
    public <IN, OUT> ByteBuf encode(IN in) {
        // TODO
        throw new UnsupportedOperationException("encode"); }}public interface RespDecoder<V> {

    V decode(ByteBuf buffer);
}

public class DefaultRespDecoder implements RespDecoder {

    @Override
    public Object decode(ByteBuf buffer) {
        throw new IllegalStateException("decoder"); }}public class LineStringDecoder implements RespDecoder<String> {

    @Override
    public String decode(ByteBuf buffer) {
        returnCodecUtils.X.readLine(buffer); }}public class RespSimpleStringDecoder extends LineStringDecoder {}public class RespErrorDecoder extends LineStringDecoder {}public class RespIntegerDecoder implements RespDecoder<Long> {

    @Override
    public Long decode(ByteBuf buffer) {
        int lineEndIndex = CodecUtils.X.findLineEndIndex(buffer);
        // No line end, exception
        if (-1 == lineEndIndex) {
            return null;
        }
        long result = 0L;
        int lineStartIndex = buffer.readerIndex();
        boolean negative = false;
        byte firstByte = buffer.getByte(lineStartIndex);
        / / negative
        if (RespConstants.MINUS_BYTE == firstByte) {
            negative = true;
        } else {
            int digit = firstByte - '0';
            result = result * 10 + digit;
        }
        for (int i = lineStartIndex + 1; i < (lineEndIndex - 1); i++) {
            byte value = buffer.getByte(i);
            int digit = value - '0';
            result = result * 10 + digit;
        }
        if (negative) {
            result = -result;
        }
        // Reset the read cursor to the first byte after \r\n
        buffer.readerIndex(lineEndIndex + 1);
        returnresult; }}public class RespBulkStringDecoder implements RespDecoder<String> {

    @Override
    public String decode(ByteBuf buffer) {
        int lineEndIndex = CodecUtils.X.findLineEndIndex(buffer);
        if (-1 == lineEndIndex) {
            return null;
        }
        Long length = (Long) DefaultRespCodec.DECODERS.get(ReplyType.INTEGER).decode(buffer);
        if (null == length) {
            return null;
        }
        // Bulk Null String
        if (RespConstants.NEGATIVE_ONE.equals(length)) {
            return null;
        }
        // Bulk Empty String
        if (RespConstants.ZERO.equals(length)) {
            return RespConstants.EMPTY_STRING;
        }
        // The length of the actual byte content
        int readLength = (int) length.longValue();
        if (buffer.readableBytes() > readLength) {
            byte[] bytes = new byte[readLength];
            buffer.readBytes(bytes);
            // Reset the read cursor to the first byte after \r\n
            buffer.readerIndex(buffer.readerIndex() + 2);
            return new String(bytes, RespConstants.UTF_8);
        }
        return null; }}public class RespArrayDecoder implements RespDecoder {

    @Override
    public Object decode(ByteBuf buffer) {
        int lineEndIndex = CodecUtils.X.findLineEndIndex(buffer);
        if (-1 == lineEndIndex) {
            return null;
        }
        // Parse the number of elements
        Long length = (Long) DefaultRespCodec.DECODERS.get(ReplyType.INTEGER).decode(buffer);
        if (null == length) {
            return null;
        }
        // Null Array
        if (RespConstants.NEGATIVE_ONE.equals(length)) {
            return null;
        }
        // Array Empty List
        if (RespConstants.ZERO.equals(length)) {
            return Lists.newArrayList();
        }
        List<Object> result = Lists.newArrayListWithCapacity((int) length.longValue());
        / / recursion
        for (int i = 0; i < length; i++) {
            result.add(DefaultRespCodec.X.decode(buffer));
        }
        returnresult; }}Copy the code

(E-A-20191009 C-2-D)

Technical official account (Throwable Digest), push the author’s original technical articles from time to time (never plagiarize or reprint) :

Entertainment public account (” Sand carving every Day “), select the interesting sand carving text and video push irregularly, relieve the pressure of life and work:

mo4tech.com (Moment For Technology) is a global community with thousands techies from across the global hang out!Passionate technologists, be it gadget freaks, tech enthusiasts, coders, technopreneurs, or CIOs, you would find them all here.

In this article you can thoroughly understand the Redis serialization protocol. you can also write Redis clients

The premise

Introduction of RESP

The network layer

Request-response model

Data types supported by RESP

RESP Simple String -Simple String

RESP Error message -Error

RESP Integer -Integer

RESP Fixed length String -Bulk String

RESP Array – Array

Other RESP related content

Send the command to the Redis server

Batch commands and pipelines

Inline command

Write a high-performance parser based on RESP

Parsing simple strings

Parsing error messages

Parse integer numbers

Parse a fixed-length string

Parse the RESP array

summary

link

The appendix

In this article you can thoroughly understand the Redis serialization protocol. you can also write Redis clients

The premise

Introduction of RESP

The network layer

Request-response model

Data types supported by RESP

RESP Simple String -Simple String

RESP Error message -Error

RESP Integer -Integer

RESP Fixed length String -Bulk String

RESP Array – Array

Other RESP related content

Send the command to the Redis server

Batch commands and pipelines

Inline command

Write a high-performance parser based on RESP

Parsing simple strings

Parsing error messages

Parse integer numbers

Parse a fixed-length string

Parse the RESP array

summary

link

The appendix

Related Posts

NG family barrel full stack project practice summary

HashMap source code analysis study notes

How to design a stable and reliable second kill system