Analysis of twemProxy source code for Twitter’s Redis&memcached Middleware

Twemproxy is redis and Memcached connection pool middleware

Github project address: github.com/twitter/twe…

Project Introduction Reference: github.com/twitter/twe…

Documentation: github.com/twitter/twe…

The core processes

The main process is to start an event loop, and all the logic is executed by calling the callback function from the event

Message flow

Messages are circulated among FDS of three roles: Client, Server, and proxy

role	role
client	The FD that connects to twemProxy on behalf of the application side
proxy	The FD representing the TwemProxy server, waiting for application side access
server	Twemproxy links redis/memcache fd and the proxy FD handler transfers data read from client FD to server FD

The core process is as follows:

Event system: epoll/ Kqueue/EVport encapsulation

Linux uses epoll, MAC uses Kqueue, and Sun Solaris uses EVport

In order to unify the underlying API calls, TwemProxy has unified the interfaces of the above three event processing apis: event/ NC_event. H.

function	role
event_base_create()	Create an event loop management FD
event_add_in()	Add a fd read event to the event manager
event_add_out()	The read and write events of a FD are managed by the event manager
event_add_conn()	The read and write events of a FD are managed by the event manager
event_wait()	Get the list of FD’s ready to read/write from the event manager and call CB for processing

Core structure event_base reference: SRC /event/nc_event.h

struct event_base {
    int                ep;      /* epoll descriptor */
    struct epoll_event *event;  /* event[] - events that were triggered int nevent; /* # event */
    event_cb_t         cb;      /* event callback */
};
Copy the code

field	role
ep	Event manager fd
struct epoll_event	The types of events that the event manager can handle (read, write, HUP signals)
int nevent	The maximum number of events
event_cb_t cb	Event handling callback function

Twemproxy makes extensive use of event callbacks to drive data flow, such as event_cb_t above

typedef int (*event_cb_t)(void *, uint32_t);
Copy the code

Core data structures and algorithms

Array Specifies a variable length array

struct array {
    uint32_t nelem;  /* # element */
    void     *elem;  /* element */
    size_t   size;   /* element size */
    uint32_t nalloc; /* # allocated element */
};
Copy the code

Dynamic arrays are implemented by adjusting nELEm and * ELEm pointer contents

String Indicates a variable length character string

struct string {
    uint32_t len;   /* string length */
    uint8_t  *data; /* string data */
};
Copy the code

Red and black tree

struct rbnode {
    struct rbnode *left;     /* left link */
    struct rbnode *right;    /* right link */
    struct rbnode *parent;   /* parent link */
    int64_t       key;       /* key for ordering */
    void          *data;     /* opaque data */
    uint8_t       color;     /* red | black */
};
Copy the code

Consistency of the hash

Reference: the SRC/hashkit nc_ketama. C

The queue

The queue encapsulation used by TwemProxy supports several different types of queues

Reference: SRC /nc_queue.h, this queue file comes from the FreeBSD Linx kernel: github.com/freebsd/fre…

Configuration file Parsing

Twemproxy uses the yamL configuration file, which is resolved using the Libyaml library, see github.com/yaml/libyam…

SRC /nc_conf.h SRC /nc_conf.c

Protocol Abstract interface

Twemproxy supports memcache and Redis. Abstracting the protocol into an interface can decouple the protocol from twemProxy’s main flow and facilitate the addition of new protocol support

memcached

void memcache_parse_req(struct msg *r);
void memcache_parse_rsp(struct msg *r);
bool memcache_failure(struct msg *r);
void memcache_pre_coalesce(struct msg *r);
void memcache_post_coalesce(struct msg *r);
rstatus_t memcache_add_auth(struct context *ctx, struct conn *c_conn, struct conn *s_conn);
rstatus_t memcache_fragment(struct msg *r, uint32_t ncontinuum, struct msg_tqh *frag_msgq);
rstatus_t memcache_reply(struct msg *r);
void memcache_post_connect(struct context *ctx, struct conn *conn, struct server *server);
void memcache_swallow_msg(struct conn *conn, struct msg *pmsg, struct msg *msg);
Copy the code

redis

void redis_parse_req(struct msg *r);
void redis_parse_rsp(struct msg *r);
bool redis_failure(struct msg *r);
void redis_pre_coalesce(struct msg *r);
void redis_post_coalesce(struct msg *r);
rstatus_t redis_add_auth(struct context *ctx, struct conn *c_conn, struct conn *s_conn);
rstatus_t redis_fragment(struct msg *r, uint32_t ncontinuum, struct msg_tqh *frag_msgq);
rstatus_t redis_reply(struct msg *r);
void redis_post_connect(struct context *ctx, struct conn *conn, struct server *server);
void redis_swallow_msg(struct conn *conn, struct msg *pmsg, struct msg *msg);
Copy the code

You can see that the Redis and memcached protocols are encapsulated into two sets of peer interfaces. For details, see nc_memcache.c nc_redis.c

The monitoring system

Twemproxy uses a separate thread to process monitoring data acquisition requests. See SRC/NC_stats.c :: stats_start_Aggregator

static rstatus_t
stats_start_aggregator(struct stats *st)
{
    rstatus_t status;
    // error handle...
    status = stats_listen(st);
    // error handle...

    status = pthread_create(&st->tid, NULL, stats_loop, st);
    // error handle...
    return NC_OK;
}
Copy the code

The main thread writes real-time monitoring data to this structure. After receiving the client’s request for monitoring data, the monitoring thread reads the monitoring data from this structure and outputs it to the client. The core structure reference is SRC/NC_stats.h

typedef enum stats_type {
    // ...
} stats_type_t;

struct stats_metric {
    // ...
};

struct stats_server {
    // ...
};

struct stats_pool {
    // ...
};

struct stats_buffer {
    // ...
};

struct stats {
    // ...
};
Copy the code

Using the curl command -s 127.0.0.1:22222 | jq. You can view real-time monitoring information

The signal processing

For details, see SRC /nc_signal.h SRC /nc_signal.c

static struct signal signals[] = {
    { SIGUSR1, "SIGUSR1".0,                 signal_handler },
    { SIGUSR2, "SIGUSR2".0,                 signal_handler },
    { SIGTTIN, "SIGTTIN".0,                 signal_handler },
    { SIGTTOU, "SIGTTOU".0,                 signal_handler },
    { SIGHUP,  "SIGHUP".0,                 signal_handler },
    { SIGINT,  "SIGINT".0,                 signal_handler },
    { SIGSEGV, "SIGSEGV", (int)SA_RESETHAND, signal_handler },
    { SIGPIPE, "SIGPIPE".0,                 SIG_IGN },
    { 0.NULL.0.NULL}};Copy the code

You can see that TwemProxy calls back multiple signals: Signal_Handler

void
signal_handler(int signo)
{
    / /...

    switch (signo) {
    case SIGUSR1:
        break;

    case SIGUSR2:
        break;

    case SIGTTIN:
        actionstr = ", up logging level";
        action = log_level_up;
        break;

    case SIGTTOU:
        actionstr = ", down logging level";
        action = log_level_down;
        break;

    case SIGHUP:
        actionstr = ", reopening log file";
        action = log_reopen;
        break;

    case SIGINT:
        done = true;
        actionstr = ", exiting";
        break;

    case SIGSEGV:
        log_stacktrace();
        actionstr = ", core dumping";
        raise(SIGSEGV);
        break;

    default:
        NOT_REACHED();
    }

    log_safe("signal %d (%s) received%s", signo, sig->signame, actionstr);

    if(action ! =NULL) {
        action();
    }

    if (done) {
        exit(1); }}Copy the code

You can see the use of SIGTTIN/ to increase the log output level, SIGTTOU to decrease the log input level, SIGHUP to reopen the log file, SIGINT to exit the program, and SIGSEGV to output the memory core dump file

A couple of points to note

This paper is written based on TwemProxy 0.4.1

Although CODIS is now the de facto choice for Redis middleware, TwemProxy, as the first memcache& Redis middleware, is still of great learning value and supports memcache

The resources

1. Man7.org/linux/man-p…
2. Blog. Just4fun. Site/command – too…
3. Unix.stackexchange.com/questions/1…
4. www.jianshu.com/p/83ac498e1…