Set up a distributed IM system for yourself

CIM(cross-IM) An IM system for developers. It also provides components to help developers build their own horizontally extensible IM.

With CIM you can fulfill the following requirements:

IM instant messaging system.

Message push middleware for APP.

Transparent message transmission middleware in massive IOT connection scenarios.

Architecture design

Let’s look at the specific architectural design.

All components of the CIM are built using SpringBoot.

Netty + Google Protocol Buffer was used to build the underlying communication.

Redis stores routing information, account information and online status of each client.

Zookeeper is used to register and discover the IM server service.

The whole is mainly composed of the following modules:

cim-server

IM server; Used to receive client connections, transparent message transmission, and message push.

Cluster deployment is supported.

cim-forward-route

Message routing server; It is used to process message routing, message forwarding, user login, user logout, and some operation tools (obtaining online users, etc.).

cim-client

IM client; A messaging terminal for users that can be started with a command and initiate communication with others (group chat, private chat); At the same time built-in some common commands for easy use.

The flow chart

The overall process is also relatively simple, the flow chart is as follows:

The client initiates a login to route.

Successful login Select an available IM-server from Zookeeper and return it to the client, and save the login and routing information to Redis.

The client initiates a long connection to the IM-server and maintains the heartbeat after the connection succeeds.

When a client goes offline, use route to clear status information.

So when we deploy ourselves we need the following steps:

Build basic middleware Redis and Zookeeper.

Deploy the CIM-Server, which is a real IM server that supports horizontal scaling to meet performance requirements. You only need to register with the same Zookeeper.

Deploy the CIM-forward-route, which is the routing server through which all messages pass. Since it is stateless, Nginx proxies can also be used to improve availability.

Cim -client is a user-oriented client. After startup, it will automatically connect to the IM server and you can send and receive messages on the console.

The detailed design

Next, focus on the specific implementation, such as group chat, private chat message flow; IM server load balancing; How the service registers discovery and so on.

IM the service side

Let’s start with the server side; It mainly realizes the functions of client login and logout and message delivery.

The first is service startup:

Because it is built in SpringBoot, the Netty service needs to be started when the application is started.

In pipline, you can see that Protobuf codec is used (the specific message is analyzed on the client).

Registration found

To meet the horizontal expansion requirements of the IM server, the CIM – Server advertises its data to the registry.

Therefore, you need to register your own data with Zookeeper after the application is successfully started.

The main purpose is to register the current application IP + CIM -server-port+ HTTP -port.

Above are two instances of CIM-Server (on the same server, but with different ports) that I registered in the demo environment.

In this way, the client (listening on the Zookeeper node) can know the available service information in real time.

The login

After the client requests the login interface of cim-forward-route (see below) to complete service verification (similar to daily login to other websites), the client initiates a long connection to the server, as shown in the previous flow:

In this case, the client sends a special packet indicating that the current information is login information.

The server needs to save the userID of the client and the relationship between the current Channel.

It also caches user information, namely userID and user name.

offline

If the client is disconnected, you also need to clear the cached information.

You also need to call the Route interface to clear the information (see below for details).

IM routing

As can be seen from the architecture diagram, the routing layer is very important; It provides a set of HTTP services for both client and server.

Currently, the following interfaces are used.

Registered interface

Since every client requires a login to use it, the first step is naturally to register.

Here the design is relatively simple, directly use Redis to store user information; The user information is only ID and userName.

In Redis, KV stores a VK, so that the ID and userName must be unique.

Login interface

Different from the login in CIM – Server, the login is of service nature.

After a successful login, check whether the login is repeated (one user can run only one client).

After a successful login, you need to obtain the service list (CIM – Server) from Zookeeper and select a service based on an algorithm to return it to the client.

After a successful login, you need to save the routing information, that is, the service instance assigned by the current user to Redis.

To achieve only one user login, the set in Redis is used to save the login information. With the userID as the key, repeated logins will fail to write.

Similar to Java’s HashSet, it can only be resaved.

Obtaining an available routing instance is also simple:

Get all service instances from Zookeeper for an internal cache.

Polling to select a server (currently only this algorithm, will be added later).

Of course, before obtaining the service instance in Zookeeper, you need to listen on the node registered with cim- Server.

The specific code is as follows:

It also monitors the routing nodes in Zookeeper after the application starts and updates the internal cache once changes occur.

Guava’s cache is used here, which is based on ConcurrentHashMap, so atomicity of clearing and adding caches is guaranteed.

Group chat interface

This is a real messaging interface, and the effect is that one client sends a message and all the other clients receive it!

The process must be that the client sends a message to the server, which then iterates through all channels in the SessionSocketHolder described above and sends the message.

The server can also be stand-alone, but now is a cluster design. Therefore, all clients are assigned to different CIM-Server instances based on the previous polling algorithm.

This is where the routing layer comes into play.

After receiving the message, the routing interface first iterates the relationship between all clients and service instances.

Routing relationships are stored in Redis as follows:

Due to the single-threaded nature of Redis, when the data volume is large; Once keys are used to match all CIM-route :* data, Redis cannot process other requests.

Instead, use the scan command to traverse all cim- routes :*.

Each client’s HTTP interface is then called to push the message.

The implementation of cim- Server is as follows:

After receiving the message, the CIM server queries the channel of the userID in the internal cache and sends the message.

Online User Interface

This is an auxiliary interface for querying information about current online users.

Implementation is also very simple, that is, before the query to save the “user login state to reset the set” can be.

Private chat interface

The reason why it is said to acquire online users is a secondary interface, in fact, it is used to assist private chat.

Usually we use private chat before we know which users are currently online, then you know who you want to chat with.

Something like this:

In our scenario, the prerequisite for private chat is to obtain the userID of the online user.

Therefore, after receiving a message, the private chat interface needs to query information about the CIM-Server instance where the receiver resides. The subsequent steps are the same as those in a group chat. Invokes the HTTP interface of the recipient’s instance to deliver the message.

Just group chat is the difference between traversing all online users and private chat only sending one.

Offline interface

Once the client is offline, we need to remove some of the information previously stored in Redis (routing information, login status).

IM client

Some of the logic in the client side was actually covered earlier.

The login

The first step is login. You need to invoke the login interface of route during startup to obtain cim- Server information and then create a connection.

During the login process, route interface will determine whether it is repeated login, and repeat login will directly exit the program.

The next step is to create a connection using the CIM – Server instance information (IP +port) returned by the Route interface.

The last step is to send a login flag message to the server to keep the relationship between the client and the Channel.

Custom protocol

Some login packets and real message packets mentioned above can be distinguished by our user-defined protocols.

Since you are using Google Protocol Buffer codec, take a look at the original format.

In fact, there are only three fields in this protocol:

RequestId can be understood as userId.

ReqMsg is the real message.

Type is the message type mentioned above.

At present, there are three types, corresponding to different services:

The heartbeat

In order to maintain the connection between the client and the server, the heartbeat is automatically sent every time there is no message sent.

The current policy is to send a heartbeat packet to the server every minute:

The server will receive a ping heartbeat packet every minute if it does not receive a service message:

The built-in command

The client also has some basic commands built into it for ease of use.

For example, typing :q will exit the client and close some system resources.

When :olu(short for onlineUser) is typed, route’s fetch all onlineUser interfaces are called.

Group chat

Group chat is as simple as typing a message into the console and pressing enter.

The route’s group chat interface is called.

The private chat

Private chat is the same, but the premise is to trigger the keyword; Use the userId;; The message content format is used to send a message to a user, so the :olu command is usually used first so that online users can use it easily.

The message callback

To meet some custom requirements, such as messages need to be saved and so on.

So after the client receives the message, it calls back to an interface in which you can customize the implementation.

Therefore, a caller bean is created first. The bean contains a CustomMsgHandleListener interface. You only need to implement this interface.

Custom interface

Since I’m not good at writing interfaces myself, I’m sure no one else will. So group chats, private chats, online user acquisition, message callbacks, and so on (and thereafter) are all provided in the form of interfaces on the client side.

Also convenient to do page integration behind, just need to tune these interfaces on the line; The implementation doesn’t really matter.

conclusion

Cim is only the first version at present, with many bugs and few functions (only a few group friends have been tested); But the follow-up will continue to improve, at least this version will give those who do not have the relevant experience of friends to bring some ideas.