What is the NSQ

NSQ is a realtime distributed messaging platform designed to operate at scale, It is based on the Go language and is highly regarded for its light weight. It is a simple and easy-to-use messaging middleware developed by Bitly and distributed under the MIT open source protocolCopy the code

The term is introduced

When an application first publishes a message, it creates a Topic Channels. Topic Channels are related to consumers. They are load balancers among consumers. Nsqlookupd is a daemon that receives, queues, and delivers messages to clients. Nsqlookupd: Nsqlookupd is a daemon that manages topology information. Nsqadmin is a set of WEB UI used to gather real-time statistics of a cluster and perform different administrative tasks nsq_to _file: consume specified topics/channels and write them to files, optionally scroll and/or compress files nsq_to _http: Consume specified topics/channels and perform HTTP requests (GET/POST) to specified endpoints NSQ_to-_NSQ: consumer specified topics/channels and republish messages to destination NSQD via TCPCopy the code

Application scenarios of NSQ

Process asynchronization, code decoupling, traffic peak clipping, high availability, high throughput, and broadcast distribution achieve data consistency and meet the requirements of specific business scenariosCopy the code

What do you get out of this project?

Golang entryCopy the code

The theme

  • NSQ system architecture

    It consists of three components with different roles: NSQ > NSQLOOKUP > NSD Admin

    1. NSQ
    3. NSD Admin

    And just to make sense of it, you can think of it as a plane ride. NSQ is regarded as an airplane, NSQLOOKUP is regarded as the dispatching console of an airplane, NSQ Admin is the app of booking tickets, and we ourselves are the message in the whole process. NSQ will carry us to wherever we want to go, and the flight number is TOPIC.

  • NSQ code structure

    This is also roughly divided into three parts

    1. Entry function, build app

    2. Common abstractions

    3. Service layer of NSQ, NSQlookup and Admin to realize the actual func, which will be focused on later

  • Component analysis of NSQ

  • nsqd

    NSQD is a daemon that receives, queues, and delivers messages to clients. It can run on its own, but it is typically configured by the cluster in which the NSQlookupd instance resides

    NSQD execution logic

    Producer –> message –> topic –> channels –> Consumer

    NSQD object

    TCP service HTTP service diagram as can be seen, there are two core objects, topicMap and Listener topicMap store topic information. TcpListener and httpListener listen for TCP and HTTP services. Other objects are built around this, for example: context with guaranteed context, sync.RWMutex with atomization, poolSize with message pool management, etcCopy the code

    NSQD startup process

    1. It starts TCP server and HTTP Server and listens in daemon mode2. The system continuously sends the registration heartbeat to NSQlookupd to ensure successful registration and survival in NSQdLookup

    Tips: NSQD manages daemon processes with the help of third-party SVC packages. “github.com/judwhite/go-svc” is a service running framework

    Message passing

    How does NSQD deliver messages? What should I pay attention to in the process of distribution?

    Let’s first look at how NSQD handles messages

    The producer, which delivers the message to Topics to A, then distributes the specified message to the specific Channel through which the message will be consumed, thus balancing the load of the consumer

    1. Memory optimization of messages

    Sync. Pool is used to save temporary objects for reuse to avoid storage redistribution and reduce the GC pressure of programs, so as to avoid: large concurrency – large memory footprint – slow GC – Reduced concurrency – large concurrency

    2. Message transfer process Object uses Byte for message transfer to avoid performance degradation caused by frequent format transformation

    3. Message format

    Timestamp (8byte) +msgID (16byte) +body (*byte)

    4. High reliability of messages

    If the value exceeds the threshold, some memory will be converted to disks for persistence

    5. Queue channels through the min heap

    NSQD receives messages

    1, a coroutine listens to TCP server 2, and a coroutine processes the requested client and keeps the long connection. When CONN fails, the coroutine also actively cedes CPU 3, and a coroutine listens to TCP socket and continues to process messagesCopy the code

    [TCP Server Listening]

    [Continuously processing requests from clients]


    Nsqlookupd is the daemon responsible for managing topology information. The client finds the producers of a given topic by querying nsQlookupd, and the NSQD node broadcasts topic and channel information. There are two interfaces: the TCP interface, which NSQD uses to broadcast. HTTP interface that clients use to discover and manage

    The NSQLOOKUP object is simpler with one atomic object and six objects listening for TCP and HTTP services, respectively

    • TCP services:
    • The HTTP service:

  • NSQAdmin

    Nsqadmin is a set of WEB UI, used to gather real-time statistics of the cluster, and perform different administrative tasks because this content is relatively simple, not too much parsingCopy the code

Bright spot

Performance optimization method

- Sync. pool Implements memory overcommitment and fragmentation persistence to avoid GC excessively high due to frequent collection - Byte stream transfer to avoid performance degradation due to frequent serialization - CPU allocation, for example, when conn err, Active SURRENDER CPU - Minimum Heap sort - Concurrent dynamic adjustment - Probabilistic expiration algorithm - Custom datagramCopy the code

Development means

- Design mode: Factory methods, decorators, etc. - Go concurrent processing, dynamic tuning goroutiune - elegant OOP thinkingCopy the code


Although NSQ is not perfect and the overall structure is not complex, and the content of this paper is not comprehensive, please pay attention to the details

The appendix

NSQ’s Git: github.com/nsqio/nsq, as a 20K star messaging middleware