1, the introduction
A brief introduction to what Jingmai is:
Jingmai workbench is jingdong business city for jingdong business prepared a background management tool, it can make you do not log in the business background can be order production, quickly realize the order download delivery process. Something like taobao’s Wangwang Merchant version (now called Taobao Qianniu).
2. The author of this article
Zhang Songran:
3. Network structure of the TCP gateway
4. TCP gateway long-connection container architecture
5. IO model of TCP gateway Netty Server
The specific implementation process is as follows:
- 1) Create ServerBootstrap and set BossGroup and WorkerGroup thread pools;
- 2) Bind the specified port to start listening and accepting client links (if the system has only one port to listen on, set the number of BossGroup threads to 1).
- 3) register childHandler on ChannelPipeline to handle request frames in client links.
6. Thread model of TCP gateway
NioEventLoop is a Netty Reactor thread that plays the following roles:
- 1) Boss Group: as a server Acceptor thread, it accepts client links and forwards them to threads in the WorkerGroup;
- 2) Worker Group: as an I/O thread, I/O reads and writes packets from SocketChannel or writes packets to SocketChannel;
- 3) Task Queue/Delay Task Queu: performs scheduled tasks, such as detecting link idle and sending heartbeat messages.
7. Sequence diagram of TCP gateway execution
Step one:
Step 2:
Step 3:
Step four:
Step 5:
Step 6:
Step 7:
Step 8:
Step 9:
Step 10:
Step 11:
Step 12:
Step 13:
8, TCP gateway source code analysis
8.1 the Session management
Create Session;
8.2 the heart
8.3 Data Uplink
8.4 Data Downlink
To get the return value using blocking:
Questions about BlockingOperationException in StackOverflow question, I was very lucky to get the Norman Maurer (core contributor of Netty) answer:
9. Summary of this paper
Click here to enter
Appendix: More detailed information summary
[1] Network programming basics:
TCP/IP, rounding
Chapter 11 ·UDP: User datagram protocol
TCP/IP, rounding
Chapter 17: TCP: Transmission control Protocol
TCP/IP, rounding
Chapter 18: Establishment and termination of TCP connections
TCP/IP, rounding
Chapter 21. TCP Timeouts and retransmission
Once upon a time in technology: TCP/IP changed the world
Easy to Understand – In-depth understanding of TCP (part 1) : Theoretical basis
Easy to Understand – In-depth understanding of TCP (part 2) : RTT, sliding window, and congestion handling
Theory classic: TCP protocol three handshake and four wave process in detail
Connection between theory and practice: Wireshark captures packets to analyze TCP three-way handshake and four-way wave
Protocol Diagram of Computer Network Communication (Chinese Edition)
What is the maximum size of a packet in UDP?
P2P technology details (a) : NAT details – detailed principle, P2P introduction
(2) : P2P NAT traversal (hole) solution
P2P technology details (three) : P2P technology STUN, TURN, ICE details
Easy to Understand: Quickly understand the PRINCIPLE of NAT penetration in P2P technology
High performance network programming (a) : the number of concurrent TCP connections can be a single server
High performance Network programming (II) : The last 10 years, the famous C10K concurrent connection problem
High performance Network Programming (III) : In the next 10 years, it is time to consider C10M concurrency
High performance Network programming (IV) : Theoretical exploration of high performance network applications from C10K to C10M
Unknown Network Programming (I) : A Brief Analysis of the DIFFICULT Problems in TCP Protocol (Part I)
Unknown Network Programming (II) : A Brief Analysis of the DIFFICULT Problems in TCP Protocol (Part II)
Why TIME_WAIT and CLOSE_WAIT when closing TCP connections
Unknown network programming (iv) : In-depth analysis of TCP abnormal shutdown
Network programming unknown (5) : UDP connectivity and load balancing
Unknown network programming (6) : Understand UDP in depth and use it well
Quick Understanding of Network Communication Protocols (Part 1)
A Quick Understanding of Network Protocols (Part 2)
Network programming lazy introduction (three) : a quick understanding of the TCP protocol is enough
Network programming lazy starter (4) : Quickly understand the difference between TCP and UDP
Netty dry goods sharing: Jingdong Jingmai production grade TCP gateway technology practice summary
More of the same…
[2] NIO asynchronous network programming
The principle of Java new generation network programming model AIO and the introduction of Linux system AIO
11 Questions and answers about “Why choose Netty”
Open Source NIO Framework gossip – MINA or Netty came first?
Netty or Mina: A Deep Study and Comparison
Netty or Mina: An In-depth Study and Comparison (II)
NIO framework introduction (a) : server based on Netty4 UDP bidirectional communication Demo
NIO framework introduction (ii) : Server based on MINA2 UDP bidirectional communication Demo
NIO framework introduction (3) : iOS and MINA2, Netty4 cross-platform UDP two-way communication actual combat
NIO framework introduction (four) : Android and MINA2, Netty4 cross-platform UDP two-way communication actual combat
Netty 4.x learning (a) : ByteBuf detail explanation
Netty 4.x learning (2) : Channel and Pipeline details
Netty 4.x learning (3) : Details of the threading model
Apache Mina Framework Advanced Part 1: IoFilter details
Apache Mina Framework Advanced Part ii: IoHandler details
MINA2 Thread Principle Summary (including simple test example)
Apache MINA2.0 Development Guide
MINA, Netty source code (online reading version) has been collated and released
Solve the problem of TCP sticky packet and missing packet in MINA data transmission (source code)
Resolve the coexistence of multiple instances of Filter of the same type in Mina
Practice summary: Netty3.x upgrade netty4. x encountered those pits (threads)
Practice summary: Netty3.x vs. Netty4.x threading model
Netty security in Detail: Introduction to the principles, code demo (Part 1)
Netty security in Detail: Introduction to the principles, code demo (Part 2)
Explain the elegant exit mechanism and principle of Netty
NIO framework details: Netty’s high-performance approach
Twitter: How to Use Netty 4 to reduce JVM GC overhead
Absolute Dry goods: Technical essentials of Netty based mass access push service
Netty dry goods sharing: Jingdong Jingmai production grade TCP gateway technology practice summary
More of the same…
[3] Selection of IM/ push communication format and protocol:
This section describes the differences between TRANSPORT layer protocols TCP and UDP
Why does QQ use UDP instead of TCP?
Mobile im protocol selection: UDP or TCP?
How to choose the data transmission format of instant messaging application
The strong column suggests using Protobuf as the data transfer format for your instant messaging app
Full evaluation: Is Protobuf 5 times faster than JSON?
Technical Problems in mobile TERMINAL IM development (including communication protocol selection)
Briefly describe the pitfalls of mobile IM development: architectural design, communication protocols, and clients
Theory and Practice: a set of typical IM communication protocol design details
Share the technical practice of protocol design of home real-time message system
How to use Google’s Protobuf in NodeJS
More of the same…
[4] About IM/ push heartbeat alive processing:
A two-process daemon for Android6.0
Android6.0 and above (process prevention)
Android6.0 and above survivability practices (killed and resurrected)
How to keep the Android process alive: One article answers all your questions
Summary of Message push on Android: implementation principle, heartbeat survival, problems encountered, etc
Let’s take a closer look at the small matter of Android notifications
Why does TCP – based mobile IM still need the heartbeat keepalive mechanism?
Wechat team original sharing: Android version of wechat background to keep alive combat sharing (process to keep alive)
Wechat team original sharing: Android version of wechat background to live combat sharing (network to live)
Mobile IM practice: To realize the intelligent heartbeat mechanism of wechat on Android
Mobile IM practice: Analysis of heartbeat strategy of WhatsApp, Line and wechat
More of the same…
[5] About instant messaging development on the WEB:
Beginner’s note: the most complete explanation of instant messaging technology on the Web
Web side im technology inventory: short polling, Comet, Websocket, SSE
SSE technology details: a new HTML5 server push event technology
Comet technology description: Based on HTTP long connection Web end real-time communication technology
Quick start for beginners: WebSocket concise tutorial
WebSocket details (I) : a preliminary understanding of WebSocket technology
WebSocket detail (2) : technical principle, code demonstration and application cases
WebSocket details (3) : In-depth details of WebSocket communication protocol
Socket. IO implementation of message push practice and ideas
LinkedIn instant messaging practice on the Web: hundreds of thousands of long connections in a single machine
The development of Instant messaging technology on Web and the practice of WebSocket and socket. IO technology
Instant Messaging Security on the Web: Cross-site WebSocket hijacking (with sample code)
Open source framework Pomelo practice: Build a high-performance distributed IM chat server on the Web end
WebSocket and SSE technology is used to push Web messages
Detailed explanation of the evolution of Web side communication mode: from Ajax, JSONP to SSE, Websocket
More of the same…
[6] On IM Architecture design:
Brief talk about IM system architecture design
Briefly describe the pitfalls of mobile IM development: architectural design, communication protocols, and clients
Sharing of a set of mobile TERMINAL IM Architecture Design practice of massive Online Users (with detailed pictures and texts)
A set of original distributed instant messaging (IM) system theoretical architecture scheme
From zero to excellence: The evolution of the technical architecture of JINGdong customer service INSTANT messaging system
Architecture selection for mogujie INSTANT messaging /IM server development
Tencent QQ140 million online users technical challenges and architecture evolution road PPT
Design practice of time-sequence-based cold and hot classification architecture for massive data in wechat background
Speech by technical Director of wechat on Architecture: The Way of wechat — Avenue to Simplicity
How to interpret “wechat Technical Director on Architecture: The Way of wechat — The Road to Simplicity”
Fast Fission: Witness the evolution of wechat’s powerful background architecture from 0 to 1 (I)
17 years of practice: Technical methodology of Tencent’s massive products
How to ensure the efficiency and real time of pushing large-scale group messages in MOBILE IM?
Discussion on synchronization and storage scheme of chat messages in modern IM system
More of the same…
[7] IM Security
Instant messaging Security part 1: Understand and use the Android encryption algorithm correctly
Instant messaging security (2) : discuss the application of combined encryption algorithm in IM
Instant messaging security (3) : common encryption and decryption algorithms and communication security
Instant Messaging Security part 4: Example analysis of key hardcoding risks in Android
Instant messaging security (5) : Symmetric encryption technology in the Android platform application practice
Instant Messaging Security part 6: Asymmetric encryption principles and practices
Introduction and Demo of Java platform implementation of transport layer security protocol SSL/TLS
Theory and Practice: A Typical DESIGN of IM Communication Protocol (including security Layer design)
Wechat new generation communication security solution: MMTLS based on TLS1.3
From Ali OpenIM: Technical practice sharing for building secure and reliable INSTANT messaging service
This section describes the working principles of end-to-end encryption (E2EE) in real-time audio and video chat
Mobile terminal security communication weapon — end-to-end encryption (E2EE) technology details
Instant Messaging Security on the Web: Cross-site WebSocket hijacking (with sample code)
Easy to understand: a master of instant messaging message transmission security principles
More of the same…
[8] For real-time audio and video development:
Interview with the head of wechat video technology: Evolution of video chat technology in Micro-reliance
Instant messaging audio and video development (I) : a theoretical overview of video codec
Instant messaging audio and video development (ii) : digital video introduction to video codec
Instant messaging audio and video development (iii) : coding basis of video codec
Instant communication audio and video development (iv) : introduction to video codec predictive technology
Instant messaging audio and video development (5) : Understanding the mainstream video coding technology H.264
Instant messaging audio and Video development (vi) : How to start learning audio codec technology
Instant messaging audio and Video development (7) : Introduction to audio fundamentals and coding principles
Instant messaging audio and Video development (8) : common real-time voice communication coding standards
Instant messaging audio and Video development (ix) : Overview of echo and echo cancellation for real-time voice communications
Instant messaging audio and Video development (10) : real-time voice communication echo cancellation technology details
Instant communication audio and video development (11) : Real-time voice communication packet loss compensation technology details
Instant messaging audio and video development (12) : Multi-party real-time audio and video chat architecture discussion
Instant communication audio and Video development (13) : features and advantages of real-time video coding H.264
Instant communication audio and Video development (xiv) : Introduction of real-time audio and video data transmission protocols
Instant messaging audio and video development (15) : Talk about P2P and real-time audio and video applications
Instant Messaging audio and Video development (16) : Some suggestions for mobile real-time audio and video development
Instant communication audio and video development (17) : video coding H.264, VP8 past life
Brief introduction of audio processing and coding compression techniques in real-time voice chat
Netease Video Cloud technology sharing: Audio processing and compression technology quick start
Learn the basic knowledge of RFC3550: RTP/RTCP real-time transmission protocol
The advantages and disadvantages of open source real-time audio and video technology WebRTC are introduced
Conscience Sharing: WebRTC Zero-Base Developer Tutorial (Chinese)
Application of RTP/RTCP data transmission protocol in WebRTC
Research on Real-time Streaming Media Technology Based on RTMP Data Transfer Protocol
Audio Net Architect on difficulties in Implementing real-time Audio and Video Cloud (Video interview)
Brief discussion on the technical essentials of developing real-time video live broadcasting platform
Still testing real-time voice call quality with Hello hello? This article teaches you the scientific evaluation method!
To realize the practice sharing of 1080P real-time audio and video broadcast with a delay of less than 500 milliseconds
Practice of real-time video broadcast technology on mobile terminal: How to achieve real-time second open, smooth and smooth
How to test your live audio and video solution in the easiest way
Technology revealed: Facebook live video with millions of fans interacting
This section describes the working principles of end-to-end encryption (E2EE) in real-time audio and video chat
Detailed explanation of real-time audio and video broadcast technology on mobile terminal (I) : Introduction
Mobile terminal real-time audio and video broadcast technology details (ii) : acquisition
Mobile terminal real-time audio and video broadcast technology details (three) : processing
Mobile terminal real-time audio and video broadcast technology details (IV) : coding and packaging
Mobile terminal real-time audio and video broadcast technology details (five) : push stream and transmission
Mobile terminal real-time audio and video broadcast technology details (six) : delay optimization
Combine theory with practice: realize a simple real-time video live broadcast based on HTML5
IM real-time audio and video chat echo cancellation technology in detail
Brief introduction to several key technical indicators that directly affect user experience in real-time audio and video live broadcast
How to optimize the transmission mechanism to achieve ultra-low latency of real-time audio and video?
First disclosure: How does Kuaishou achieve millions of viewers watching the live broadcast can still open seconds and no delay?
Real-time communication RTC stack: video codec
Open source real-time audio and video technology WebRTC under Windows concise compilation tutorial
Android live introduction practice: start to build a simple live system
More of the same…
[9] An overview of IM Development
How to ensure the efficiency and real time of pushing large-scale group messages in MOBILE IM?
The technical issues that mobile IM development needs to face
Is it better to design your own protocol using byte streams or character streams to develop IM?
Does anyone know the mainstream implementation of voice message chat?
Realization of IM message delivery guarantee mechanism (I) : to ensure the reliable delivery of online real-time messages
Realization of IM message delivery guarantee mechanism (II) : Ensure the reliable delivery of offline messages
How to ensure the “timing” and “consistency” of IM real-time messages?
A low cost method to ensure IM message timing
Should I use “push” or “pull” to synchronize online status in IM chat and group chat?
IM group chat messages are so complex, how to ensure that not lost and not heavy?
Talk about optimizing login requests in mobile IM development
How to save traffic by pulling data during IM login on the mobile terminal?
A brief discussion on the principle of multi-point login and message roaming of mobile TERMINAL IM
How to design a “retry failure” mechanism for a completely self-developed IM?
Easy to understand: Load balancing solution sharing at the mobile IM access layer based on a cluster
Technical Experiment and Analysis of wechat’s Influence on Network
Principle, Technology and Application of Instant Messaging System (Technical Paper)
The status of open source IM project “Mogujie TeamTalk” : An open source show with no end
QQ Music team share: Android image compression technology in detail (part 1)
QQ Music team share: Android image compression technology details (part 2)
Tencent original share (a) : how to greatly improve the mobile QQ picture transmission speed and success rate under the mobile network
How to greatly reduce APP traffic Consumption in mobile Network (Part 1)
(2) How to greatly reduce the Traffic consumption of APP in mobile Network (Part 2)
As promised: Mars, a cross-platform component library for wechat’s mobile IM network layer, has been officially open source
How does Social network-based Yelp achieve lossless compression of massive user images?
More of the same…
[10] Open source MOBILE IM technology Framework data:
Open source mobile IM technology framework MobileIMSDK: Quick start
Open source mobile IM Technology framework MobileIMSDK: FAQ
Open source mobile IM Technology framework MobileIMSDK: Stress Test Report
More of the same…
[11] Articles on push technology:
Details of iOS push service APNs: design ideas, technical principles and defects
Walk through the APNS pit on iOS10 together
Summary of Message push on Android: implementation principle, heartbeat survival, problems encountered, etc
Literacy stickers: Understand THE MQTT communication protocol
A full Android push Demo based on MQTT communication protocol
Interview with IBM technical manager: formulation process and development status of MQTT protocol, etc
Android message push: GCM, XMPP, MQTT three advantages and disadvantages
Analysis of real-time message push technology on mobile terminal
Talk about the principle and difference of iOS and Android background real-time message push
Absolute Dry goods: Technical essentials of Netty based mass access push service
Mobile IM Practices: A Study of Google’s Push Messaging Service (GCM) (from wechat)
Why do IM tools like wechat and QQ not use GCM service to push messages?
Technical practice sharing of large-scale high concurrency architecture of Aurora push system
From HTTP to MQTT: An overview of APP data communication practices based on location services
Technical practice sharing of meizu’s 25 million long-connected real-time message push architecture
Interview with meizu architects: experience of massive long-connected real-time message push system
Let’s take a closer look at the small matter of Android notifications
Implementation of Message Push Practice of Hybrid Mobile Application Based on WebSocket (with code example)
A secure and extensible subscription/push service based on long connection
Practice sharing: How to build a set of highly available mobile messaging push system?
Practice of Constructing Ten-million-level Online High Concurrent Message Push System with Go Language (from 360 Company)
Tencent carrier pigeon technology sharing: ten billion real time message push experience
Practice of real-time push technology of Millions of online Meipai live barrage system
More of the same…
[12] Instant Messaging Technology
www.52im.net/forum.php?m…