• Scotland team
  • Author: Jason

preface

The author recently received a request to remotely control the synchronization operation of more than N terminals on the LAN (function similar to remote desktop on Windows). Finally, the author realized a scheme that is not mature and stable, but basically meets the current needs, and will continue to iterate in the future.

For a variety of complex reasons, I cannot use some of the most mature solutions on the market today. However, since this project is only used by internal personnel, there are not too many requirements for the maturity, reliability and maintainability of the scheme (facial expression gradually stretches).

Analysis of the

Remote control of desktop, whether 1 to 1 or 1 to many. There are only two types of core data. Audio and video information, control information (keyboard, mouse, shortcut keys, etc.). When we get these two data, we can get the video information and transmission control instructions of the controlled end in real time at the control end. So the next operation is:

1. Obtain the video stream of the controlled end

As a front-end developer, I instinctively think of the WebRTC protocol when it comes to getting audio and video information from devices. Then there is the analysis of whether the specific device and scenario is suitable. After a lot of imagination analysis, I see the line (mainly WebRTC itself is strong).

2. Send and receive control information

Due to specific business and certain arcane restrictions and trade-offs. It is not possible to transmit messages using TCP. That has nothing to say, UDP save me! The good news was that, after a series of complicated maneuvers, I got the client’s interface to simulate mouse and keyboard operation. All right, all right!

What if UDP transmission is unreliable?

In the early stage of development, the biggest problem encountered by the author is that UDP transmission is connectionless and reliability is not guaranteed. However, in this business scenario, multiple controlled ends need to compare the differences after a period of operation to report exceptions. If the instructions received are slightly different due to network problems, the availability will be greatly reduced.

As we all know, UDP transmission is not reliable, it is not like TCP to ensure that data is ordered, not lost transmission. But the limits are there to make your head big. So how do you ensure reliable UDP transmission in this scenario?

How to solve the problem that UDP transmission is unreliable in this scenario

After a series of imagination, my mind produced the following dialogue:

(1) Clueless stage

A: One server controls multiple clients. How do they transfer data to each other? B: UDP multicast? A: Does the service allow packet loss? B: No, and it has to be done in an orderly fashion.Copy the code

Conclusion: In this scenario, the author can only use UDP multicast or broadcast, but does not allow packet loss.

(2) Proposal stage

A: For every message sent by the sender, the receiver accepts A message to acknowledge receipt. B: The server sends one message and receives dozens of acknowledgements? What if you send 30 pieces of data a second? A: After A client sends A packet loss request through multicast, other clients do not send the same packet after receiving it? B: What if multiple clients lose packets at the same time? Random wait time to send? So what about the delay? What if it's out of sync? And do you have to listen for channels while you're waiting?Copy the code

Conclusion: Imitating THE TCP validation mechanism is basically not feasible, you need to take another path.

(3) Preliminary decision on option A

A: If the client cannot reply to the received message, the client must determine the packet loss itself. B: Yes, the client needs to determine the packet loss and report back to the server! A: The only source of information on the client is the packets sent by the server, so the connection between the packets sent by the client is inherent. B: Does the client add the Index of the packet when sending it? Then the client determines whether the packet is lost according to whether the Index of the two packets is consecutive.Copy the code

Summary: When the client sends a packet, incrementing Index is added to determine whether the server loses the packet.

(4) Buffer establishment and maintenance

B: For example, now the server sends packets 1,2,3 and then 4. Client 1 receives all the messages and remains silent. Client 2 loses message 3, and when it receives message 4, it discovers that the previous packet was message 2. So it's going to send a feedback packet to the server about missing message 3, right? A: How do you send it? Peer-to-peer or go multicast? B: Multicast. If other receivers also lose the packet, they can receive it. If the package is not lost, you can also determine Index to know that it is an executed package. The server also needs to maintain a buffer. The data that has been confirmed to be sent must be cleared; otherwise, the memory will burst. A: But there is no way for the server to determine whether the packet has been received. B: If the packet loss mechanism takes effect. When the client feedback packet 9 is lost, it means that the client received all packets before packet 9. Then you can clean up the previous package. You also need to set a maximum value. A: But there are more than one sender! When one of them is received, the server clears the packet. After a while, another client reported that it lost the package again. What should I do? B: And the clients that have received feedback.Copy the code

Summary: The server maintains a buffer of maximum length and purges invalid data upon acknowledgement of receipt.

(5) Waiting and enforcement mechanisms

A: Back to just now, the client received message 2 followed by message 4 and judged that message 3 was lost. What happens to the received message 4 when a retransmission request for message 3 is sent? What if message 5, message 6 and message 7 are received while waiting for message 3? Always waiting? Or one two bags just ignore and lose? B: Absolutely not! Because the client does not know what operation the lost packet is, if it is clicked, it will cause the information between the clients is not synchronized very seriously! So we have to wait. After receiving the packet, the peer checks whether the packet with a larger Index exists. If so, enforce the package.Copy the code

Summary: The client also maintains a buffer, waiting to receive new data while dropping packets. Enforce all received packets upon receipt.

(6) Multiple channels

B: In this case, all clients receive the same message and have the same packet Index. But what if point-to-point communication is needed at this point? A: Does each client maintain A set of buffers? B: No, it's too expensive. Private chat smuggling chat channel, can not ensure reliability.Copy the code

Conclusion: The server needs both multicast and point-to-point communication. Point-to-point communication does not ensure reliability.

(7) package weighted

A: Waiting for packet return after packet loss may cause too much delay and multiple clients may be out of sync. B: Reverse, no need to wait --> ignore lost packet --> lost packet is not important --> you determine that the packet is not important. The server sends packets with the weight of N packets in the past. This way the client will know the weight of the previous package when receiving it! A: Because you judge whether the last packet was lost by the next packet, so if the next packet carries the previous N packets, how important is it? Theoretically it's possible to ignore the lost bag! B: So, you need to assign permissions to packets at the sending end. For example, mouse movement level is 1 and mouse click level is 10.Copy the code

Conclusion: In order to ignore packet loss, it is necessary to determine the weight of each packet, which is to add the weight of the previous N packets to each packet.

(8) Packing and unpacking

A: There is A requirement to store operations that have been done for A while. It can then be sent all at once to the receiver at any time. B: Sending large data at one time will cause packets to be fragmented at the IP layer. Larger packets will be divided into more slices. How big will the business script be? A: It grows directly proportional to the recording time, and the user's operation cannot be guessed, and should not be limited. So we need to manually unpack on the server side and unpack on the client side. B: You need to add the total package length and Index of the package to the split package. A: Split packets are still sent following the above reliable protocol to ensure that all subcontracts can be received.Copy the code

Conclusion: In order to avoid fragmentation at the IP layer, the server needs to manually subcontract and then combine the packets at the client side.

Problems encountered during the implementation process

Fault 1 The mouse on the client is jammed

For the purpose of not sending messages too frequently, data is throttled during mouse movement. The trade-off is that the mouse on the receiving end receives a discontinuous set of position points. Because the server is through WebRTC display a certain client screen to operate, so the user experience is very poor.

Solution:

If the server throttles mouse movement events, the client only needs to do mouse movement interpolation. That is, in the two mouse movement events received, they calculate the median value for interpolation. The measured optimization effect is very obvious.

Fault 2 The client repeatedly sends and loses packets within a short period of time, causing heavy load on the server

During debugging, it is found that the client sends the packet loss information to the server several times after the lost packet is received. The server then returns multiple packets after a period of time (because it received multiple lost packets requesting retransmission).

In order to cause unnecessary performance loss, we throttled the packet loss requests that sent the same packet.

Problem 3 The client is stuck because of continuous packet loss

After some practice, it is found that packet loss is easy to be continuous. For example, the Index sequence of the message received by the client is 1,2,3,4,10. The client considers packet 5 lost and sends the lost message of packet 6 after packet 5 is received. By the time packet 9 is accepted, other clients may have received packets 20,30 or larger. This can lead to long periods of out-of-sync between clients, which is unacceptable for this business scenario.

Solution: The client describes all packet loss once, and the server returns the packet loss once.