Welcome to cloud + community, get more Tencent mass technology practice dry goods oh ~
Author: Ou Yanxing
Introduction to the
Recently, there is a demand related to video chat. I have seen that Tencent Cloud has a video live broadcast product before, but I am not sure whether it also supports video chat. After some searching, I found that in addition to the live demo, there is also a special video chat demo, so I came down to study.
This is the document: www.qcloud.com/document/pr…
This is a live video demo project: github.com/zhaoyang21c…
This is the video chat demo project: github.com/zhaoyang21c…
This article is just a video chat demo. Download it and configure it to get the project running. After registering two accounts, you can call each other. Note here, you need to turn on camera permissions, otherwise it will be a black screen. The interface will look like this, there will be two render Windows, one for myself, one for each other, and some buttons on the right, you can operate beauty, switch camera, etc. (there is only my own window).
Masks and masks
Once the project is up and running, it’s time to do some customization. Requirements require masking and blurring (don’t ask why, there is). The mask is relatively simple, just put an ImageView layer on it. Blur, if you want to do Gaussian blur, you have to process the video data. First try a simple black layer to see how it looks. To facilitate testing, two operation buttons are added on the right side. Add the corresponding item to the layout.
Mask effect:
Mask effect :(50% black mask)
Gray scale, rotation, blur
From the effect, the mask layer is really bad, it seems that we can only start from the video data. Thumbed through the document again, and finally find a way to customize the video (www.qcloud.com/document/pr… There are two ways to manipulate video data to achieve the desired effect.
1. Intercept the camera data of SDK, process it, and then send it back to SDK.
2. Collect camera data, process it, and upload it to SDK.
Try to use the first way, you need to call setLocalVideoPreProcessCallback data correction to intercept cameras, pay attention to the video data format for I420 knowledge (video data format can refer to this article the graphic explanation YUV420 data format). It must be called after the camera is initialized, otherwise it will fail.
For video data, the modification of graying and rotation is first tried.
Gray:
Rotation :(note that width and height should also be reversed)
Fuzzy:
Real Gaussian blur performance requirements are high, it is difficult to achieve real-time requirements, need to use Shader to achieve, here I want to experience the effect of blur, so choose the simplest way to achieve. In order to obfuscate, it needs to be converted to RGB format first and then converted to YUV data after processing. The process is like this: YUV -> RGB -> blur ->YUV, where blur uses approximate algorithm instead (find fastBlur method online). For the method of data conversion, you can refer to this article “Image RGB2YUV and YUV2RGB format interconversion introduction”, test, Tencent cloud data is YUV rather than YCbCr, so the use of 2, 3 conversion.
(Fuzzy radius: 10)
(Blur radius: 30)
Fuzzy processing takes 150 to 200 ms
The follow-up work
This paper uses the Java layer directly for processing, data processing performance efficiency is very poor (150-200ms), the interface will be very stuck (frame drop). If it can really be used in the actual business, the second method must be used for data processing, which is to collect data by ourselves, use hardware GPU for data processing, and then submit SDK for rendering and transmission, so as to achieve the balance between performance and effect. Of course, the difficulty of development quantity and quality assurance of this method will increase accordingly.
Please check the attachment for detailed code.
reference
Tencent cloud document: www.qcloud.com/document/pr…
YUV data format: www.cnblogs.com/azraelly/ar…
YUV and RGB conversion: www.cnblogs.com/qiqibaby/p/…
reading
Introduction to the unity engine and c# scripting
How do you tune hyperparameters in Keras?
Network problems from Traceroute
This article has been authorized by the author yunjia community published, reproduced please indicate the source of the article