Surprised! Sichuan pepper programmer has a new 3D girlfriend!

The background,

With the expansion of the live broadcasting business of Pepper and the improvement of the demand of anchors and viewers for multi-dimensional dynamic perception, the traditional 2D plane rendering has gradually been unable to meet the demand, and the 3D technology highlighting three-dimensional sense, interaction, randomness and individuality has been paid more and more attention. Multi-dimensional interactive experience of gifts can be realized through 3D rendering technology, and virtual live broadcasting provides a larger space for interest and social interaction, so we developed a virtual image system based on 3D rendering. This paper aims to describe the 3D rendering technology of Prickly ash, which is divided into the following eight aspects:

Multi-level rendering layer thousands of changes, layered rendering

Virtual visualization wanduan, replacement pendant

High shoes pick cloud fiber clothes quite, bone plastic

Frost dyed layer forest flowers contend, dyeing easy capacity

Near and far trail misty, depth jolting

Moon transparent lake mirror gauze wear, transparent hair quality

Plum shadow fairy neon dance, shadow effect

Red sun and stars! Physical light

The first four aspects (layered rendering, changing pendant, bone shaping, dyeing and easy capacity) mainly introduce the realization of the functional components of the system, and the last four aspects (deep jerking, transparent hair quality, shadow effect, physical lighting) highlight the improvement of the overall quality effect of 3D rendering.

Two, technical introduction

1. Multi-level rendering layer thousands of changes, layered rendering

Livestreaming is a complex system, and layered rendering is particularly important for rendering. As you can see below, there are 3D buffs on the lower level of the studio and avatar gifts on the upper level. Different layers simultaneously use 3D rendering, and the life cycle of each effect is different, which requires rendering to support multi-instance, multi-window, which is essential for layered rendering.

Taking Android as an example, SurfaceView and TextureView can be set on the UI layer. Each View can contain multiple Windows, i.e. rendering targets, and each window can also have multiple rendering scene layers. Layer refers to specific scene rendering instances such as 3D particles and virtual images, as shown in the figure below:

The UI View layer contains the final rendering purpose for the top layer, and multiple Windows in the layer can be set to different screen sizes. The advantages of layered rendering are as follows:

• Each layer is independent, suitable for expansion, and the life cycle of each layer is controllable, and the upper layer business is randomly combined and developed;

• Multi-layer rendering can be interspersed with the corresponding upper View, such as bullet screen can be displayed between two 3D rendering layers;

• Encoding upload, video recording, screenshot, etc. can be arbitrarily selected by the upper level;

2. Virtual visualization wanduan, replace the pendant

Dressing is an important part of the virtual image, and the implementation of dressing and attaching is slightly different. Dressing refers to replacing the vertices of the mesh on the basis of the original bone animation. For example, the ancient Qipao is MESh0, while the cowboy is mesH1, mesH0 and MESH1 share the same bone animation. Vertices change with the animation of the skeleton, which is the basic idea of the costume change.

Hang in contrast, the pendant is equivalent to the original object of child, the child object can also contain skeletal animation, the virtual image can have animation, for example, the widget wings can also have a skeletal animation, the child object as the parent of sports is sports, general weapons, wings for a pendant, also is the image of the object, the rendering system for extensibility, Support sub-objects can also skin, coloring, bone kneading and other operations similar to the virtual image, as shown in the figure below, among which coloring and bone kneading are introduced in the following chapters.

Costume changes were made between image 1 and image 2, and bone kneading operation was added to the face shape. Image 3 was the attachment of pendant wings.

As shown in the figure:

3. High shoes pick cloud fiber clothes quite, bone plastic

A leopard’s head and eyes, a swallow’s beard,

Face like heavy jujube, lips like grease,

Red phoenix eyes, crouching silkworm eyebrows, —————————- typical figures typical appearance

The appearance of the virtual image can be adjusted with the user’s preferences. The external characteristics of the image can be modified according to the user’s preferences, such as skin color, eyebrows, blush, beard, face, nose, mouth, etc., and thousands of faces can be completed through osteoplastic surgery. With skin, eyes, cheek is red, whiskers, such as texture map can be used to complete replacement, and face shape modification requires the use of skeletal animation migration to complete, the following main tell about through bone adjustment complete high heels replacement technology principle, high heels and flats change includes cross-dressing, pinched bone two combinations, including change in accordance with part 2, When changing from flat shoes to high heels, the height and foot bones of the standard posture will both change. It only needs to modify the transformation matrix of the corresponding bones. Generally, the position of the bones will be shifted so that the vertices will change accordingly.

• When flat shoes are changed into high heels, bone increments are used to change the height and foot bones of the standard posture during operation, and corresponding bone changes are achieved through incremental changes.

• When the high heel is replaced by the flat shoe, the height and bone of the foot are restored, and the apex of the flat shoe matches the standard posture.

The chest changes are exactly the same as the implementation of high-heeled shoes, including two combined operations of dressing and bone kneading. As shown in the picture: the first two images are the changes of flat shoes and high heels, and the last two images are the changes of the chest bones.

4 frost dyed forest flowers contend, dyeing easy capacity

One of the important components of the virtual image is dyeing, such as skin color, eyeballs, clothes, pendants, etc., can be dyed for dynamic change.

Here we need to mention an HSV color space, which is somewhat different from the well-known RGB space. HSV contains hue, saturation and shade of color. Hue refers to the mixing degree of RGB three colors, saturation refers to the degree close to pure color, and shade refers to brightness and shade. As shown in the figure, RGB spatial model is on the left and HSV spatial model is on the right.

HSV space is the most suitable space for human eyes, and the dyeing needs to be carried out in HSV space, and the saturation and shade remain unchanged. Only adjust the hue value to complete the transformation, and the transformation is RGB->HSV->H adjustment ->RGB to complete the dyeing. The color value corresponding to the hue ring is shown in the figure, and the dyeing purpose is achieved by adjusting the hue ring value.

Through the above methods, the image of the hair dyeing effect as shown below:

5. The trail of near and far is misty and jolted in depth

Z-depth is an important concept in 3D rendering. When rendering opaque mesh vertices, the actual depth of each vertex will be recorded in a Z-buffer. When rasterization is performed pixel by pixel in the window space, the corresponding interpolated Z-depth will be compared. Only the smaller color substitution of Z is retained, so as to ensure that every surface of rendering is independent of sequence and correct. Among them, Z depth involves the camera perspective. Z-fighting is a common 3D rendering problem, which will lead to serious mode wearing problems with the movement of objects or changes of camera. Z-fighting is essentially a computational accuracy problem, which cannot be eradicated technically and can only reduce the degree of defects to a certain extent. As shown in the figure, the part in red is the z-fighting problem.

The solutions are as follows:

• Try not to draw objects too close to each other. For example, if the vertices of jeans and legs are too close to each other, problems may arise. Therefore, try to design a mesh that deviates slightly.

• In the figure above, if the near-plane value is set to 1, defects will appear, but if it is set to 100, defects will disappear. The premise is that too many triangular planes of clip need to be considered, which needs to be weighed.

• Make sure that the camera is not too far away from the object. If the distance is too high, this problem may occur.

• Improving the accuracy of z-buffer requires a trade-off with computational efficiency.

• 6. Moon transparent lake mirror wearing gauze, transparent hair

  The rendering of avatars’ hair is similar to the rendering of leaf meadows in that it involves rendering transparent objects, with some of the vertices being transparent so that the hair is more beautiful, whereas if the design is all opaque and the hair is completely flaky, the rendering will be less effective. In Chapter 5, we mentioned z-depth detection, but if it involves the rendering of transparent objects, if we turn on Z-detection, due to the sequence is not relevant, there will be hollow error in rendering, as shown in the following picture: hollow and sparse in the lush grass on the left, hollow error in the hair on the right.

  

  Rendering transparent objects has always been a relatively complex problem, and the general solutions are:

  • Painter algorithm, all triangles in strict order, from far to near in turn rendering; Disadvantages, triangular surface if the intersection exists then invalid, vertex near and far calculation will appear problems.

  • Deep stripping requires sorting the depth of the transparent layer. For z-depth layer, multiple passes are required to complete it. The disadvantage is that the calculation is too large and it is not suitable for real-time rendering on the mobile terminal.

  • Alpha to Coverage, transparency test: set the threshold value and when the transparency reaches a certain threshold, the transparent surface will not be drawn, and the hollowing error will be removed. Combined with MSAA, anti-aliasing processing will be carried out. The shortcoming is that the burr phenomenon is serious.

  In the comprehensive calculation of computational complexity, we adopted two passes. The first pass first rendered opaque vertices, and the second passed rendered transparent vertices. Transparent vertices were color superimposed without depth writing. The figure on the right shows the effect of the current algorithm.

6. Moon transparent lake mirror wearing gauze, transparent hair

The rendering of avatars’ hair is similar to the rendering of leaf meadows in that it involves rendering transparent objects, with some of the vertices being transparent so that the hair is more beautiful, whereas if the design is all opaque and the hair is completely flaky, the rendering will be less effective. In Chapter 5, we mentioned z-depth detection, but if it involves the rendering of transparent objects, if we turn on Z-detection, due to the sequence is not relevant, there will be hollow error in rendering, as shown in the following picture: hollow and sparse in the lush grass on the left, hollow error in the hair on the right.

Rendering transparent objects has always been a relatively complex problem, and the general solutions are:

• Painter algorithm, all triangles in strict order, from far to near in turn rendering; Disadvantages, triangular surface if the intersection exists then invalid, vertex near and far calculation will appear problems.

• Deep stripping requires sorting the depth of the transparent layer. For z-depth layer, multiple passes are required to complete it. The disadvantage is that the calculation is too large and it is not suitable for real-time rendering on the mobile terminal.

• Alpha to Coverage, transparency test: set the threshold value and when the transparency reaches a certain threshold, the transparent surface will not be drawn, and the hollowing error will be removed. Combined with MSAA, anti-aliasing processing will be carried out. The shortcoming is that the burr phenomenon is serious.

In the comprehensive calculation of computational complexity, we adopted two passes. The first pass first rendered opaque vertices, and the second passed rendered transparent vertices. Transparent vertices were color superimposed without depth writing. The figure on the right shows the effect of the current algorithm.

7. Plum shadow fairy neon dance, shadow effect

Shadow is an indispensable part in the 3 d rendering, reflected the relative position between the objects, the lack of shadow posted objects will feel the lack of stereo feeling, no shadow estimates that by one thousand years ago will also feel lonely, “a toast to invite the bright moon, the shadow into three people, on both confused drink, shadow ACTS with me”, the ancients of the shadow also have experience greatly. Currently, the mainstream shadow rendering is mainly implemented in two ways:

• Hard shadow, stencil volume method. After trying, there are problems in self-shadow, especially in the transition zone. If the number of triangles is limited in accuracy, the shadow of the transition zone will appear serious jagged flicker with animation, serious flaw feeling, and large amount of calculation, so we adopt the soft shadow scheme.

• Soft shadow is dominated by shadow map, which reflects the relationship between light and objects. When light is blocked by objects, there will naturally be dark shadow, and the part that is not blocked will be lit up. Based on this idea, shadow map is completed by 2Pass, and its depth map is obtained from the perspective of light in 1Pass. 2 when using the camera Angle to render pass, in which each vertex again into the light Angle of the corresponding z depth, and with the corresponding z depth before 1 pass depth map, if had larger shadow, otherwise the light is light, soft shadows exist problem is due to the limitation of the depth map resolution, as well as the camera, the light Angle and the distance of the object, Lead to over sampling, sampling, also causes the shadow edges of sawtooth shape, used the PCF filter, using neighborhood comparison superposition of multiple point filter according to proportion, soften edges, to achieve the effect of diffusion shadow, the final effect, as shown in the figure above accessories, clothing ribbon, and the dark parts of the leg on the hem of the robe.

8. Red sun, starry sky! Physical light

Illumination is undoubtedly the most important element in 3D rendering. The early illumination model was mainly PhongShading and Blinn-phong, but the effect was quite different from the current physically based rendering PBR(Physical based Rendering). The biggest difference between PBR and the previous illumination model is that it takes into account the energy conservation of illumination, and based on the roughness value of the micro-surface, the calculation of light reflection is more consistent with the statistical results, so that the rendering effect is closer to the real physical rendering. At present, PBR technology is used as lighting model to improve the rendering quality to the greatest extent.

PBR illumination is divided into direct illumination and indirect illumination. Including direct illumination is the direction of point light source, light source, spotlights, etc., while the indirect illumination using a high dynamic diagram HDR, global environment illumination, high dynamic figure HDR is for general LDR figure, LDR problem is when the superposition of multiple source brightness easily more than 8 bit of precision, more than part of the uniform saturated to 1, HDR uses a higher accuracy than 8bit, such as float. When multiple light sources are stacked, they are not affected by saturation to 1, thus improving the details of high brightness and shadow areas, which is more suitable for global ambient light simulation. Currently, both kinds of lighting are implemented in current rendering systems.

The influence of light on an object can be divided into two types: diffuse reflection and specular reflection. After light irradiates on an object, it will be refracted and reflected. When it is refracted inside the object, atomic collisions will occur again and then scattered to the outside world, which is manifested as diffuse reflection effect. And specular reflection is a reflection of the surrounding environment, such as smooth metal, arrived at metal light reflection to the internal basic all converted to internal energy, instead of scattering, the basic ignore its diffuse, and specular highlights part is more apparent, can reflect the actual illumination environment around, which highlights part mainly has the following three function calculation formula:

• The Normal Distribution Function describes the probability of the Normal Distribution of the micro surface, that is, the probability of the Normal Distribution toward the vertex of the material. Due to different roughness, the micro surface normals will have different orientations.

• The Fresnel Equation describes how much light is reflected from the surface of a river when viewed from a high Angle (near the top). For example, when viewed from a low Angle (near the ground), the surface of a river is mainly refracted, so that you can see fish of rocks underwater. When viewed from a low Angle, the surface is mainly reflected.

• Geometry Function: Describes the properties of microplanes that are self-shadowed, and the possibility of mutual occlusion due to roughness. This Function describes this phenomenon.

The core of diffuse and specular reflection is the final color display by the superposition of all reflected light through the spherical integral of the point-by-point reflection effect on all surrounding light sources. Direct light source in the superposition of multiple light sources, and the indirect illumination part, relatively complex, HDR around can be considered to countless of point light source, the integral calculation is too large, real-time, can be obtained by way of baking offline here involves the half-and-half surface integral, at this time some of the key points to make use of importance sampling integral calculation, to get a better result, The diffuse reflection part can be simulated by spherical harmonic function, and the high-light reflection part can be simulated by importance sampling and inverse sampling, etc. Then, these baking results can be used in real-time rendering. Due to the influence of roughness of micro surface, different baking textures can be generated by considering different roughness particles using MiPMAP. In actual rendering, mIP level is selected according to roughness to improve rendering quality.

PBR requires several textures to improve the specific effect. The first is the inherent color map, which is the basic diffuse texture. If it is a metal texture, it is considered as its reflection, but it cannot have the influence of lighting and shadow effect, so that it can produce different rendering effects in different light source environment. The second is the roughness map, which reflects the flatness of the micro surface, which is one of the most important parameters of PBR. The third is the metallicity map, metal and non-metal in diffuse reflection contribution is greatly different, in addition, metal material for specular reflection in Fresnel calculation has different reflection; The fourth is normal mapping, which can create a detailed effect with a high model through lighting. The fifth is the environment mask map, which can set the dark Angle of static objects; The sixth is self-illumination, which provides a unique light source. For example, the gold titanium alloy of the Iron Man suit can emit more intense light to highlight the effect. Below is the texture of the avatar’s clothes displayed in sequence, from left to right, inherent color, roughness, metallicity, normals, ambient mask, and self-illumination.

The final PBR and shadow renderings are as follows, in which the background is HDR environment, the pure metal in the accessories can reflect the ambient light, the normal map of the dress folds the skirt, the dark corner at the bottom is the environment mask map, the pattern on the ribbon is the self-lighting effect, and the sleeves are matte and metal texture respectively.

Three, afterword.

The realization of rendering technology and special effects suitable for mobile terminal requires long-term accumulation attempts, involving some basic knowledge of graphics, GPU and mathematical physics, such as:

White dress wins the snow wind dance, long hair splash-ink with snow fly – physical engine, skirt, long hair with the wind and move, smooth and natural.

Residual smoke why to cause catkins, angry sword flying noise wild sand – particle effects, smoke, catkins, flying flowers, sand, randomness, group, periodicity.

The basic knowledge of 3D rendering technology needs to be accumulated, and it takes time to build the rendering system.

The road is long,

Walk as you sing,

Unconsciously looked up,

The sky is full of glow,

Red pepper all over!