Panoramic video can be rotated freely when playing. If combined with the gyroscope of the mobile phone, panoramic video can have a better browsing experience on the mobile end. This article mainly introduces how to implement a panorama player based on AVPlayer.

First look at the final result:

In the last article, we learned how to graphically manipulate a video. (If you don’t already know, I suggest you read it first. Portal)

The encoding format of general panoramic video is no different from that of ordinary video, except that each frame records 360 degree image information. Panorama player needs to do things, can be set by the parameters, play the image of the specified area.

So, we need to implement a filter that takes in some Angle related parameters and renders the image of the specified area. We can then apply this filter to the video as we did in the previous article to achieve the panoramic player effect.

1. Construct the sphere

Each frame of the panoramic video is actually a spherical texture. So, our first step is to construct the sphere and then attach the texture to it.

Let’s start with a piece of code:

/// generate sphere data
/// @param slices Slice numbers, the more the smoother
/// @param radius
/// @param vertices array
// @param indices
// @param verticesCount Specifies the length of the vertex array
// @param indicesCount Index array length
- (void)genSphereWithSlices:(int)slices
                     radius:(float)radius
                   vertices:(float **)vertices
                    indices:(uint16_t **)indices
              verticesCount:(int *)verticesCount
               indicesCount:(int *)indicesCount {
    / / (1)
    int numParallels = slices / 2;
    int numVertices = (numParallels + 1) * (slices + 1);
    int numIndices = numParallels * slices * 6;
    float angleStep = (2.0f * M_PI) / ((float) slices);
    
    / / (2)
    if(vertices ! =NULL) {
        *vertices = malloc(sizeof(float) * 5 * numVertices);
    }
    
    if(indices ! =NULL) {
        *indices = malloc(sizeof(uint16_t) * numIndices);
    }
    
    / / (3)
    for (int i = 0; i < numParallels + 1; i++) {
        for (int j = 0; j < slices + 1; j++) {
            int vertex = (i * (slices + 1) + j) * 5;
            
            if (vertices) {
                (*vertices)[vertex + 0] = radius * sinf(angleStep * (float)i) * sinf(angleStep * (float)j);
                (*vertices)[vertex + 1] = radius * cosf(angleStep * (float)i);
                (*vertices)[vertex + 2] = radius * sinf(angleStep * (float)i) * cosf(angleStep * (float)j);
                (*vertices)[vertex + 3] = (float)j / (float)slices;
                (*vertices)[vertex + 4] = 1.0f - ((float)i / (float)numParallels); }}}/ / (4)
    if(indices ! =NULL) {
        uint16_t *indexBuf = (*indices);
        for (int i = 0; i < numParallels ; i++) {
            for (int j = 0; j < slices; j++) {
                *indexBuf++ = i * (slices + 1) + j;
                *indexBuf++ = (i + 1) * (slices + 1) + j;
                *indexBuf++ = (i + 1) * (slices + 1) + (j + 1);
                
                *indexBuf++ = i * (slices + 1) + j;
                *indexBuf++ = (i + 1) * (slices + 1) + (j + 1);
                *indexBuf++ = i * (slices + 1) + (j + 1); }}}/ / (5)
    if (verticesCount) {
        *verticesCount = numVertices * 5;
    }
    if(indicesCount) { *indicesCount = numIndices; }}Copy the code

This code reference since bestswifter/BSPanoramaView the library. It generates an array of vertices and indexes by dividing the number and the radius of the ball.

Now let’s explain the code line by line:

(1) This part of the code is to segment the original image. There are examples of slices = 10:

In the figure, slices represent the number of slices, which are cut horizontally into 10. NumParallels represents the number of layers, divided vertically into five portions. Since the texture needs to cover 360 degrees horizontally and only 180 degrees vertically when it is attached to a sphere, the number of vertical splits is half of the number of horizontal splits. You can think of them as latitude and longitude to help you understand.

The numVertices denote the number of vertices, such as the blue dots in the graph. NumIndices indicates the number of indexes. When using EBO to draw rectangles, a rectangle needs 6 indexes, so multiply the number of rectangles by 6.

AngleStep represents the Angle increment corresponding to each slice after the texture is attached to the sphere.

(2) Apply the memory space of vertex array and index array according to the number of vertices and indexes.

(3) Start creating vertex data. Here each vertex is iterated over, and the vertex coordinates and texture coordinates for each vertex are calculated.

For convenience, Angle AOB is denoted as α, Angle COD as β, and radius as r.

When I and j are both 0, that’s the G point on the graph. In fact, all 11 points in the first row are going to coincide with the G point.

For point A in the figure, its coordinates are:

X = r sine alpha sine beta y = r cosine alpha z = r sine alpha cosine betaCopy the code

It is easy to get the formula of vertex coordinate.

Texture coordinates only need to grow proportionally according to the segmentation number. It is worth noting that since the origin of the texture coordinate is in the lower left corner, the y value of the texture coordinate should be reversed, that is, the texture coordinate corresponding to G point is (0, 1).

(4) Calculate the value of each index. It’s pretty straightforward, if you look at the first rectangle, you take the first two vertices of the first row and the first two vertices of the second row, and then you split those four vertices into two triangles.

(5) Return the length of the generated vertex array and index array, which is needed in the actual rendering. Because we have 5 variables per vertex, we have to multiply by 5.

Plotting the data generated above, you can see that the sphere has been generated:

Second, perspective projection

OpenGL ES uses orthographic projection by default. Orthographic projection features the same size of near and far images.

In this example, we need to use perspective projection. A perspective projection defines the frustum of the visible space, in which objects are rendered near, large, and small.

As shown in figure, we need to use GLKMatrix4MakePerspective (float fovyRadians, float aspect, float nearZ, float farZ) to construct the perspective projection transformation matrix.

FovyRadians indicates the field of vision. The larger the fovyRadians, the larger the field of vision. Aspect indicates the scale of the window, nearZ indicates the near plane, and farZ indicates the far plane.

In practice, nearZ is usually set to 0.1 and farZ is usually set to 100.

The specific code is as follows:

GLfloat aspect = [self outputSize].width / [self outputSize].height;
CGFloat perspective = MIN(MAX(self.perspective, kMinPerspective), kMaxPerspective);
GLKMatrix4 matrix = GLKMatrix4MakePerspective(GLKMathDegreesToRadians(perspective), aspect, 0.1.100.f);
Copy the code

Because the default coordinates of the camera are (0, 0, 0), and the radius of the sphere is 1, in the range 0.1 to 100. So through the matrix transformation of perspective projection, what you see is the image from inside the sphere, from the flat frustum.

Since it is an image of the interior of a sphere, it is mirrored (this problem will be solved later).

Third, perspective movement

Mobile devices have gyroscopes built into them that provide real-time roll, pitch and YAW information, known as Euler angles.

Anyone who has ever used Euler Angle will encounter a universal joint deadlock problem, which can be solved by quaternion. So instead of reading the euler Angle of the device directly, we use a quaternion, which is converted into a rotation matrix.

Fortunately, the system also provides a quaternion direct access interface:

CMQuaternion quaternion = self.motionManager.deviceMotion.attitude.quaternion;
Copy the code

However, the quaternion obtained cannot be used directly, and three steps of transformation are needed:

Step 1: Invert the Y-axis

matrix = GLKMatrix4Scale(matrix, 1.0f, 1.0f, 1.0f);
Copy the code

Considering the previous X-axis mirroring problem, this step is actually:

matrix = GLKMatrix4Scale(matrix, 1.0f, 1.0f, 1.0f);
Copy the code

Step 2: Invert the vertex shader y component

// Panorama.vsh
gl_Position = matrix * vec4(position.x, -position.y, position.z, 1.0);
Copy the code

Step 3: Invert x component of quaternion

CMQuaternion quaternion = self.motionManager.deviceMotion.attitude.quaternion;
double w = quaternion.w;
double wx = quaternion.x;
double wy = quaternion.y;
double wz = quaternion.z;
self.desQuaternion = GLKQuaternionMake(-wx, wy, wz, w);
Copy the code

Then the correct rotation matrix can be calculated by self.desQuaternion.

GLKMatrix4 rotation = GLKMatrix4MakeWithQuaternion(self.desQuaternion);
matrix = GLKMatrix4Multiply(matrix, rotation);
Copy the code

4. The camera moves smoothly

Self. DesQuaternion constantly changes as we constantly move our phones. Since the speed of a mobile phone changes, self.desQuaternion’s increment is not fixed. The result is a stuttering picture.

So you need to do smoothing, linear interpolation in increments between the current quaternion and the target quaternion. This ensures that the camera movement does not mutate.

float distance = 0.35;   // The smaller the number, the smoother it is and the slower it moves
self.srcQuaternion = GLKQuaternionNormalize(GLKQuaternionSlerp(self.srcQuaternion, self.desQuaternion, distance));
Copy the code

5. Render parameter transfer

In the actual rendering process, external rendering parameters can be adjusted to modify the result of rendering.

Take perspective, for example, and see how specific parameters are passed when the view size is changed.

// MFPanoramaPlayerItem.m
- (void)setPerspective:(CGFloat)perspective {
    _perspective = perspective;
    NSArray *instructions = self.videoComposition.instructions;
    for (MFPanoramaVideoCompositionInstruction *instruction ininstructions) { instruction.perspective = perspective; }}Copy the code

In MFPanoramaPlayerItem, when the perspective changes, will get from the current videoComposition MFPanoramaVideoCompositionInstruction array, traverse the assignment again.

// MFPanoramaVideoCompositionInstruction.m
- (void)setPerspective:(CGFloat)perspective {
    _perspective = perspective;
    self.panoramaFilter.perspective = perspective;
}
Copy the code

In MFPanoramaVideoCompositionInstruction, modify the perspective will give panoramaFilter assignment. Then when MFPanoramaFilter begins rendering, in the startRendering method, a new transformation matrix is generated based on the Perspective property.

Avoid background rendering

Since OpenGL ES does not support background rendering, it is important to note that the APP should be paused before switching to the background.

NSNotificationCenter *center = [NSNotificationCenter defaultCenter];
[center addObserver:self
           selector:@selector(willResignActive:)
               name:UIApplicationWillResignActiveNotification
             object:nil];
Copy the code

- (void)willResignActive:(NSNotification *)notification {
    if (self.state == MFPanoramaPlayerStatePlaying) {
        [selfpause]; }}Copy the code

The source code

Check out the full code on GitHub.

reference

Making 】【 bestswifter/BSPanoramaView
OpenGL ES Learning combat (360 panoramic video player)

Use OpenGL ES to implement panorama Player

mo4tech.com (Moment For Technology) is a global community with thousands techies from across the global hang out!Passionate technologists, be it gadget freaks, tech enthusiasts, coders, technopreneurs, or CIOs, you would find them all here.

Use OpenGL ES to achieve panorama player

1. Construct the sphere

Second, perspective projection

Third, perspective movement

4. The camera moves smoothly

5. Render parameter transfer

Avoid background rendering

The source code

reference

Use OpenGL ES to achieve panorama player

1. Construct the sphere

Second, perspective projection

Third, perspective movement

4. The camera moves smoothly

5. Render parameter transfer

Avoid background rendering

The source code

reference

Related Posts

[Flutter 2-9] Flutter’s Hands-On tutorial on UI Layout and Widgets — Flexible Layout control

IOS event delivery and handling

IOS development – 1 installation components | snapkit install and use