I’ve been learning OpenGLES recently, and I’ve also learned about textures, so I thought I’d do a little project to solidify my knowledge. And a panorama browser just include vertex coordinates, texture coordinates, index drawing, MVP matrix transformation and so on knowledge, is a very good training project. I’m going to write it in Swift, which of course is a little bit more cumbersome, especially when it comes to handling Pointers. I wrote two implementations at the same time, one is based on iOS encapsulation GLKit, one is to use GLSL to write, the specific implementation please see the source code -HSGLPanoViewer.

Panoramic browser implementation ideas

The idea is pretty simple, first we need to calculate the vertex coordinates of a sphere (including texture coordinates and index arrays), then paste the panorama onto the sphere as a texture, and the rest is MVP matrix magic.

Vertex coordinates of the ball, texture coordinates, index array

For a sphere, we can peel it into a rectangle by cylindrical projection, and then cut it into small rectangles, and the vertices of these rectangles are the coordinates of the sphere.

OpenGLES can only be drawn by triangles. A rectangle can be divided diagonally into two triangles, and for easy drawing, you need an index array to indicate which points form a triangle.

For this calculation we can refer to the “OpenGL ES 3.0 Programming Guide” for example code about spherical coordinates – esshapes.c, here is my Swift implementation code:

private var vertices = [GLfloat] ()// Vertex coordinates, including texture coordinates
private var indices = [GLushort] ()// Index coordinates

private func generateSphereVertices(slice: Int.radius: Float) {
        let parallelsNum = slice / 2
        let verticesNum = (parallelsNum + 1) * (slice + 1)
        let indicesNum = parallelsNum * slice * 6
        let angleStep = (2 * Float.pi) / Float(slice)

        // The vertex coordinates and texture coordinates are multiplied by 5 to represent the x,y,z components of the vertex coordinates and the U, V components of the texture coordinates
        var vertexArray: [GLfloat] = Array(repeating: 0, count: verticesNum * 5)
        // Vertex coordinate index array
        var vertexIndexArray: [Int] = Array(repeating: 0, count: indicesNum)
        
        X = r * sin α * sin β y = r * cos α z = r * sin α * cos β */
        for i in 0..<(parallelsNum + 1) {
            for j in 0..<(slice + 1) {
                let vertexIndex = (i * (slice + 1) + j) * 5
                vertexArray[vertexIndex + 0] = (radius * sinf(angleStep * Float(i)) * sinf(angleStep * Float(j)))
                vertexArray[vertexIndex + 1] = (radius * cosf(angleStep * Float(i)))
                vertexArray[vertexIndex + 2] = (radius * sinf(angleStep * Float(i)) * cosf(angleStep * Float(j)))
                
                vertexArray[vertexIndex + 3] = Float(j) / Float(slice)
                vertexArray[vertexIndex + 4] = Float(1.0) - (Float(i) / Float(parallelsNum))
            }
        }
        
        var vertexIndexTemp = 0
        for i in 0..<parallelsNum {
            for j in 0..<slice {
                vertexIndexArray[0 + vertexIndexTemp] = i * (slice + 1) + j
                vertexIndexArray[1 + vertexIndexTemp] = (i + 1) * (slice + 1) + j
                vertexIndexArray[2 + vertexIndexTemp] = (i + 1) * (slice + 1) + (j + 1)
                
                vertexIndexArray[3 + vertexIndexTemp] = i * (slice + 1) + j
                vertexIndexArray[4 + vertexIndexTemp] = (i + 1) * (slice + 1) + (j + 1)
                vertexIndexArray[5 + vertexIndexTemp] = i * (slice + 1) + (j + 1)
                
                vertexIndexTemp + = 6}}self.vertices = vertexArray
        self.indices = vertexIndexArray.map { GLushort($0)}}Copy the code

MVP matrix

MVP matrix is:

  • Model Matrix
  • View Matrix
  • Projection Matrix

Through these three matrices, you can achieve a panoramic sphere, a 360-degree view and an asteroid view. So the first thing we need to know is that OpenGLES is a right-handed coordinate system, so we’re facing the phone screen, X is on the right, Y is on the top, origin is in the center of the phone screen, and Z is pointing from the center of the screen toward us.

  1. Panoramic ball

    The panoramic sphere is the simplest, we just need to place the sphere at the origin of the coordinate system, and then our perspective (i.e. camera position), placed on the Z axis, and need to be larger than the radius of the sphere, we can see the complete sphere.

  2. 360 degrees to browse

    To achieve a 360-degree circle view, the sphere is still placed at the origin, and our point of view is placed inside the sphere, so that we can see the panorama around 360 degrees. So our camera position on the Z axis is less than the radius of the ball.

  3. An asteroid

    The asteroid effect needs to be matched with the projection matrix, of course, the above two methods also need to be matched with the projection matrix, but in the asteroid effect, the projection FOV is larger than the first two methods, which gives the feeling of being close to the sphere, so our camera Angle needs to be placed on the radius of the ball. You can imagine that we make a small hole in the sphere and our eyes look into it.

Specific implementation can be viewed in the source code viewTransform.swift

Share experience and trample pits

Removed the GLKit API deprecation warning

‘GLKViewController’ was deprecated in iOS 12.0: OpenGLES API deprecated. (Define GLES_SILENCE_DEPRECATION to silence these warnings)

The GLKit API has been deprecated since iOS12.

To avoid Xcode’s full screen yellow warning ⚠️, we found Preprocessor Macros in Project–Build Settings, Then set GLES_SILENCE_DEPRECATION=1 to remove opengles-related deprecation API warnings. This makes Xcode’s editing interface much cleaner.

Note the index array type

The glDrawElements method is called when OpenGLES is finally drawn:

glDrawElements(GLenum(GL_TRIANGLES), GLsizei(vertexIndices.count), GLenum(GL_UNSIGNED_SHORT), nil)
Copy the code

The third parameter to this method tells OpenGLES what type of index array it is. For example, GL_UNSIGNED_SHORT is specified above, so our index array must be defined as [GLushort], that is, UInt16, We can say GLushort is defined as public TypeAlias GLushort = UInt16)

I made a silly mistake here: I clicked GL_UNSIGNED_SHORT to see if it was defined as public var GL_UNSIGNED_SHORT: Int32 {get} : Int32; Int32 {get} : Int32;

The size of the array in memory

There are some methods in OpenGLES that need to pass in the amount of memory that a number takes up. Such as:

// Copy the vertex array to the vertex cache in the GPU
glBufferData(GLenum(GL_ARRAY_BUFFER), vertices.size(), vertices, GLenum(GL_STATIC_DRAW))
Copy the code

Here I’ve added an extension to Array that makes it easier to get the actual size of the Array:

extension Array {
    // Get the size of the array based on its type and length (Bytes)
    public func size(a) -> Int {
        return MemoryLayout<Element>.stride * self.count
    }
}

Copy the code

Matrix construction

Building and calculating matrices is a more complex part, so it’s best to leave it to a program.

GLKit provides a number of handy classes and methods, such as

  • GLKMatrix4MakeLookAt: Builds the camera view matrix

  • GLKMatrix4MakePerspective: build projection view matrix

  • GLKMatrix4Multiply: Used to multiply matrices

  • GLKMatrix4RotateX: rotation matrix about the X-axis

    …

Using these methods is convenient in the GLKit implementation, but how to extend the implementation in GLSL? Here is the code to update the MVP matrix:

private func updateMVPMatrix(a) {
        var modelViewMatrix = GLKMatrix4Identity
        modelViewMatrix = GLKMatrix4RotateX(modelViewMatrix, xAxisRotate)
        modelViewMatrix = GLKMatrix4RotateY(modelViewMatrix, yAxisRotate)
        modelViewMatrix = GLKMatrix4Multiply(panoViewType.viewTransform.viewMatrix, modelViewMatrix)
        
        let width = frame.size.width * UIScreen.main.scale
        let height = frame.size.height * UIScreen.main.scale
        let aspect = GLfloat(width / height)
        let projectionMatrix = panoViewType.viewTransform.projectionMatrix(aspect: aspect)
        
        // The final MVP matrix
        var mvpMatrix = GLKMatrix4Multiply(projectionMatrix, modelViewMatrix)
        
        // get the pointer to mvpMatrix
        let components = MemoryLayout.size(ofValue: mvpMatrix.m) / MemoryLayout.size(ofValue: mvpMatrix.m.0)
        withUnsafePointer(to: &mvpMatrix.m) {
            $0.withMemoryRebound(to: GLfloat.self, capacity: components) {
                glUniformMatrix4fv(glGetUniformLocation(shaderProgram, "mvpMatrix"), 1.GLboolean(GL_FALSE), $0)}}}Copy the code

GlUniformMatrix4fv = glUniformMatrix4fv = glUniformMatrix4fv = glUniformMatrix4fv = glUniformMatrix4fv

The resources

  • OpenGL – Tutorial – matrix
  • How to Create a 360 Video Player
  • OpenGLES3-Book