Web panorama was often used for street view and 360° panoramic image viewing in the past with limited bandwidth. It’s self-immersive, with simple operations that allow users to view objects around them. With the rollout of stream-free services such as King card by some operators and the popularity of 4G environment, the application of heavy traffic is also gradually promoted. For example, can we change the static low-flow panorama picture into dynamic live panorama video? Under a certain network speed bandwidth, it can be achieved. Later, we will learn how to implement panoramic video on the Web side. Let’s take a look at an example GIF:

tl; dr;

  • Use three.js to implement panoramic technology
  • Introduction to the principle of UV mapping
  • 3D coordinate principle and motion control
  • Introduction to Web gyroscope
  • Iv-panorama simple library introduction

Based on Three. Js

Panoramic video is based on 3D space, and in the Web, the technology that can easily touch 3D space is WebGL. For simplicity, we’ll just use the three.js library. The specific working principle is to map the playing video elements into the space of texture, which is directly attached to a sphere through UV mapping. The simplified code is:

letcamera = new THREE.PerspectiveCamera(75, window.innerWidth / window.innerHeight, 1, 1100); // Add camera. Target = new THREE.Vector3(0, 0, 0); // Set the camera's observation position, usually in the center of the sphere scene = new three.scene ();letgeometry = new THREE.SphereBufferGeometry(400, 60, 60); Geometry. Scale (-1, 1, 1); Var texture = new THREE. VideoElement; var material = new THREE.MeshBasicMaterial({ map: texture }); mesh = new THREE.Mesh(geometry, material); scene.add(mesh); renderer = new THREE.WebGLRenderer(); renderer.setPixelRatio(window.devicePixelRatio); Renderer. setSize(window.innerWidth, window.innerheight); container.appendChild(renderer.domElement);Copy the code

This is pretty much the code above. There are two points in the above code that need to be paid attention to, one is the value of the camera’s field of view, and the other is the related parameter Settings of the geometric sphere.

Camera field of view

The specific code is:

let camera = new THREE.PerspectiveCamera(75, window.innerWidth / window.innerHeight, 1, 1100); 
Copy the code

Here, the perspective camera is mainly used to simulate the effect of human eyes. To set the proper view effect, the range also depends on the diameter of the sphere, usually 2* RADIUS + 100, as long as it is larger than the diameter of the sphere.

Geometric sphere parameter Settings

letgeometry = new THREE.SphereBufferGeometry(400, 60, 60); Geometry. Scale (-1, 1, 1);Copy the code

There are actually two parts that need to be explained

  • There are three important attribute values in sphere parameter setting. The API format is:SphereBufferGeometry(radius, widthSegments, heightSegments,...).
    • Raidus: Sets the radius of the sphere, the larger the radius, the larger the content of the video drawn on the canvas will be enlarged, as long as this setting is appropriate.
    • Width /height Segments: the number of slices is mainly used to control how many triangular slices the sphere is subdivided into at most in the two dimensions of width and height. The higher the texture is, the clearer the edges and angles are. However, there is no unlimited high, and there is a simultaneous performance loss.
  • In geometric drawing, pixel points of X axis face inward through coordinate transformation, so that the user does not appear to have the effect of protruding and enlarging. The specific code is:geometry.scale(-1, 1, 1).

The UV mapping

The above is a brief introduction to the code, if only for application purposes, then this is enough. However, if later encountered optimization problems, do not know the lower level or more details of the content, it will feel very awkward. In the panorama video, there are two very important points:

  • The UV mapping
  • 3 d mobile

Here, we will explore the details of UV mapping. The main purpose of UV mapping is to map 2D images to 3d objects. The most classical explanation is:

The box is a three-dimensional object, much like a curved network (“mesh”) square added to the scene. Spread the box out on a table if you cut the box along the edges or creases. When we look down at the table from above, we can think of U as left and right and V as up and down. The picture on the box is in a two-dimensional coordinate. We use U and V for “textured coordinates” instead of X and Y, which are usually used in three-dimensional space. When the box is reassembled, the specific UV coordinates on the cardboard are mapped to a spatial (X, Y, Z) position in the box. This is what a computer does when it wraps a 2D image around a 3D object.

Zhejiang research report

Here, let’s take a closer look at the code. We need to complete a texture, paste the following Sprite onto a cube.

from iefreer

Here, we first load the image into the texture space:

var material = new THREE.MeshPhongMaterial( { map: THREE.ImageUtils.loadTexture('images/texture-atlas.jpg')});Copy the code

So, we now have an area of texture space as follows:

This content actually involves the knowledge of WebGL. Texture space and physical space are not the same. GLSL syntax in WebGL is to map texture content to the surface of the specified triangular region through relevant rules.

It is important to note that there is no so-called minimum triangle region in texture space, but only the triangle region divided in physical space. For simplicity, we set boxGeometry to use only Segments of 1, reducing the number of triangles that need to be divided.

Thus, there are 12 triangular areas that need to be pasted. Here, we need to use Vector2 to manually divide the region of the texture space. In fact, when mapping, it is to map the points of the physical space and the points of the texture space one by one in order, so as to achieve the steps of connecting the texture and the physical space together.

Because, when dividing the physical space, the order of the defined face decomposition triangle is divided according to the counterclockwise direction, as shown in the figure below:

According to the definition in the figure above, we can get the coordinate value of each geometric object’s face mapped to texture space, which can be divided into:

Left-bottom = [0,1,3] right-top = [1,2,3]Copy the code

So, we need to define texture coordinates:

face1_left = [new THREE.Vector2(0, 0),new THREE.Vector2(.5, 0),new THREE.Vector2(0, .333)] face1_right = [new THREE.Vector2(.5, 0),new THREE.Vector2(.5, .333),new THREE.Vector2(0, .333)] //... We're left with 10 noodlesCopy the code

The specific format of fixed-point UV mapping API is:

geometry.faceVertexUvs[ 0 ][ faceIndex ][ vertexIndex ]
Copy the code

Then, the mapping of the specific plane is defined as:

geometry.faceVertexUvs[0][0] = face1_left; geometry.faceVertexUvs[0][0] = face1_right; / /... We're left with 10 noodlesCopy the code

If you’ve written native WebGL code, it should be easy to understand how UV mapping works.

3D Movement Principle

It’s important to note here that Web Panorama is not WebVR. Panorama does not have the immersive experience of VR, involving only three dimensions of rotation without moving distance.

In the description above, the concepts of three dimensions and rotation Angle are mentioned, which easily reminds us of a coordinate system learned in High School Mathematics — spherical coordinate system (the default is right-handed coordinate system here).

  • φ is the Angle between the z axis and the positive direction <=180°
  • ∂ is the Angle between the positive x direction <=180°
  • P is the straight-line distance from the origin

The calculation formula is:

Now, if we apply Web panorama, we know a few things:

  • P: Specifies the radius of the defined sphere
  • ∆φ : the distance that the user moves on the y axis
  • ∂ : the distance the user moves on the x axis

P is constant, whereas ∆φ and ∆∂ are values that are determined by user input. Here, you need an algorithm to unify the protocol. The main content of the algorithm control is:

The user’s finger ∆x/∆ Y on the X/Y plane axis is converted into ∆φ/∆∂ by a certain proportion

If the gyroscope is considered, it is:

The user’s finger ∆ X /∆ Y on the X/Y plane axis is converted into ∆φ/∆∂ by a certain proportion. The user’s rotation Angle value ∆φ’/∆∂’ on the X/Y axis is combined with the Angle of view respectively to calculate the result.

For wider compatibility, the second algorithm is described here. The above change of ∆φ/∆∂ mainly reflects the change of our field of view.

In Threejs, that’s how you control the camera’s field of view. So how do we control the field of view in ThreeJS? Here is the simplest code:

phi = THREE.Math.degToRad(90 - lat);
theta = THREE.Math.degToRad(-lon);
camera.position.x = distance * Math.sin(phi) * Math.cos(theta);
camera.position.y = distance * Math.cos(phi);
camera.position.z = distance * Math.sin(phi) * Math.sin(theta);
Copy the code

Here the earth coordinates are mainly simulated:

  • Lat stands for latitude: the value that the user slides up and down to change, or the phone rotates up and down
  • Lon stands for Lontitude: the value that users swipe left or right to change, or rotate their phone left or right

The specific contents are as follows:

In general practice, there are two ways to change the dimensions of a panoramic view, one directly by hand sliding, and the other by gyroscopic rotation.

Simply put, you listen for touch and orientation events and manually change the value of LAT/LON based on the trigger information. There’s a caveat, though:

Latitude can only be achieved at most (-90,90), otherwise the screen will flip, which is a very bad experience.

Let’s do this in code.

Add touch control

Touch related events on the Web, you can actually tell until you crash, for example, how many fingers does the user Touch the screen with? What is the user’s specific gesture on the screen (swipe, Zoom)?

Here, for the sake of simplicity, only the distance of a finger slide is used as the data of camera Angle movement. The specific code is:

swipe(e=>{
	lat += y * touchYSens;
	lon += x * touchXSens;
	
	lat = Math.max(-88, Math.min(88, lat));
})
Copy the code
  • TouchYSens /touchXSens is used to control the sensitivity, which can be self-debugging, such as 0.5.
  • X/Y: the distance of a single finger movement
  • Math.max(-88, Math.min(88, lat)): controls the movement range value of Latitude

Add gyroscope control

The Web information of gyroscope is mainly obtained through deviceorientation events. It provides the relevant gyroscopic parameters, alpha, beta, gamma. If you don’t know how it works, you basically won’t use it just by looking at its parameters. For details, see the Orientation Gyroscope API.

In simple terms, the parameters of the gyroscope in the standard case, the mobile phone has two coordinates:

  • Earth coordinates X /y/z: in any case, constant direction
  • Mobile phone plane coordinates X/Y/Z: corresponds to the orientation defined on the mobile phone screen

Take the mobile phone itself as the coordinate point, and the earth coordinate is shown in the figure:

  • X: indicates east-west orientation. X is pointing east
  • Y: indicates the north-south direction. Y is pointing north
  • Z: Perpendicular to the center of the earth, z pointing upwards

The phone reference point is the phone plane, and there are also three coordinate systems X/Y/Z.

  • X: Parallel to the screen to the right
  • Y: Parallel to the screen up
  • Z: Forward means perpendicular to the phone screen

And then, as the phone itself rotates or moves, you take the transformation and you get alpha, beta, gamma.

For the rest, refer directly to the gyroscope API. Here, let’s look directly at how to change the camera Angle with the gyroscope:

lat -= beta * betaSens;
lon += gamma * gammaSens;

lat = Math.max(-88, Math.min(88, lat));
Copy the code

Beta is the phone turning up and down, lon is the phone turning left and right. Each time the change in orientation is returned, the specific latitude and Lontitude change results are applied.

For 3D live broadcasting, there are many points to be said, such as panorama click, panorama switch and so on. If you want to manually build a panorama component, this is not necessary. Here, the Now IVWeb team provides an IV-Panorama component, which provides many convenient features, such as touch control, gyroscope control, picture panorama, video panorama and so on.

Iv – panorama

Iv-panorama is a player developed by the IVWEB team for the hot spot of Panorama live. At present, the Web is not particularly friendly to VR support, but for panoramic video, under the premise of machine upgrading, the performance bottleneck of panoramic video gradually disappears. Its main characteristics are:

  • Depends on three.js and needs to be pre-mounted to the Window object
  • Flexible configuration, built-in support for gyroscope and touch control.
  • Support dynamic adjustment of sensitivity parameters
  • Use ES6 syntax
  • React compatible with jQuery

The project is located at iv-Panorama. The project is very simple to use, there are two panoramic modes, one is a picture, one is a video:

import VRPlayer from 'iv-panorama';

new VRPlayer({
        player: {
            url: '/test/003.mp4'
        },
        container:document.getElementById('container')}); // imagelet panorama = new VRPlayer({
    image: {
            url: './banner.png'
        },
        container:document.getElementById('container')});Copy the code

Panorama resources have been put in the Github repository, interested in practice can observe.