preface

Some time ago, I paid attention to a female front-end engineer of Tencent, who knew the use of WASM in front-end video from her official account, and I happened to be uploading the project. As for the preview, there was a problem. The format of frame capture was too few, only video format of MP4, OGG and WebM was supported. So into the pit began, the whole process is more difficult, the first is completely do not understand this piece, do not know how to start, the second is because this piece of article is very few, can ask few people, during the time consulted with the big men, but big men do not bird me, have no way, can only their liver for a month

About WebAssembly in front end applications

WebAssembly can bring to the front is part of the performance enhancement, the front-end of the business is more and more complex, the code will be more and more, then change to the requirement of memory, in some slower memory computer start a front-end project to spend even more than ten seconds, so the front end performance issues in the future will face a new challenge, We BAssembly can be understood by compiling c/ C ++ library functions to call the front-end, so as to solve the front-end browser processing pressure problems, such as some large 3D game pages, such as the image compression, frame capture, video transcoding, etc

To prepare

If you want to start learning wASM, you need to have the following pre-knowledge:

• Basic Linux Syntax
• Emsdk configuration
• The grammar of the wasm

Of course, I use the Windows environment here, so it is the emscripten compilation environment and configuration built by docker, so the basic syntax of docker also need to understand some ^_^

1. Install the compilation environment

Our ultimate goal is to compile a C/C ++ code into a WASM version to provide js calls, so the installation of the compilation environment, if other systems directly refer to the Emscripten official website installation, I have been looking at the Installation of Windows, has not been successful,(if there is a big man in Windows installation success Install an Emscripten environment with Docker:

Download desktop Docker and open terminal:

1. Pull the mirror

docker pull emscripten/emsdk
Copy the code

2. Run a container to start the Emscripten environment: let’s say I have two containers open:

Verify whether the current environment is an Emscripten environment:

emcc -vLook at the current version number:

Since I’m using vscode, installing the docker plugin in vscode makes it more intuitive to see your containers and images as well as the currently running containers and shared folders

And it is easier to open in vscode with terminal operation:

You can look directly at the file that we compiled using the Linux directive right here

2. Compile the wASM file and asm.js file

After the compiler environment is successfully installed, we will first find a simple HelloWorld case on the official website to compile js and wASM to run.

The simplest way to create a c++ program directly using a script:

# create helloworld.cpp
cat << EOF > helloworld.cpp
#include <iostream>
int main() {
  std::cout << "Hello World!" << std::endl;
  return 0;
}
EOF
Copy the code

Then run

emcc hellworld.cpp -o helloworld.js 
Copy the code

As shown: successfully compiled a c++ into an asm.js file and a wasm file

Run helloworld.js using Node

3. Use the ASm. js file in the vue project

The generated asm.js file can be used in a vue project. For example, let’s rewrite a c++ function

extern "C" {int aa(int x){ if(x<=0) return 0; if(x<=2) return 1; return x-2; }}Copy the code

Compiling statements in Emscripten:

emcc   -s EXPORTED_FUNCTIONS="['_aa']"   -s EXPORTED_RUNTIME_METHODS=["cwrap"]  a.cpp -o a.js -s  WASM=0
Copy the code

In the VUE project:

Look at the downcalls in the browser

Note:

1. Extern C is required if you want to call C ++ code with JS code on the front end otherwise an error will be reported and the function will not be found

2. WASM=0 must be used to compile, otherwise the function will not be correctly used before initialization and error will be reported

This is a simple example of how to use asM.js. The compiled js file (86KB) is larger than the wASM file (1KB), but the readability of the js file is better and there is no browser compatibility problem. But the execution speed of binary WASM may be about 8 times that of JS, which I did not go to learn in detail. The specific C/C ++ and J are mutual call methods refer to Ruan Yinfeng’s ASM. js and Emscripten introductory tutorial

4. Use wASM files in vUE projects

We used the EMCC compiled JS file in the previous section. Now we do the same with the wASM file. To use wASM in a vue project, we need to read and compile this file through an arrayBuffer WebAssembly.Module, then instantiate WebAssembly, using the exported methods, detailed usage and case methods in the # WebAssembly technical documentation

Change the test demo directly in the vue file:

Note :importObject is a required field. It is a function that is exported by WASM.

5. Use video capture frame ffmpeg.wasm

Just use some simple c method, to compile a complete ffmpeg library is more complex, because there are a lot of c/c + + code, is a header file reference, not simply a single function, about video cutter frame have bosses completed a fit can call js file directly, his article also have detailed configuration, you can Pull the code directly from Github to learn, the big guy has written all the compilation in the script, including webpack configuration direct one-click NPM run build to generate a usable JS file! The beginning of learning is from the big guy’s code, in addition to understand the basic knowledge of the code preparation or to learn first, otherwise introduced also do not understand the process, later can not change the code, and then their own package belongs to their own project suitable for the truncated frame file.

# Front-end video frame extraction FFMPEG + Webassembly

Wasm and js files are compiled directly from ffmpeg header files and library files. The syntax is as follows:

export FFMPEG_PATH=/include
export TOTAL_MEMORY=33554432
emcc  -Iinclude libavformat.a libavcodec.a libswscale.a libavutil.a capture.c -O3  -s WASM=1 -s TOTAL_MEMORY=${TOTAL_MEMORY} -s EXPORTED_FUNCTIONS='["_main", "_free", "_capture"]'  -s ASSERTIONS=1  -s ALLOW_MEMORY_GROWTH=1  -o capture.js

Copy the code

We have compiled the wASM file, and the usage is the same as the previous section. I will not repeat it here. (After all, there are already a complete set of big guys, thanks again for your contribution ~~~~~).

conclusion

Everything is in order to understand and use the ffmpeg. Wasm to realize the video cutter frame start operation, from environment to build to compile operation success step by step I feel are pit, paper will sleep shallow ah, according to data and technical document does not necessarily come out, I spent a lot of early twilight time, no one can ask baidu did not answer, very crazy… But think about the front has someone really made a complete JS script could not help but envy and admire, as a front-end in addition to constantly learning how to do, but it is finally come to an end, after I can also play a simple change of C/C ++ package to match the use of the project, vegetables chicken peck rice, do not like spray!