Small knowledge, big challenge! This article is participating in the “Essentials for Programmers” creative activity. This article has participated in the “Digitalstar Project” to win a creative gift package and challenge the creative incentive money.

Recently, the group leader gave me a task to try to run the famous video transcoding library FFMPEG (written in C) in the browser, I was confused at that time, can still play like this? I did some research and found something called WebAssembly that could do just that.

What is WebAssembly?

• A new type of code that runs in a Web browser, provides some new features and focuses primarily on high performance
• Not primarily for writing, but for C/C++, C#, Rust, etc., so you can take advantage of it even if you don’t know how to write WebAssembly code
• Code written in other languages can run at close to native speed, and client apps can run on the Web
• WebAssembly modules can be imported into a browser or Node.js, and JS frameworks can use WebAssembly to gain huge performance advantages and new features while being functionally easy to use

The goal of WebAssembly

1. Fast, efficient, and convenient – cross-platform execution at near-native speed by leveraging some common hardware capabilities
2. Readable, debuggable – WebAssembly is a low-level assembly language, but it also has a human-readable text format that makes it possible to write code, view code, and debug code.
3. Secure – WebAssembly explicitly runs in a secure, sandbox execution environment, similar to other Web code, which enforces same-origin and some permission policies.
4. Don’t break existing Web – WebAssembly is designed to run compatible with other Web technologies and maintain backward compatibility.

How is WebAssembly compatible with the Web?

The Web platform can be viewed as having two parts:

1. A virtual machine (VM) is used to run Web application code, such as a JS engine that runs JS code
2. A series of Web apis that Web applications can call to control the functionality of Web browsers/devices to do certain things (DOM, CSSOM, WebGL, IndexedDB, Web Audio API, etc.)

For a long time, a VM could only load JS to run, and JS might be enough for our needs, but now we have all kinds of performance issues such as 3D games, VR/AR, computer vision, image/video editing, and other areas that require native performance.

At the same time, it is very difficult to download, parse and compile large JS applications, and this performance bottleneck will be more severe on some more resource-constrained platforms, such as mobile devices.

WebAssembly is a different language from JavaScript. It is not designed to replace JS, but is designed to complement and collaborate with JS, enabling Web developers to reuse the best of both languages:

1. JS is a high-level language that is flexible and expressive, dynamically typed, does not require compilation steps, and has a powerful ecology that makes it easy to write Web applications.
2. WebAssembly is a low-level, assembly-like language that uses a compact second-level format, runs at near-native performance, and provides a low-level memory model. It is the target of compilation for languages such as C++ and Rust to enable code written in such languages to run on the Web (note that, WebAssembly will provide high-level goals such as an in-memory model for garbage collection in the future)

With the advent of WebAssembly, the VM mentioned above can now load two types of code execution: JavaScript and WebAssembly.

JavaScript and WebAssembly can interoperate. In fact, a piece of WebAssembly code is called a module, and WebAssembly modules have many features in common with ES2015 modules.

Key concepts of WebAssembly

To understand how WebAssembly works on the Web, you need to understand a few key concepts:

1. Module: WebAssembly binaries compiled into executable machine code by the browser. Module is stateless, like a Blob, and can pass between Windows and workerspostMessageShared, a Module that declares import and export similar to ES2015 modules.
2. Memory: A resizable ArrayBuffer that contains linear arrays of bytes read and written by WebAssembly’s low-level Memory access instructions.
3. Table: A resizable array of typed references (such as functions) that, for security and portability reasons, cannot be stored in memory as raw bytes
4. Instance: A Module that contains all the states it uses at runtime, including Memory, Table, and a set of imported values. An Instance, like an ES2015 Module, is loaded into a specific global variable with a specific set of imports

WebAssembly’s JavaScript API provides developers with the ability to create Modules, Memory, tables, and instances. Given a WebAssembly Instance, JS code can call its exports synchronously — exported as normal JavaScript functions. Any JavaScript function can be called synchronously by WebAssembly code by passing JavaScript functions as imports to WebAssembly Instance.

Because JavaScript has complete control over how WebAssembly code is downloaded, compiled, and run, JavaScript developers can assume that WebAssembly is just a new feature of JavaScript — the ability to efficiently generate high-performance functions.

In the future, WebAssembly modules can be loaded as ES2015 module loads, such as

How to use WebAssembly in an application?

WebAssembly adds two things to the Web platform: a binary format code and a set of apis for loading and executing binaries.

WebAssembly is at an embryonic point, and there are sure to be a lot of tools coming, but there are four main entry points:

• Use EMScripten to port C/C++ applications
• Write and generate WebAssembly code directly at the assembly level
• Write the Rust application and use WebAssembly as its output
• Use AssemblyScript, a typescript-like language that compiles to WebAssembly binaries

Porting C/C++ applications

Although there are some other tools such as:

• WasmFiddle

• WasmFiddle++

• WasmExplorer

However, all of these tools lack EMScripten’s tool chain and optimization operations. The specific operation process of EMScripten is as follows:

1. EMScripten feeds the C/C++ code to the Clang compiler, a C/C++ compiler based on the LLVM compilation architecture, which compiles to LLVM IR
2. EMScripten converts LLVM IR to.wASM binary bytecode
3. A WebAssembly can’t get the DOM directly. It can only call JS, passing in raw data types such as integers or floating-point types. Therefore, a WebAssembly needs to call JS to get the Web API and call. EMScripten does this by creating HTML files and JS glue code

In the future, WebAssemblies can also call Web apis directly.

The JS glue code above is not as simple as you might think. At first, EMScripten implemented some popular C/C++ libraries such as SDL, OpenGL, OpenAL, and some POSIX libraries based on Web apis. So you need JS glue code to help WebAssembly interact with the underlying Web API.

So, part of the glue code implements the functions of the corresponding libraries that C/C++ code needs to use, and the glue code also contains the logic to call the WebAssembly JavaScript API above to get, load, and run.wasm files.

The generated HTML document loads the JS glue code and writes the output to the . If the application uses OpenGL and the HTML contains the

element as the rendering target, you can easily rewrite the EMScripten output. Convert it to the form required by the Web application.

Write WebAssembly code directly

If you want to build your own compiler, toolchain, or JS library that generates WebAssembly code at runtime, you can opt for hand-written WebAssembly code. Like physical assembly, WebAssembly’s binary format has a text representation that you can write or generate manually and convert text to binary using WebAssembly’s text-to-binary tools.

Write Rust code and compile it into WebAssembly

Thanks to the tireless efforts of the Rust WebAssembly Working group, we can now compile Rust code into WebAssembly code.

Check out this link: developer.mozilla.org/en-US/docs/…

Using AssemblyScript

AssemblyScript is the best choice for Web developers who want to try out WebAssembly writing in typescript-like form without learning the details of C or Rust. AssemblyScript compiles TypeScript variants to WebAssembly, allowing Web developers to use typescript-compatible toolchains such as Prettier, VSCode Intellisense, You can check its documentation to see how to use it.

How to compile newly written C/C++ code into WebAssembly?

The EMScripten tool allows you to compile newly written C/C++ code for Use by WebAssembly.

Preparation conditions

In order to be able to use the Emscripten tool, we need to install it. First Clone related code:

git clone https: // github . com / emscripten-core / emsdk . git

cd emsdk
Copy the code

Then execute the following script to configure emSDK:

Git pull /emsdk install latest # Activate the latest SDK tool for the current user. /emsdk activate Latest # Add sdK-related commands to PATH and activate other environment variables source./emsdk_env.shCopy the code

With the above operations, we can use emscripten-related commands at the command line. Generally, there are two scenarios when we use Emscripten:

• Compile to WASM and then create HTML documents to run the code, along with JavaScript glue code to run the WASM code in a Web environment
• Compile to WASM code and create only JavaScript files

Generate HTML and JavaScript

First create a folder in the emSDK directory: WebAssembly, and then create a C code in the folder: hello.c as follows:

#include <stdio.h>



int main() {

    printf("Hello World\n");

}
Copy the code

Then navigate to the hello.c directory from the command line and run the following command to invoke Emscripten for compilation:

emcc hello.c -s WASM=1 -o hello.html
Copy the code

The above commands are explained as follows:

• emccCommand line command for Emscripten
• -s WASM=1Tells Emscripten that it needs to output the WASM file, which will be output by default if this parameter is not specifiedasm.js
• -o hello.htmlTells the compiler to generate a file namedhello.htmlTo run the code, as well as the WASM module and the corresponding JavaScript glue code for compiling and instantiating WASM so that wASM can be used in a Web environment

After running the command above, you should have three more files in your WebAssembly directory:

• Binary WASM module code:hello.wasm
• JavaScript file containing glue code:hello.js, which translates native C functions into JavaScript/ WASM code
• An HTML file:hello.htmlTo load, compile, and instantiate wASM code and display the output of the WASM code on the browser.

Run the code

All that’s left to do now is load hello.html in a WebAssembly-enabled browser.

Supported by default in Firefox 52+, Chrome 57+, and the smallest Opera browsers, You can also allow experimentation by enabling javascript.options.wasm in About :config in Firefox 47+ and Chrome ://flags in Chrome 51+ and Opera 38+ WebAssembly special effects support.

Since modern browsers do not support file:// XHR requests, they cannot load.wasm and other related files in HTML, so additional local server support is required to see the effect, by running the following command:

npx serve .
Copy the code

NPX is a convenient tool for NPM command execution launched by NPM after 5.2.0+, such as the above serve, when running, it first checks whether local exists, if not, it downloads the original corresponding package and executes the corresponding command, and it is a one-time operation, which avoids installation before permission. Operations that require temporary use of local memory.

In WebAssembly folder to run a local Web server, and then open the http://localhost:5000/hello.html to check the effect:

You can see that the code we wrote in C to print Hello World is successfully printed in the browser. You can also open the console and see the corresponding output:

Congratulations to you! You’ve compiled a C module into a WebAssembly and run it in a browser!

Use custom HTML templates

The Emscripten default HTML template is used in the above example, but there are many scenarios where you need a custom HTML template, such as integrating WebAssembly into an existing project. Let’s look at how to use custom HTML templates.

Create a new hello2.c file in the WebAssembly directory and write the following:

#include <stdio.h>



int main() {

    printf("Hello World\n");

}
Copy the code

Clone the shell_minimal. HTML file from the emSDK repository and copy it to the html_template subfolder in the WebAssembly directory (this folder needs to be created). The file structure in the WebAssembly directory now looks like this:

. ├ ─ ─ hello. C ├ ─ ─ hello. HTML ├ ─ ─ hello. Js ├ ─ ─ hello.html wasm ├ ─ ─ hello2. C └ ─ ─ html_template └ ─ ─ shell_minimal. HTMLCopy the code

Navigate to WebAssembly from the command line and run the following command:

emcc -o hello2.html hello2.c -O3 -s WASM=1 --shell-file html_template/shell_minimal.html
Copy the code

As you can see, there are several changes in parameter passing:

• By setting the-o hello2.html, the compiler will outputhello2.jsJS glue code as wellhello2.htmlThe HTML file
• At the same time set--shell-file html_template/shell_minimal.htmlThis command provides the address of the HTML template that you will use to generate the HTML file.

Now let’s run the HTML with the following command:

npx serve .
Copy the code

In the browser to navigate to: localhosthttp: / / localhost: 5000 / hello2. HTML to access the run results, before a similar effect can be observed and:

You can see that the Emscripten header is missing, but everything else is the same as before. If you look at the WebAssembly directory, you can see that similar JS and Wasm code is generated:

. ├ ─ ─ hello. C ├ ─ ─ hello. HTML ├ ─ ─ hello. Js ├ ─ ─ hello.html wasm ├ ─ ─ hello2. C ├ ─ ─ hello2. HTML ├ ─ ─ hello2. Js ├ ─ ─ hello2. Wasm └ ─ ─ Html_template └ ─ ─ shell_minimal. HTMLCopy the code

Note: You can specify that only JavaScript glue code is output, rather than a full HTML document, by specifying it as a.js file after the -o tag, such as emcc -o hello2.js hello2.c -o3 -s WASM=1, You can then customize the HTML file and import the glue code to use, however this is a more advanced method and the usual form is to use the supplied HTML template:

• Emscripten requires a lot of JavaScript glue code to handle memory allocation, memory leaks, and a host of other issues.

Call a custom function in C

If you define a function in C code and want to call it in JavaScript, you can use Emscripten’s ccall function and the EMSCRIPTEN_KEEPALIVE declaration (this declaration adds your C function to the function output list, The specific working process is as follows:

Create a hello3.c file in the WebAssembly directory and add the following:

#include <stdio.h>

#include <emscripten/emscripten.h>



int main() {

    printf("Hello World\n");

}



#ifdef __cplusplus

extern "C" {

#endif



EMSCRIPTEN_KEEPALIVE void myFunction(int argc, char ** argv) {

    printf("MyFunction Called\n");

}



#ifdef __cplusplus

}

#endif
Copy the code

Emscripten-generated code only calls main by default, and the rest of the functions are removed as “dead code.” Adding the EMSCRIPTEN_KEEPALIVE declaration before the function name prevents this “deletion” from happening. You need to import the emscripten.h header file to use the EMSCRIPTEN_KEEPALIVE declaration.

Note that we have added the #ifdef block to our code to make sure that this works when imported into C++ code, because C and C++ naming may have some confusing rules, so the above function to add EMSCRIPTEN_KEEPALIVE declaration may not work. So add external to the function in C++, treat it as an external function, and it will work in C++.

Then, for the sake of demonstration, the HTML file still uses the shell_minimal. HTML file we put in the html_template directory, and then use the following command to compile the C code:

emcc -o hello3.html hello3.c -O3 -s WASM=1 --shell-file html_template/shell_minimal.html -s NO_EXIT_RUNTIME=1  -s "EXTRA_EXPORTED_RUNTIME_METHODS=['ccall']"
Copy the code

Note that in the above compilation we added the NO_EXIT_RUNTIME parameter because the program will exit when main is finished, so we added this parameter to ensure that the other functions will still run as expected.

The extra EXTRA_EXPORTED_RUNTIME_METHODS is used to export CCall methods to WebAssembly modules, allowing exported C functions to be called in JavaScript.

When you run it through NPX serve. you can still see similar results:

Now we can try using the myFunction function in JavaScript by opening the hello3. HTML file in the editor, then adding a element and calling the myFunction function when the element is clicked:

<! <button class="mybutton">Run myFunction</button> <script type='text/javascript'> //... Document.queryselector ('.myButton ').addeventListener ('click', function() { alert('check console'); var result = Module.ccall( 'myFunction', // name of C function null, // return type null, // argument types null // arguments ); }); </script> <! -- Other content -->Copy the code

Save the above content and refresh the browser to see the following result:

When we click the button in the image above, we get the following result:

MyFunction Called (” MyFunction Called “); MyFunction Called (” MyFunction Called “);

The above example shows that you can call functions exported from C code in JavaScript via ccall.

How to compile an existing C module into WebAssembly?

A core usage scenario for WebAssembly is to reuse libraries in the existing C ecosystem and compile them for use on the Web platform without having to re-implement a set of code.

These C libraries often rely on C’s standard library, operating system, file system, or other dependencies, and Emscripten provides most of these dependencies, although there are some limitations.

Let’s compile the WebP encoder of the C library into WASM to see how to compile existing C modules. The source code of WebP Codec is implemented in C, which can be found on Github, and some OF its API documents can be seen.

Clone WebP encoder source code to local, and emSDK, WebAssembly directory equivalent:

git clone https://github.com/webmproject/libwebp
Copy the code

To get started, we can import the “WebPGetEncoderVersion” function from the “encode.h” header file to JavaScript. First create the “webp.c” file in the “WebAssembly” folder and add the following:

#include "emscripten.h"

#include "src/webp/encode.h"



EMSCRIPTEN_KEEPALIVE

int version() {

  return WebPGetEncoderVersion();

}
Copy the code

The above example is a quick way to verify that the source code for libwebp is compiled correctly and that its functions can be used successfully, since the above functions can be executed successfully without complicated arguments and data structures.

To compile the above function, we first have to tell the compiler how to find the libwebp library header file, tell the compiler the address by adding the flag I at compile time, then specify the location of the libwep header file, and pass it all the C files in libwebp that the compiler needs. An effective strategy is to pass all C files to the compiler and rely on the compiler itself to filter out unnecessary files. This can be done by writing the following command on the command line:

emcc -O3 -s WASM=1 -s EXTRA_EXPORTED_RUNTIME_METHODS='["cwrap"]' \

 -I libwebp \

 WebAssembly/webp.c \

 libwebp/src/{dec,dsp,demux,enc,mux,utils}/*.c
Copy the code

Note: Many projects rely on libraries such as Autoconfig/Automake to generate system-specific code before compiling. Emscripten provides emconfigure and Emmake to encapsulate these commands. And inject appropriate parameters to smooth out projects that have pre-dependencies.

This will produce a copy of the a.ut.js glue code and the a.ut.wasm file. Then you need to create an HTML file in the output directory of the A.ut.js file and add the following code to it

<script src="./a.out.js"></script>

<script>

  Module.onRuntimeInitialized = async _ => {

    const api = {

      version: Module.cwrap('version', 'number', []),

    };

    console.log(api.version());

  };

</script>
Copy the code

After the WebAssembly module is initialized, use the cwrap function to export the C function version to use, by running the similar NPX serve. Command, and then open the browser to see the following effect:

Libwebp uses the ABC of 0xabc in hexadecimal to represent the current version A.B.C. For example, v0.6.1 is encoded as 0x000601 in hexadecimal and 1537 in decimal. The value is 66049 in decimal and 0x010201 in hexadecimal, indicating that the current version is V1.2.1.

Get the image in JavaScript and run it in WASM

Having just verified that the libwebp library has been successfully compiled into WASM by calling the encoder’s WebPGetEncoderVersion method to get the version number, and can then be used in JavaScript, we’ll move on to more complicated operations, How to convert image formats using libwebp’s coding API.

Libwebp’s Encoding API needs to accept an array of bytes for RGB, RGBA, BGR, or BGRA, so the first question to answer is, how do you put images into WASM to run? Fortunately, the Canvas API have a CanvasRenderingContext2D getImageData method, can return a Uint8ClampedArray, the array contains RGBA image data format.

First we need to write a function to load the image in JavaScript and write it to the HTML file created in the previous step:

<script src="./a.out.js"></script> <script> Module.onRuntimeInitialized = async _ => { const api = { version: Module.cwrap('version', 'number', []), }; console.log(api.version()); }; Async function loadImage(SRC) {const imgBlob = await fetch(SRC).then(resp => resp.blob()); const img = await createImageBitmap(imgBlob); Const canvas = document.createElement('canvas'); const canvas = document.createElement('canvas'); canvas.width = img.width; canvas.height = img.height; // Draw the image to canvas const CTX = Canvas.getContext ('2d'); ctx.drawImage(img, 0, 0); return ctx.getImageData(0, 0, img.width, img.height); } </script>Copy the code

Now all that remains is how to copy image data from JavaScript to WASM. To do this, we need to expose additional methods in the previous webp.c function:

• A method to allocate memory for images in WASM
• A way to free up memory

Modify webp.c as follows:

EMSCRIPTEN_KEEPALIVE Uint8_t * create_buffer(int width, int height) { return malloc(width * height * 4 * sizeof(uint8_t)); } EMSCRIPTEN_KEEPALIVE void destroy_buffer(uint8_t* p) { free(p); }Copy the code

1 uint8_t (uint8_t); malloc (); When this pointer is returned to JavaScript for use, it is treated as a simple number. When the corresponding C function exposed to JavaScript is retrieved via the cwrap function, you can use this pointer number to find where the memory to copy the image data starts.

We added additional code to the HTML file as follows:

<script src="./a.out.js"></script> <script> Module.onRuntimeInitialized = async _ => { const api = { version: Module.cwrap('version', 'number', []), create_buffer: Module.cwrap('create_buffer', 'number', ['number', 'number']), destroy_buffer: Module.cwrap('destroy_buffer', '', ['number']), encode: Module.cwrap("encode", "", ["number","number","number","number",]), free_result: Module.cwrap("free_result", "", ["number"]), get_result_pointer: Module.cwrap("get_result_pointer", "number", []), get_result_size: Module.cwrap("get_result_size", "number", []), }; const image = await loadImage('./image.jpg'); const p = api.create_buffer(image.width, image.height); Module.HEAP8.set(image.data, p); / /... call encoder ... api.destroy_buffer(p); }; Async function loadImage(SRC) {const imgBlob = await fetch(SRC).then(resp => resp.blob()); const img = await createImageBitmap(imgBlob); Const canvas = document.createElement('canvas'); const canvas = document.createElement('canvas'); canvas.width = img.width; canvas.height = img.height; // Draw the image to canvas const CTX = Canvas.getContext ('2d'); ctx.drawImage(img, 0, 0); return ctx.getImageData(0, 0, img.width, img.height); } </script>Copy the code

As you can see, in addition to importing the create_buffer and Destroy_buffer we added earlier, the code also has many functions for encoding files, which we’ll cover later. In addition, the code first loads an image of image.jpg. We then call C to allocate memory for the image data, get the corresponding pointer to the WebAssembly Module.HEAP8, write the image data at the beginning of memory, and finally free the allocated memory.

Coding image

Now that the image data has been loaded into WASM’s memory, you can call the libwebp encoder method to complete the coding process. Looking at the WebP documentation, you can use the WebPEncodeRGBA function to do the work. This function takes a pointer to the image data and its size, along with an optional quality parameter ranging from 0 to 100. During coding, WebPEncodeRGBA allocates a block of memory for output data, and we need to call WebPFree to free this memory after coding.

We open the webp.c file and add the following code to handle the encoding:

int result[2];

EMSCRIPTEN_KEEPALIVE

void encode(uint8_t* img_in, int width, int height, float quality) {

  uint8_t* img_out;

  size_t size;



  size = WebPEncodeRGBA(img_in, width, height, width * 4, quality, &img_out);



  result[0] = (int)img_out;

  result[1] = size;

}



EMSCRIPTEN_KEEPALIVE

void free_result(uint8_t* result) {

  WebPFree(result);

}



EMSCRIPTEN_KEEPALIVE

int get_result_pointer() {

  return result[0];

}



EMSCRIPTEN_KEEPALIVE

int get_result_size() {

  return result[1];

}
Copy the code

The result of the above WebPEncodeRGBA function is to allocate a block of memory for the output data and the size of the returned memory. Array because C function cannot be used as the return value (unless we need to carry on the dynamic memory allocation), so we use a global static array to obtain the results returned, it may not be very normative writing C code, at the same time it requires wasm pointer to 32 – bit, but for the sake of simplicity we can tolerate this kind of practice for the time being.

Now that the c-side logic has been written, you can call the encoding function on the JavaScript side to get the pointer to the image data and the memory occupied by the image, save this data in JavaScript’s own memory, and then release the memory allocated by WASM when processing the image. Let’s open the HTML file to complete the logic described above:

<script src="./a.out.js"></script> <script> Module.onRuntimeInitialized = async _ => { const api = { version: Module.cwrap('version', 'number', []), create_buffer: Module.cwrap('create_buffer', 'number', ['number', 'number']), destroy_buffer: Module.cwrap('destroy_buffer', '', ['number']), encode: Module.cwrap("encode", "", ["number","number","number","number",]), free_result: Module.cwrap("free_result", "", ["number"]), get_result_pointer: Module.cwrap("get_result_pointer", "number", []), get_result_size: Module.cwrap("get_result_size", "number", []), }; const image = await loadImage('./image.jpg'); const p = api.create_buffer(image.width, image.height); Module.HEAP8.set(image.data, p); api.encode(p, image.width, image.height, 100); const resultPointer = api.get_result_pointer(); const resultSize = api.get_result_size(); const resultView = new Uint8Array(Module.HEAP8.buffer, resultPointer, resultSize); const result = new Uint8Array(resultView); api.free_result(resultPointer); api.destroy_buffer(p); }; Async function loadImage(SRC) {const imgBlob = await fetch(SRC).then(resp => resp.blob()); const img = await createImageBitmap(imgBlob); Const canvas = document.createElement('canvas'); const canvas = document.createElement('canvas'); canvas.width = img.width; canvas.height = img.height; // Draw the image to canvas const CTX = Canvas.getContext ('2d'); ctx.drawImage(img, 0, 0); return ctx.getImageData(0, 0, img.width, img.height); } </script>Copy the code

In the above code we load a local image.jpg image using the loadImage function. You need to prepare an image to use in the emCC compiler’s output directory, which is our HTML file directory.

Note: New Uint8Array(someBuffer) will create a new view on the same memory block, while new Uint8Array(someTypedArray) will only copy the data of someTypedArray.

When your image is large, because WASM cannot expand the memory that can hold the input and output image data, you may get the following error:

However, the image used in our example is small, so simply add a filter parameter -s ALLOW_MEMORY_GROWTH=1 at compile time and ignore the error message:

emcc -O3 -s WASM=1 -s EXTRA_EXPORTED_RUNTIME_METHODS='["cwrap"]' \

    -I libwebp \

    test-dir/webp.c \

    libwebp/src/{dec,dsp,demux,enc,mux,utils}/*.c \

    -s ALLOW_MEMORY_GROWTH=1
Copy the code

Running the above command again yields the wASM code with the encoding function added and the corresponding JavaScript glue code, so that when we open the HTML file, it is already able to encode a JPG file into WebP format. To further prove this point, We can display the image on the Web interface by modifying the HTML file and adding the following code:

<script> // ... api.encode(p, image.width, image.height, 100); const resultPointer = api.get_result_pointer(); const resultSize = api.get_result_size(); const resultView = new Uint8Array(Module.HEAP8.buffer, resultPointer, resultSize); const result = new Uint8Array(resultView); Const blob = new blob ([result], {type: 'image/webp'}); const blobURL = URL.createObjectURL(blob); const img = document.createElement('img'); img.src = blobURL; document.body.appendChild(img) api.free_result(resultPointer); api.destroy_buffer(p); </script>Copy the code

Then refresh the browser and you should see the WebP image displayed on the Web side. By downloading the file locally, you can see the format changed to WebP:

We successfully compiled the existing libwebp C library into WASM, and converted JPG images into WebP format and displayed them on the Web interface. Using WASM to handle computation-intensive transcoding operations can greatly improve the performance of Web pages. This is one of the main advantages WebAssembly brings.

How to compile FFmpeg to WebAssembly?

• emconfigureIs to replace the compiler from GCC to emcc (or g++ to em++) : compile C projects
• MakeGenerates WASM Object Files.ofile

In the second example we successfully compiled an existing C module into WebAssembly, but many projects rely on libraries such as AutoConfig/Automake to generate system-specific code before compiling. Emscripten provides emconfigure and Emmake to encapsulate these commands and inject appropriate parameters to smooth out projects that have front-loaded dependencies. Next we’ll show you how to handle this reliance on libraries such as AutoConfig/Automake to generate specific code by actually compiling FFMPEG.

After practice, ffMPEG compilation depends on a specific FFMPEG version, Emscripten version, operating system environment, etc., so the following FFMPEG compilation is limited to a specific condition, mainly for the general FFMPEG compilation to provide a way of thinking and debugging methods.

Compilation step

When compiling most complex C/C++ libraries using Emscripten, there are three main steps:

1. useemconfigureRunning the projectconfigureThe file will be C/C++ code compiler fromgcc/g++Switch toemcc/em++
2. throughemmake makeTo build C/C++ projects that generate WASM objects.ofile
3. Call manually in order to generate a particular form of outputemccTo compile a specific file

Install specific dependencies

In order to verify ffMPEG validation, we need to rely on a specific version, and the various file versions we rely on are described in detail below.

First, install the Emscripten compiler version 1.39.18. Before entering, Clone to the local emsdk project and run the following command:

/emsdk install 1.39.18./emsdk activate 1.39.18 source./emsdk_env.shCopy the code

Run the following command on the CLI to check whether the switchover is successful:

Emcc -v # output 1.39.18Copy the code

Download ffMPEG code with branch N4.3.1 from emSDK:

Git clone - the depth of 1 - branch n4.3.1 https://github.com/FFmpeg/FFmpegCopy the code

Use emconfigure to process configure files

Process the configure file with the following script:

export CFLAGS="-s USE_PTHREADS -O3" export LDFLAGS="$CFLAGS -s INITIAL_MEMORY=33554432" emconfigure ./configure \ -- target-OS =none # Set to none to remove some OS specific dependencies --arch=x86_32 # select architecture x86_32 --enable-cross-compile # handle cross-platform operations -- disable-x86ASM \ # Disable x86ASM --disable-inline-asm \ # Disable in-line ASM \ # Disable stripping --disable-doc -- add some flag output --extra-cflags="$cflags "\ -- extra-cxxFlags ="$CFLAGS" \ -- extra-ldFlags ="$ldflags "\ --nm="llvm-nm" \ # Use the LLVM compiler --ar=emar \ --ranlib=emranlib \ --cc=emcc \ # replace GCC with emcc -- CXX =em++ \ # replace g++ with em++ --objcc=emcc \ --dep-cc=emccCopy the code

The above script does a few things:

• USE_PTHREADSopenpthreadssupport
• -O3Optimizes code size at compile time, typically from 30MB to 15MB
• INITIAL_MEMORYSet it to 33554432 (32MB), mainly because Emscripten can take up 19MB, so set it to a larger memory capacity to avoid running out of memory to allocate during compilation
• The actual useemconfigureTo configure theconfigureFile, replacegccThe compiler toemcc, as well as setting up the necessary actions to handle compilation bugs that may be encountered, and finally generating a configuration file for compilation builds

Use emmake make to build dependencies

Now that you have the configuration files in place, you need to build the actual dependencies using emmake by running the following command from the command line:

# build the final ffmpeg.wasm file emmake make-j4Copy the code

Through the above compilation, the following four files are generated:

• ffmpeg

• ffmpeg_g

• ffmpeg_g.wasm

• ffmpeg_g.worker.js

The first two are JS files, the third is WASM module, and the fourth is the function that deals with the running of relevant logic in worker. The ideal form of the files generated above should be three. In order to achieve such customized compilation, it is necessary to customize emCC command for processing.

Use EMCC for personalized compilation

Create a wASM folder in the FFmpeg directory to place the built files, and customize the compiled files to the following output:

mkdir -p wasm/dist emcc \ -I. -I./fftools \ -Llibavcodec -Llibavdevice -Llibavfilter -Llibavformat -Llibavresample -Llibavutil -Llibpostproc -Llibswscale -Llibswresample \ -Qunused-arguments \ -o wasm/dist/ffmpeg-core.js fftools/ffmpeg_opt.c fftools/ffmpeg_filter.c fftools/ffmpeg_hw.c fftools/cmdutils.c fftools/ffmpeg.c \ -lavdevice -lavfilter-lavformat -lavcodec -lswresample-lswscale-lavutil-lm \ -o3 \ -s USE_SDL=2 \ # Use SDL2 -s USE_PTHREADS=1 \ -s INVOKE_RUN=0. -s INVOKE_RUN=0 EXPORTED_FUNCTIONS="[_main, _proxy_main]" \ -s EXTRA_EXPORTED_RUNTIME_METHODS="[FS, cwrap, setValue, writeAsciiToMemory]" \ -s INITIAL_MEMORY=33554432Copy the code

The above script has the following major improvements:

1. -s PROXY_TO_PTHREAD=1Set at compile timepthread, so that the program has responsive special effects
2. -o wasm/dist/ffmpeg-core.jsWill the originalffmpegThe output of the js file is renamed toffmpeg-core.js, corresponding outputffmpeg-core.wasm 和 ffmpeg-core.worker.js
3. -s EXPORTED_FUNCTIONS="[_main, _proxy_main]"Export ffmpeg corresponding C file inmainThe function,proxy_mainBy settingPROXY_TO_PTHREAD The agentmainFunction for external use
4. -s EXTRA_EXPORTED_RUNTIME_METHODS="[FS, cwrap, setValue, writeAsciiToMemory]"Export helper functions for exporting C functions, handling file systems, and pointer operations

The following three files are output through the above compilation command:

• ffmpeg-core.js

• ffmpeg-core.wasm

• ffmpeg-core.worker.js

Use the compiled FFMPEG WASM module

Create an ffmpeg.js file in the wASM directory and write the following code:

const Module = require('./dist/ffmpeg-core.js');



Module.onRuntimeInitialized = () => {

  const ffmpeg = Module.cwrap('proxy_main', 'number', ['number', 'number']);

};
Copy the code

Then run the above code with the following command:

node --experimental-wasm-threads --experimental-wasm-bulk-memory ffmpeg.js
Copy the code

The above code is explained as follows:

• OnRuntimeInitialized is the logic to be executed after the WebAssembly module is loaded. All of our related logic needs to be written in this function

• Cwrap is used to export proxy_main from C file (fftools/ffmpeg.c). The signature of the function is int main(int argc, char **argv). Int corresponds to JavaScript as number, and char **argv is a pointer to C that can also map to number

• To run ffmPEG-hide_banner from the command line, in our code we need to call main(2, [“./ffmpeg”, “-hide_banner”]). So how do we pass an array of strings? This question can be broken down into two parts:

  • We need to convert JavaScript strings into character arrays in C
  • We need to convert an array of numbers in JavaScript to an array of Pointers in C

The first part is easy, because Emscripten provides a helper function, writeAsciiToMemory, to do this:

const str = "FFmpeg.wasm"; const buf = Module._malloc(str.length + 1); / / allocate a byte of space to store extra 0 indicates the end of the string. The Module writeAsciiToMemory (STR, buf);Copy the code

The second part is a bit harder. We need to create an array of Pointers to 32-bit integers in C. We can use setValue to help us create this array:

const ptrs = [123, 3455];

const buf = Module._malloc(ptrs.length * Uint32Array.BYTES_PER_ELEMENT);

ptrs.forEach((p, idx) => {

  Module.setValue(buf + (Uint32Array.BYTES_PER_ELEMENT * idx), p, 'i32');

});
Copy the code

Putting the above code together, we get a program that can interact with FFMPEG:

const Module = require('./dist/ffmpeg-core');



Module.onRuntimeInitialized = () => {

  const ffmpeg = Module.cwrap('proxy_main', 'number', ['number', 'number']);

  const args = ['ffmpeg', '-hide_banner'];

  const argsPtr = Module._malloc(args.length * Uint32Array.BYTES_PER_ELEMENT);

  args.forEach((s, idx) => {

    const buf = Module._malloc(s.length + 1);

    Module.writeAsciiToMemory(s, buf);

    Module.setValue(argsPtr + (Uint32Array.BYTES_PER_ELEMENT * idx), buf, 'i32');

  })

  ffmpeg(args.length, argsPtr);

};
Copy the code

Then run the program with the same command:

node --experimental-wasm-threads --experimental-wasm-bulk-memory ffmpeg.js
Copy the code

The results of the above run are as follows:

You can see that we compiled and ran ffmpeg 🎉 successfully.

Process the Emscripten file system

Emscripten has built in a virtual file system to support standard file reads and writes in C, so we need to write audio files to the file system before passing them to FFmpeg.wASM.

You can click here to see more about file system APIS.

Fs.writefile () and fs.readfile () are just two functions in the FS module to accomplish the above tasks. For all data read and written from the file system, it requires the Uint8Array type in JavaScript. So it is necessary to agree on data types before consuming data.

We’ll read the video file named flame. Avi with the fs.readfilesync () method and then write it to the Emscripten file system using fs.writefile ().

const fs = require('fs');

const Module = require('./dist/ffmpeg-core');



Module.onRuntimeInitialized = () => {

  const data = Uint8Array.from(fs.readFileSync('./flame.avi'));

  Module.FS.writeFile('flame.avi', data);



  const ffmpeg = Module.cwrap('proxy_main', 'number', ['number', 'number']);

  const args = ['ffmpeg', '-hide_banner'];

  const argsPtr = Module._malloc(args.length * Uint32Array.BYTES_PER_ELEMENT);

  args.forEach((s, idx) => {

    const buf = Module._malloc(s.length + 1);

    Module.writeAsciiToMemory(s, buf);

    Module.setValue(argsPtr + (Uint32Array.BYTES_PER_ELEMENT * idx), buf, 'i32');

  })

  ffmpeg(args.length, argsPtr);

};
Copy the code

Compile the video using FFmpeg. wASM

Now that we can save the video file to the Emscripten file system, it’s time to actually transcode the video using the compiled FFMEPG.

We modify the code as follows:

const fs = require('fs'); const Module = require('./dist/ffmpeg-core'); Module.onRuntimeInitialized = () => { const data = Uint8Array.from(fs.readFileSync('./flame.avi')); Module.FS.writeFile('flame.avi', data); const ffmpeg = Module.cwrap('proxy_main', 'number', ['number', 'number']); const args = ['ffmpeg', '-hide_banner', '-report', '-i', 'flame.avi', 'flame.mp4']; const argsPtr = Module._malloc(args.length * Uint32Array.BYTES_PER_ELEMENT); args.forEach((s, idx) => { const buf = Module._malloc(s.length + 1); Module.writeAsciiToMemory(s, buf); Module.setValue(argsPtr + (Uint32Array.BYTES_PER_ELEMENT * idx), buf, 'i32'); }); ffmpeg(args.length, argsPtr); const timer = setInterval(() => { const logFileName = Module.FS.readdir('.').find(name => name.endsWith('.log')); if (typeof logFileName ! == 'undefined') { const log = String.fromCharCode.apply(null, Module.FS.readFile(logFileName)); if (log.includes("frames successfully decoded")) { clearInterval(timer); const output = Module.FS.readFile('flame.mp4'); fs.writeFileSync('flame.mp4', output); }}}, 500); };Copy the code

In the above code, we added a timer. Since the process of ffMPEG transcoding video is asynchronous, we need to constantly read whether there is a good transcoding file flag in the Emscripten file system. When we get the file flag and it is not undefined, We use the module.fs.readfile () method to read transcoded video files from the Emscripten file system, and then write the video to the local file system via fs.writefilesync (). In the end we receive the following result:

Use FFMPEG to transcode the video in your browser and play it

In the previous step, we successfully transcoded avi to MP4 format using ffMPEG compiled on the Node side. Next, we will transcode the video using FFMPEG in the browser and play it in the browser.

The ffMPEG we compiled can convert AVI files to MP4, but mp4 files cannot be played directly in the browser. Because this encoding is not supported, we need to use the libx264 encoder to encode MP4 files into a browser playable encoding format.

X264 encoder source code:

curl -OL https://download.videolan.org/pub/videolan/x264/snapshots/x264-snapshot-20170226-2245-stable.tar.bz2

ttar xvfj x264-snapshot-20170226-2245-stable.tar.bz2
Copy the code

Then go to the x264 folder and create a build.sh file and add the following:

#! /bin/bash -x ROOT=$PWD BUILD_DIR=$ROOT/build cd $ROOT/x264-snapshot-20170226-2245-stable ARGS=( --prefix=$BUILD_DIR --host=i686-gnu # use i686 gnu --enable-static # enable building static library --disable-cli # disable cli tools --disable-asm # disable asm optimization --extra-cflags="-s USE_PTHREADS=1" # pass this flags for using pthreads ) emconfigure ./configure "${ARGS[@]}" emmake make install-lib-static -j4 cd -Copy the code

Note that you need to run the following command in the WebAssembly directory to build x264:

bash x264-snapshot-20170226-2245-stable/build-x264.sh
Copy the code

Configure. Sh = configure.sh = configure.sh = configure.sh = configure.sh

#! /bin/bash -x emcc -v ROOT=$PWD BUILD_DIR=$ROOT/build CD $ROOT/ffmpeg-4.3.2-3 CFLAGS="-s USE_PTHREADS -I$BUILD_DIR/include" LDFLAGS="$CFLAGS -L$BUILD_DIR/lib -s INITIAL_MEMORY=33554432" # 33554432 bytes = 32 MB CONFIG_ARGS=( --target-os=none # use none to prevent any os specific configurations --arch=x86_32 # use x86_32 to achieve minimal architectural optimization --enable-cross-compile # enable cross compile --disable-x86asm # disable x86 asm --disable-inline-asm # disable inline asm --disable-stripping --disable-programs # disable programs build (incl. ffplay, ffprobe & ffmpeg) --disable-doc # disable doc --enable-gpl ## required by x264 --enable-libx264 ## enable x264 --extra-cflags="$CFLAGS" --extra-cxxflags="$CFLAGS" --extra-ldflags="$LDFLAGS" --nm="llvm-nm" --ar=emar --ranlib=emranlib --cc=emcc --cxx=em++ --objcc=emcc --dep-cc=emcc ) emconfigure ./configure "${CONFIG_ARGS[@]}" # build ffmpeg.wasm emmake make -j4 cd -Copy the code

Then create a script file called build-ffmpeg.sh for customizing the output build file:

ROOT=$PWD BUILD_DIR=$ROOT/build CD ffmpeg-4.3.2-3 ARGS=(-i. -I./fftools -i $BUILD_DIR/ include-llibavcodec -Llibavdevice  -Llibavfilter -Llibavformat -Llibavresample -Llibavutil -Llibpostproc -Llibswscale -Llibswresample -L$BUILD_DIR/lib -qunused-arguments # add -lpostproc and -lx264 to the line -o wasm/dist/ffmpeg-core.js fftools/ffmpeg_opt.c fftools/ffmpeg_filter.c fftools/ffmpeg_hw.c fftools/cmdutils.c fftools/ffmpeg.c -lavdevice -lavfilter -lavformat -lavcodec -lswresample -lswscale -lavutil -lpostproc -lm -lx264 -pthread -O3 # Optimize code with performance first -s USE_SDL=2 # use SDL2 -s USE_PTHREADS=1 # enable pthreads support -s PROXY_TO_PTHREAD=1 # detach main() from browser/UI main thread -s INVOKE_RUN=0 # not to run the main() in the beginning -s EXPORTED_FUNCTIONS="[_main, _proxy_main]" # export main and proxy_main funcs -s EXTRA_EXPORTED_RUNTIME_METHODS="[FS, cwrap, setValue, writeAsciiToMemory]" # export preamble funcs -s INITIAL_MEMORY=268435456 # 268435456 bytes = 268435456 MB ) emcc "${ARGS[@]}" cd -Copy the code

Ffmpeg transcoding is actually used

We will create a Web page and then provide a button to upload the video file and play the uploaded video file. Although it is not possible to play avi format video files directly on the Web, we can use FFMPEG transcoding to play them.

Create the index.html file in the WASM folder in the FFmpeg directory and add the following:

<html> <head> <style> html, body { margin: 0; width: 100%; height: 100% } body { display: flex; flex-direction: column; align-items: center; } </style> </head> <body> <h3> </h3> <video ID ="output-video" controls></video><br/> <input type="file" id="uploader"> <p id="message"> The ffmPEG script takes 5 seconds </p> <script type="text/javascript"> const readFromBlobOrFile = (blob) => (new Promise(resolve, resolve, reject) => { const fileReader = new FileReader(); fileReader.onload = () => { resolve(fileReader.result); }; fileReader.onerror = ({ target: { error: { code } } }) => { reject(Error(`File could not be read! Code=${code}`)); }; fileReader.readAsArrayBuffer(blob); })); const message = document.getElementById('message'); const transcode = async ({ target: { files } }) => { const { name } = files[0]; Message. innerHTML = 'write a file to the Emscripten file system '; const data = await readFromBlobOrFile(files[0]); Module.FS.writeFile(name, new Uint8Array(data)); const ffmpeg = Module.cwrap('proxy_main', 'number', ['number', 'number']); const args = ['ffmpeg', '-hide_banner', '-nostdin', '-report', '-i', name, 'out.mp4']; const argsPtr = Module._malloc(args.length * Uint32Array.BYTES_PER_ELEMENT); args.forEach((s, idx) => { const buf = Module._malloc(s.length + 1); Module.writeAsciiToMemory(s, buf); Module.setValue(argsPtr + (Uint32Array.BYTES_PER_ELEMENT * idx), buf, 'i32'); }); Message. innerHTML = 'start transcoding '; ffmpeg(args.length, argsPtr); const timer = setInterval(() => { const logFileName = Module.FS.readdir('.').find(name => name.endsWith('.log')); if (typeof logFileName ! == 'undefined') { const log = String.fromCharCode.apply(null, Module.FS.readFile(logFileName)); if (log.includes("frames successfully decoded")) { clearInterval(timer); Message. innerHTML = 'Complete transcoding '; const out = Module.FS.readFile('out.mp4'); const video = document.getElementById('output-video'); video.src = URL.createObjectURL(new Blob([out.buffer], { type: 'video/mp4' })); }}}, 500); }; document.getElementById('uploader').addEventListener('change', transcode); </script> <script type="text/javascript" src="./dist/ffmpeg-core.js"></script> </body> </html>Copy the code

Open the above webpage to run, we can see the following effect:

Congratulations to you! Ffmpeg was successfully compiled and used on the Web side.

reference

• www.ruanyifeng.com/blog/2017/0…
• Pspdfkit.com/blog/2017/w…
• Hacks.mozilla.org/2017/02/wha…
• www.sitepoint.com/understandi…
• www.cmake.org/download/
• Developer.mozilla.org/en-US/docs/…
• Research.mozilla.org/webassembly…
• Itnext. IO/build – ffmpe…
• Dev. To/alfg/ffmpeg…
• Gist.github.com/rinthel/f4d…
• Github.com/Kagami/ffmp…
• Qdmana.com/2021/04/202…
• Github.com/leandromore…
• Ffmpeg.org/doxygen/4.1…
• Github.com/alfg/ffmpeg…
• Github.com/alfg/ffprob…
• Gist.github.com/rinthel/f4d…
• Emscripten.org/docs/compil…
• Itnext. IO/build – ffmpe…
• Github.com/mymindstorm…
• Github.com/emscripten-…
• Github.com/FFmpeg/FFmp…
• Yeasy. Gitbook. IO/docker_prac…

❤ ️ /Thank you for support /

That is all the content of this sharing. I hope it will help you

Don’t forget to share, like and bookmark your favorite things

Welcome to the public number programmer bus, from byte, shrimp, zhaoyin three brothers, share programming experience, technical dry goods and career planning, help you to avoid detours into the factory.