Use wechat QR code engine on web page

background

I have a tool class project developed based on the Electron cross-platform framework, and I need to make a plug-in to recognize and parse two-dimensional code on PC.

I started with Zxing webAssembly version. In the process of use, I found that in some edge cases (two-dimensional code is very small or very large, pictures tilt, etc.), the recognition ability is very poor, time-consuming and also does not support a picture of multiple codes. And from the point of view of daily use, wechat on two-dimensional code recognition and decoding ability is the peak, the recognition rate is high, fast, but also has good support for a picture of many codes, if you can put the wechat ability to transfer to the PC that is the most perfect.

And wechat big data is really terrible, instant insight into my heart, the wechat two-dimensional code recognition decoding ability open source. The wechat team has opened source the QR code engine in opencV contrib project. This engine is based on Zxing and has a number of optimizations, including deep learning, convolutional networks and other optimizations, to improve qr code recognition and decoding capabilities. This article documents some of the problems and solutions encountered in compiling the engine into webAssembly

Opencv project introduction

In order to compile this module, you need to have a general understanding of the OpencV project.

Opencv is divided into opencV master library and OpencV contrib optional module library.

The opencV main library focuses on two directories — modules and Platforms.

The modules directory contains the main library module code, such as core core function module, DNN model module, imgProc image processing module, ObjDetect image recognition module, etc. It also contains the generation of the corresponding language module, such as Java, object-C, Python, JS, etc. And this compilation of the main processing logic is also concentrated in the JS directory.

Platforms directory is a portal to create libraries available on each platform, named by platform, such as Android, ios, Linux, etc. This time we’ll focus on the JS directory, which is where the portal files and configuration for creating the webAssembly version of OpencV are located.

Take a look at the Contrib library for OpencV, which is a separate Git repository that is optional when building OpencV. The structure of this warehouse is relatively simple, modules are under modules, open modules directory, you can see the TWO-DIMENSIONAL code module wechat_qrcode of wechat.

We compiled wechat_qrcode from the Contrib repository in OpencV into webAssembly to run in a Web browser.

The build process

Opencv provides a built version of webAssembly configured by default, which can be downloaded from the Release page of the OpencV library. Of course, the default build version does not include wechat QR code module. So I need to build my own version of webAssembly for my own needs.

Compilation mode selection

Opencv also provides a webAssembly build guide. I use Docker to build, and I also recommend Docker to build, for the following reasons:

1. You no longer need to install the runtime environment required for compilation, such as Emscripten.
2. You don’t have to worry about all kinds of exceptions caused by the compiler environment being out of sync.

My compile environment is

1. Linux centos distribution
2. Docker version 19.03.1, build 74b1e89
3. Opencv 4.5.2
4. opencv_contrib commit 10d1020952f7924e94f5bab1659c328c599f1c61

Try compiling the OpencV project

Pull the opencv, opencv_contrib repositories down to the local directory ocv as follows:

- ocv
   + opencv/
   + opencv_contrib/
Copy the code

First, of course, try to go straight to the official documentation and build with the default configuration and see if it works. Avoid directly up according to a magic change, change after compiling, mistakenly thought that is a change problem, after a long time only to find that is not the case.

According to the official documentation, Emscripten version 2.0.10 is the official certified version of OpencV. So instead of using the latest version, build with version 2.0.10 for stability. The build command is as follows:

cd ocv

sudo docker run --rm -v $(pwd) : / SRC - u $(id - u) : $(id - g) emscripten/emsdk: 2.0.10 emcmake python3. / opencv/platforms/js/build_js py build_wasm --build_wasm --build_testCopy the code

Note that the –build_wasm and –build_test parameters are added to build the webAssembly version and automatically build test cases for immediate testing after the build is complete

After some frantic output from the console, the build was successful, with the following output:

=====
===== Build finished
=====
OpenCV.js location: /src/build_wasm/bin/opencv.js
OpenCV.js tests location: /src/build_wasm/bin/tests.html
Copy the code

The directory structure is as follows:

- ocv
   + opencv/
   + opencv_contrib/
   - build_wasm/
      ...
      - bin/
         - tests.html
         - opencv.js
      ...
Copy the code

The build_wasm directory is the directory generated by the build. Tests. HTML is a file automatically generated when the –build_test parameter is used. I use the HTTP-server module of NPM, and the installation command is as follows

npm install -g http-server
Copy the code

Then run the hs -p 5000 command in the OCV directory. The ocV directory is used as the server root directory to start an HTTP-server service with port 5000. Can open http://127.0.0.1:5000/build_wasm/bin/tests.html in the browser test cases to run automatically, the output.

Since nothing was changed during this compilation, the test cases are expected to pass, and the results are as expected.

Try compiling the opencv_contrib module

The wechat_qrcode module is provided in opencv_contrib. You need to add a parameter to the previous command –cmake_option=” -dopencv_extra_modules_path =/ SRC /opencv_contrib/modules”. The complete command is as follows:

cd ocv

sudo docker run --rm -v $(pwd) : / SRC - u $(id - u) : $(id - g) emscripten/emsdk: 2.0.10 emcmake python3. / opencv/platforms/js/build_js py build_wasm --build_wasm --build_test --cmake_option="-DOPENCV_EXTRA_MODULES_PATH=/src/opencv_contrib/modules"
Copy the code

The above command only introduces the opencv_contrib library but does not compile into the webAssembly file. As there is no call entry, it is optimized at build time, so additional configuration needs to be added. The wechat_qrcode module interface is exposed in the compiled webAssembly. This configuration file in opencv/platforms/js/opencv_js config. Py file, the file defines the various modules in the compiled API exposed, the API can be invoked in js.

To add the wechat_qrcode module, add the following configuration

#...

wechat_qrcode = {
  'wechat_qrcode_WeChatQRCode': ['WeChatQRCode'.'detectAndDecode']
}

white_list = makeWhiteList([core, imgproc, objdetect, video, dnn, features2d, photo, aruco, calib3d, wechat_qrcode])
Copy the code

Then execute the above command to compile, if it goes well, the compile can be tested and verified. But everything has a but. The error is as follows:

Failure on the pit

The wish is good, but the reality is harsh. Compiling wechat_qrcode failed, and in order to be able to use the module on the Web, a bit is needed to fix these problems.

No memeber named ‘vectorstd’ in namespace ‘STD’

The error message here is very clear, Can see is in the generated build_wasm/modules/js_bindings_generator/gen/bindings in the CPP file wechat_qrcode module detectAndDecode function return type is not normal

Contrast opencv_contrib/modules/wechat_qrcode/include/opencv2 / wechat_qrcode HPP this function in the statement, The bindings ::vector< STD ::string> is returned by STD ::vector< STD ::string>, but the bindings ::vectorstd::string is returned by STD ::vectorstd::string. Swallows two <> conjunctions from the original return type.

Knowing why, you need to locate the logical code that generated this function declaration, which is clearly an error in the generation process.

Traced back found generated bindings. The logic of the CPP file in opencv/modules/js/generator/embindgen py file, so we can replace the simple and crude directly the return type of the error, as follows:

The replacement place is in the gen_function_binding_with_wrapper method under the JSWrapperGenerator class.

When the replacement is complete, compile again and find that the error is gone. But there was another mistake

Unknown type name ‘string’; did you mean ‘String’

The error screenshot is as follows:

Same file, different problem, because the generated file does not contain namespace STD :: prefix, so we can also be generated in the corresponding location, simply replace the wrong string, as shown in the following figure:

So we compile again, and there are no more errors, and we compile successfully.

The successful compilation does not mean that it can run normally in the Web browser, we need to be able to recognize and decode the TWO-DIMENSIONAL code in the browser is done. To do this, we need to prepare a piece of TEST JS code

var img = document.createElement('img')
// prepare an image of qrcode and put it in the ocv directory. Name it qrcode.png
img.src = '/qrcode.png'
img.onload = () = > {
  // Read the image data
  var imgdata = cv.imread(img)

  var detector = new cv.wechat_qrcode_WeChatQRCode(
    "wechat_qrcode/detect.prototxt"."wechat_qrcode/detect.caffemodel"."wechat_qrcode/sr.prototxt"."wechat_qrcode/sr.caffemodel"
  )

  var results = detector.detectAndDecode(imgdata)

  // Print the result of the first qr code identified
  console.log(results.get(0))}Copy the code

When you instantiate the qr code engine, you need to pass in 4 model files, but these 4 model files are read from the file system in C++, but after compiling to webAssembly, how to read these 4 files?

3. Model file loading problem

Google discovered that webAssembly mimics a file system, packaging files and reading them as if they were a file system.

Specific can refer to: www.cntofu.com/book/150/zh…

For reference in this article, I packaged wechat_qrcode_files.js into wechat_qrcode_files.js using a plug-in file package.

cd ocv

# Pull the Emscripten repository locally
git clone https://github.com/emscripten-core/emscripten.git

# Yes, the 4 model files are now in build_wasm/downloads/wechat_qrcode
cp -r build_wasm/downloads/wechat_qrcode ./

# package file
sudo docker run --rm -v /data/home/marchyang/mine/ocv:/src -u $(id -u):$(id -g) emscripten/emsdk python3 emscripten/tools/file_packager.py build_wasm/bin/wechat_qrcode_files.data --preload wechat_qrcode/ --js-output=build_wasm/bin/wechat_qrcode_files.js
Copy the code

The build_wasm/bin/ directory generates two new files wechat_qrcode_files.data and wechat_qrcode_files.js. These two files correspond to the files in the file system and the JS code of the simulated file system respectively. Next, use script in tests.html to import wechat_qrcode_files.js, as shown in the figure

Note that it must be introduced after the Module declaration, as shown in the figure. This will cause problems because wechat_qrcode_files.js inserts a line of code in module. preRun to create the filesystem. If it is introduced before the Module declaration in the diagram, The preRun in the following Module will override the preRun, preventing you from creating a file system, which will result in an error reading of the file.

Then refresh the page, open the JS running console, and then run the above test code, and see the effect as shown below

Troubleshooting found that the error was reported during initialization

4. Runtime error: When initializing a classwechat_qrcode_WeChatQRCodeClass, error processing is reported

Seeing the error on the console and looking at the stack of error messages, it was a big surprise that there was nothing in the error message, so there was no way to get any useful information or guess which part of the source code had failed.

So far, almost give up, feel there is no way. There was hope, however, that Chrome was so powerful that there should be a way to debug webAssembly, so Google did, as I expected, and Chrome did provide a way to debug webAssembly, as you can see in this article

Specific practices:

1. Compile time plus-gparameter
2. Chrome Developer Tools opens webAssembly debugging
3. When compiling on Docker, you need to map paths
4. Break point, debug

According to the article, and compiling parameters, first compiled script in opencv/modules/js/CMakeLists. TXT, modify the following:

After compiling, execute the test script again, and the following error message is displayed:

You can see the source code information already in the error stack. But because MY side is on the remote development machine using Docker compilation, and then remote development machine port mapping to the local test, so the source code can not be displayed, it is not beautiful.

According to the error stack, wechat_qrcode_WeChatQRCode initialization function CV ::utils::fs::exists is called.

Why does CV ::utils::fs::exists fail when a file system mapping is loaded? I can’t figure it out. Stuck here for a long time.

Could it be that the CV ::utils::fs::exists method is not compatible when compiled to webAssembly? Immediately think, feeling and unlikely, because from the name of this method can be known, this method is opencV tool class library method, should be more stable, should not appear this kind of problem just right.

However, although I think it is unlikely, but because I can’t find a clue, I can only investigate according to this idea first.

The CV ::utils::fs::exits method is called to check whether the file path that is passed to wechat_qrcode_WeChatQRCode exists. Then invokes the CV: : : within DNN: readNetFromCaffe incoming initialization file path model of the detector.

So if we skip the test file, go directly to call CV: : : within DNN: readNetFromCaffe, if successful, not can prove CV: : utils: : fs: : it’s really something wrong with the exists? And just in the build configuration, CV: : : within DNN: readNetFromCaffe method is deduced, so can invoke this method directly in the console to verify, below:

Oh, my God, what did I see? The direct call worked. I don’t want to believe it, but the fact that CV ::utils::fs::exists is problematic.

When you know the problem, it’s easier to solve it. Comment out the call to ‘CV ::utils::fs::exists’ in opencv_contrib/modules/wechat_qrcode/ SRC /wechat_qrcode. CPP.

Then recompile and run the test code to verify. The operation effect is shown below

Wechat_qrcode_WeChatQRCode has been successfully initialized. The error in the figure was detected when the detectAndDecode method of the instance was called.

P.S., when parsing the constructor, found that if four empty strings were passed, it would no longer determine that the file exists and would not read the model file. So var wr = new CV. Wechat_qrcode_WeChartQRCode (“, “, “, “, “) can also be successfully instantiated and called WITH VR.DETECtanddecode. Of course, the detection and analysis of two-dimensional code can not be used to train the model, the identification and decoding effect may be compromised.

4. DetectAndDecode error, UnboundTypeError

Unbound: STD ::vector< STD ::string> Unbound: STD ::vector< STD ::string> The STD ::vector< STD :: String > type was not exported when compiled to webAssembly.

I thought this might be a bit of a problem to solve, but it turns out that there is a register_vector function in core_binding. CPP, as shown below:

It looks like a method that registers a vector as the corresponding JS class, so try registering a vector< STD ::string> to see if that’s OK. The final changes to this file are as follows:

Then run the compile command again to compile, no error is reported. Refresh the page, re-execute the test code on the console, and the result is as follows:

It can be found that the content of the TWO-DIMENSIONAL code has been output and no error has been reported, which is consistent with our expected results. At this point, the wechat TWO-DIMENSIONAL code recognition module has finally run on the Web page.

Reduce file size

So far, the QR code engine of wechat has been successfully compiled and run in the Web environment, but it can be found from the network that the compiled opencv.js file is very large, with 11.5m, which is really too large. When used in a production environment, the package size needs to be reduced

There are two steps:

1. Remove the -g parameter from the package. This parameter is added to enable Chrome developer tools to display and debug webAssembly. This parameter greatly increases the size of opencv.js files. The package file size can be greatly reduced. In actual measurement, if the -g parameter is removed, the file size is reduced from 11.5m to 8.9m, which is reduced by 2.6m or 22%

2. Modify the opencv/platforms/js/opencv_config js. Py files, only keep wechat_qrcode whitelist, below:

This step reduces the size of the 8.9m file to 4.6m again, reducing the size by 4.3m, and reducing the file size by nearly 50%. It’s kind of amazing

The Demo page

After successful compilation, take out your compiler product made a demo page, the address is: qwertyyb. Making. IO/wechat_qrco…

This page also output the two-dimensional code in the picture location information, the specific use method can refer to the source code

The next step

So far, the wechat TWO-DIMENSIONAL code engine has been successfully compiled into webAssembly, which can run in the Web and decode the two-dimensional code content successfully.

I built and compiled this version because I need to use it in a project based on electron, so I will introduce it in the tool next to see if there are pits.

In addition, in the compilation process, encountered these problems can also be further studied, to see if opencV can mention Mr.