OpenCV overview

OpenCV (Open Source Computer Vision Library) is an Open Source Computer Vision Library, it provides many functions, these functions are very efficient implementation of Computer Vision algorithms (most basic filtering to advanced object detection cover). OpenCV has a wide range of applications, including image Mosaic, image noise reduction, product quality inspection, human-computer interaction, face recognition, motion recognition, motion tracking, unmanned driving and so on. OpenCV also provides machine learning modules where you can use normal Bayes, K-nearest neighbor, support vector machines, decision trees, random forests, artificial neural networks, and other machine learning algorithms.

Second, OpenCV framework

2.1 Overall structure of OpenCV

OpenCV uses C/C++ development, but also provides Python, Java, MATLAB and other languages interface, and provides Windows, Android (Download the Android library), ios and other platforms.

OpenCV is made up of modules that can be divided into many layers:

• At the bottom level are various hardware optimizations based on the Hardware Acceleration Layer (HAL).
• The next layer is the OpenCV core layer, which includes Core, IMGProc, CaliB3D, ObjDetect, and more.
• The next level up is Opencv_contrib, which can be understood as an extension of OpenCV and contains some of the latest algorithms, but the API is not yet stable or fully tested.
• Next up are the language bindings and sample applications.
• At the top is the interaction between OpenCV and the operating system.

2.2 OpenCV core module

We will focus on the OpenCV core layer. The following table lists the modules contained in the OpenCV core layer:

The module	instructions
core	This module contains the OpenCV library infrastructure and basic operations.
improc	The image processing module includes basic image transformation, including filtering and similar convolution operations.
highgui	In OpenCV 3.0, it is divided into imCOdecs, Videoio and Highgui. This module contains user interaction functions that can be used to display images or simple input. This can be seen as a very lightweight Windows UI toolkit.
video	This module contains functions for reading and writing video streams.
calib3d	This module includes the implementation of algorithms for calibrating single, binocular and multiple cameras.
feature2d	This module contains algorithms for detecting, describing, and matching feature points.
objdectect	This module contains algorithms for detecting specific objects, such as faces or pedestrians. Detectors can also be trained to detect other objects.
ml	The machine learning module itself is a very complete module, containing a large number of machine learning algorithm implementation and these algorithms can naturally interact with OpenCV data types.
flann	Flann stands for “fast nearest Neighbor library”. This library contains methods that you may not use directly, but are called by functions in other modules to do nearest neighbor searches in the dataset.
GPU	Split into cudA * modules in OpenCV. The GPU module is mainly an optimized implementation of functions on CUDA Gpus. In addition, there are some gPU-only functions. Some of these functions return good results, but they require good enough computing resources, and if the hardware doesn’t have a GPU, it won’t improve.
photo	This is a fairly new module that contains some functional tools for computational photography.
stitching	This module is an exquisite image Mosaic process. This is a new feature in the library, but, like the Photo module, this area is expected to grow significantly in the future.
nonfree	In OpenCV 3.0, it was moved to Opencv_contrib/xFeatures2d. OpenCV contains some proprietary or restricted algorithms (such as SIFT). These algorithms are isolated in their own modules to indicate that you need to do some special work before you can use them in commercial products.
contrib	In OpenCV 3.0, opencv_contrib was merged. This module contains some new things that have not yet been integrated into the OpenCV library.
legacy	In OpenCV 3.0, cancelled. This module contains some old stuff that hasn’t been completely cancelled.
ocl	In OpenCV 3.0, this was removed and replaced with t-API. This is a relatively new module that can be considered similar to the GPU module in that it implements the Khronos OpenCL standard for open parallel programming. While modules currently have fewer features than GPU modules, the goal of the OCL module is to provide the ability to run on any GPU or other parallel devices that can run on Khronos. This is in stark contrast to GPU mods, which were developed using the Nividia CUDA toolkit and therefore work only on Nividia GPU devices.

inOpenCV githubAll the core Opencv modules can be seen in the project:If only part of the OpenCV module is needed in the project and the project volume is strict, it can be based on the project needsTailoring OpenCV, package the required modules separately into SO.

Third, integrate OpenCV

Before integrating OpenCV, you need to know some NDK development basics (see Android NDK Development Basics). To integrate OpenCV, you can download the official build directly from the OpenCV Release page, or go to OpenCV Github to download and build as needed. This article mainly introduces the official Android platform Opencv-4.5.5 version integration.

3.1 Product Introduction

After downloading the official Android platform Opencv-4.5.5 product, the contents are as follows:

• Sample: official example;
• SDK: OpencV core SDK;
  • Java: OpencV Java layer; Java layer package structure is consistent with the above OpencV module division, which provides the corresponding Java classes, the actual implementation of the form of JNI implementation;
  • Libcxx_helper: stores CMakeList;
  • Native: OpencV Native layer;
    • Jni: OpencV JNI layer;
    • Libs: OpencV contains all core libraries and jNI’s dynamic library;
    • Staticlibs: static library of each module of OpenVC core;
    • 3rdparty: third-party library;

The official product has a total size of 900M, from which we can select the part we need for integration, or select the part we need for business from the static library. A compiles the product for integration.

3.2 Java + Native integration

If your project needs to write OpenCV code in the Java layer, you need to integrate the Java layer with the Native layer.

3.2.1 Introduction to the OpenCV Java Class

In the above OpenCV Java directory, in addition to the Java classes that provide OpenCV core modules, Org.opencv. Android package also provides OpenCVNativeLoader loading class, JavaCameraView, CameraGLSurfaceView camera package, etc. OpenCV Java directory provides the core module Java class, the underlying implementation is through the Native layer, Java layer only references the address of the native layer object, such as OpenCV basic image class Mat:

public class Mat {
    /** * Native object address **/
    public final long nativeObj;

    public Mat(long addr) {
        if (addr == 0)
            throw new UnsupportedOperationException("Native object address is NULL");
        nativeObj = addr;
    }

    // javadoc: Mat::Mat()
    public Mat(a) {
        nativeObj = n_Mat();
    }
    // C++: Mat::Mat()
    private static native long n_Mat(a);
    
    // javadoc: Mat::dims()
    public int dims(a) {
        return n_dims(nativeObj);
    }
    // C++: int Mat::dims()
    private static native int n_dims(long nativeObj);
}
Copy the code

Opencv core module c/ C ++ methods are called in jNI:

#include "opencv2/core.hpp"

#define LOG_TAG "org.opencv.core.Mat"
#include "common.h"

using namespace cv;
/* * Class: org_opencv_core_Mat * Method: n_Mat * Signature: (IIILjava/nio/ByteBuffer; J)J * * Mat::Mat(int rows, int cols, int type, void* data, size_t step) */
JNIEXPORT jlong JNICALL Java_org_opencv_core_Mat_n_1Mat__IIILjava_nio_ByteBuffer_2J
  (JNIEnv* env, jclass, jint rows, jint cols, jint type, jobject data, jlong step);

JNIEXPORT jlong JNICALL Java_org_opencv_core_Mat_n_1Mat__IIILjava_nio_ByteBuffer_2J
  (JNIEnv* env, jclass, jint rows, jint cols, jint type, jobject data, jlong step)
{
    static const char method_name[] = "Mat::n_1Mat__IIILjava_nio_ByteBuffer_2J()";
    try {
        LOGD("%s", method_name);
        return (jlong) new Mat(rows, cols, type, (void*)env->GetDirectBufferAddress(data), (size_t)step);
    } catch(const std::exception &e) {
        throwJavaException(env, &e, method_name);
    } catch(...). {throwJavaException(env, 0, method_name);
    }

    return 0;
}
Copy the code

3.2.2 Integrated content

To integrate OpenCV into the project, we first create a new Opencv_SDK module, and add the Java directory, libcxx_Helper directory and native lib directory from the official website (only arm64-V8A and ArmeabI-V7A are selected for the architecture), as follows:

At this point, the Opencv_SDK module is integrated, and you can call opencV methods in other modules through Java objects.

3.2.3 Simple Usage

Here is a simple example: Create an OpenCV Mat using a bitmap in the Java layer and print the log. Build. Gradle: build. Gradle

dependencies {
    implementation(project(':opencv_sdk'))
}
Copy the code

Then you can use OpenCV’s Java class in the app module:

Bitmap bitmap = BitmapFactory.decodeResource(this.getResources(), R.drawable.adventure_time);
// OpenCV provides a wrapper class to load libopencv_javA4.so
OpenCVNativeLoader openCVNativeLoader = new OpenCVNativeLoader();
openCVNativeLoader.init();
ByteBuffer buffer = ByteBuffer.allocateDirect(bitmap.getByteCount());
bitmap.copyPixelsToBuffer(buffer);
Mat mat = new Mat(bitmap.getHeight(), bitmap.getWidth(), CvType.CV_8UC4, buffer);
Log.d("OpenCV"."mat channels:"+mat.channels()+", cols:"+mat.cols()+", rows:"+mat.rows());
Copy the code

If the running result is as follows, the Mat is created successfully:

D/OpenCV: mat channels:4, cols:3000, rows:4500
Copy the code

3.3 Native integration

If the project only needs to write OpenCV code in the Native layer, then it only needs to integrate the Native layer. You can remove the Java directory above to reduce the package size. Opencv_sdk module only in the items need to be introduced into OpenCV header files (product on the website of OpenCV – android SDK/SDK/native/jni/include directory) and OpenCV dynamic library, as shown below:

3.3.1 Creating a Native module

First, create a Sample_native module to store JNI and native codes. The module structure is as follows:

3.3.2 rainfall distribution on 10-12 configure cmake

Cmake and OpencV in build.gradle:

android {
    defaultConfig {
        ndk {
            // Specify the COMPILED ABI schema
            abiFilters "armeabi-v7a"."arm64-v8a"
        }
        externalNativeBuild {
            / / cmake configuration
            cmake {
                cppFlags "-std=c++11 -frtti -fexceptions"
                arguments "-DANDROID_STL=c++_shared"
                arguments "-DANDROID_TOOLCHAIN=clang"
                // Configure the OpencV header directory and dynamic library directory
                arguments "-DOPENCV_HEADER_DIR=${project.rootProject.project("opencv_sdk").file("include").path}"
                arguments "-DOPENCV_LIBS_DIR=${project.rootProject.project("opencv_sdk").file("native/libs").path}"
            }
        }
    }
    externalNativeBuild {
        cmake {
            // Build script path
            path "src/main/cpp/CMakeLists.txt"
            version "3.10.2"}}}Copy the code

This completes the basic configuration of cmake, which uses arguments to specify the location of opencV headers and dynamic libraries.

3.3.3 Writing JNI code

In sample_native module, create jNI interface as follows:

public class OpenCVSample {

    private static boolean isLoadSuccess = false;

    static {
        try {
            System.loadLibrary("native-lib");
        } catch(Throwable throwable) { throwable.printStackTrace(); }}public static native void createMat(Bitmap bitmap);

}
Copy the code

Create a CPP directory and create a native-lib.h header and a native-lib. CPP file as well as a cmakelists. TXT file.

${OPENCV_HEADER_DIR}) # exclude_directories (${OPENCV_HEADER_DIR}) # Add_library (native-lib SHARED native-lib.cpp) # add a library that is already built, IMPORTED add_library(opencv_JAVA4 SHARED) set_target_properties(opencv_JAVA4 PROPERTIES IMPORTED_LOCATION ${OPENCV_LIBS_DIR}/${ANDROID_ABI}/libopencv_java4.so") Target must first be created with add_library(); target_link_libraries( native-lib opencv_java4 jnigraphics log )Copy the code

Then, the corresponding header file native-lib.h code of JNI is improved as follows:

#include <jni.h>
#include <android/log.h>
#include <android/bitmap.h>

#include <opencv2/core.hpp>
#include <opencv2/imgproc.hpp>
#include <opencv2/imgproc/types_c.h>

#define TAG    "NativeLibSample"
#define AndroidLOGD(...)  __android_log_print(ANDROID_LOG_DEBUG, TAG, __VA_ARGS__)

extern "C"
JNIEXPORT void JNICALL
Java_com_bc_sample_OpenCVSample_createMat( JNIEnv* env, jclass thiz, jobject bitmap);

void bitmapToMat(JNIEnv *env, jobject bitmap, cv::Mat &dst);
Copy the code

Then, the corresponding JNI native-lib. CPP file code is improved as follows, here Mat is created in c++ code, and then the information of Mat is printed out:

#include "native-lib.h"

extern "C"
JNIEXPORT void JNICALL
Java_com_bc_sample_OpenCVSample_createMat( JNIEnv* env, jclass thiz, jobject bitmap) {
    cv::Mat rgbMat;
    // Convert the Java layer bitmap to Mat
    bitmapToMat(env, bitmap, rgbMat);
    // log Displays Mat information
    AndroidLOGD("native mat channels:%d, cols:%d, rows:%d", rgbMat.channels(), rgbMat.cols, rgbMat.rows);
}

void bitmapToMat(JNIEnv *env, jobject bitmap, cv::Mat &dst) {
    AndroidBitmapInfo info;
    void *pixels = 0;
    try {
        CV_Assert(AndroidBitmap_getInfo(env, bitmap, &info) >= 0);
        CV_Assert(info.format == ANDROID_BITMAP_FORMAT_RGBA_8888 ||
                  info.format == ANDROID_BITMAP_FORMAT_RGB_565);
        CV_Assert(AndroidBitmap_lockPixels(env, bitmap, &pixels) >= 0);
        CV_Assert(pixels);
        dst.create(info.height, info.width, CV_8UC4);
        if (info.format == ANDROID_BITMAP_FORMAT_RGBA_8888) {
            cv::Mat tmp(info.height, info.width, CV_8UC4, pixels);
            tmp.copyTo(dst);
        } else {
            cv::Mat tmp(info.height, info.width, CV_8UC2, pixels);
            cvtColor(tmp, dst, CV_BGR5652RGBA);
        }
        AndroidBitmap_unlockPixels(env, bitmap);
        return;
    }catch(...). {AndroidBitmap_unlockPixels(env, bitmap);
        jclass je = env->FindClass("java/lang/Exception");
        env->ThrowNew(je, "Unknown exception in JNI code {nBitmapToMat}");
        return; }}Copy the code

3.3.4 Running the program

Finally, the JNI method you just created can be called from your code as follows:

Bitmap bitmap = BitmapFactory.decodeResource(this.getResources(), R.drawable.adventure_time);
OpenCVSample.createMat(bitmap);
Copy the code

If the running result is as follows, the Mat is created successfully:

D/NativeLibSample: native mat channels:4, cols:3000, rows:4500
Copy the code

3.4.5 other

If you encounter the problem of duplicate dynamic libraries during the packaging process, you can add the following configuration to the app module build.gradle to solve the problem:

android {
    packagingOptions {
        pickFirst 'lib/arm64-v8a/libc++_shared.so'
        pickFirst 'lib/armeabi-v7a/libc++_shared.so'
        pickFirst 'lib/arm64-v8a/libopencv_java4.so'
        pickFirst 'lib/armeabi-v7a/libopencv_java4.so'}}Copy the code

The End

Please follow me to unlock more skills: BC’s Gold Digger homepage ~ 💐 BC’s CSDN homepage ~ 💐💐

OpenCV official website: opencv.org/releases/ OpenCV official documentation: docs.opencv.org/4.5.5/ OpenCV github: github.com/opencv/open… OpenCV start document: c.biancheng.net/opencv/ OpenCV start document LearnOpenCV learn about cutting OpenCV