I came into contact with PaddlePaddle and PaddleDetection tools for a long time. I was attracted by their simplicity and ease of use. At the same time, these tools greatly reduce the threshold and time required for training models, which are very suitable for beginners. It also has good performance in many practical application scenarios.
In terms of end-to-end deployment, Paddle Lite is the reasoning engine for the deployment of high-performance and lightweight END-TO-END AI applications in the flying-paddle stack, giving more possibilities to scenarios such as mobile. This engine allows us to make lightweight and efficient predictions across a wide range of hardware platforms, with a relatively short time to make a prediction and a low computing resource requirement.
Flutter is a good choice if we want to develop an App that can make predictions locally and have a consistent experience on Android and iOS. As an open source mobile UI framework, it has become a major trend in cross-platform mobile development. It can retain state for hot reloading during development, and has many stunning components and beautiful animations built in, while ensuring the same performance as native apps. Because of this, many companies have started to migrate their applications to Flutter. Many of our most familiar apps have actually been developed based on Flutter. If you’re already familiar with android native development, try it.
This time we have developed a real-time target detection application based on Flutter, which also benefits from Flutter’s ability to access native system features and SDK on iOS and Android.
Our project’s GitHub address:
Github.com/KernelErr/r…
It has a built-in fruit recognition model, which can be downloaded and directly compiled to experience. Convention:
- The Flutter end: the home directory of the Flutter project.
- Android: The Android subdirectory of the project, native Android.
The development environment
Our development environment is as follows:
- Flutter version 1.12.13 + hotfix. 8
- The Dart version 2.7.0
- Android Studio (version 3.6)
- Develop Toolchain for Android Devices (Android SDK Version 29.0.3)
Flutter has been updated with more and more new features. Some of the implementation methods of Flutter on the Internet based on the old version are no longer practical and we need to make some changes.
Prepare model
Paddle Lite needs to generate the supported lightweight models using the OPT tool. If you already have models trained by PaddleDetection, you need to export the models in PaddleDetection and convert them using the OPT tool.
If you have models trained by other frameworks, such as Caffe, TensorFlow, ONNX, etc., you can use X2Paddle to convert.
Suppose we’ve got two files:
- Model.nb – Based on Yolov3 Tiny training and optimized by OPT
- Label – Model prediction one – to – one – corresponding label
How do I support Paddle Lite in Flutter?
We just need to create a new Flutter project via Android Studio. Here we assume the name is realtime_OD.
Prepare the Paddle Lite prediction library and model files
Since we are using Android native code, we need to develop on the Android side, not the Flutter side. We downloaded the required precompiled libraries from the Precompiled prediction library provided by Paddle Lite and placed them in the appropriate folder on Android, similar to the directory on native Android. Then we continue to place the model files in the Android folder, create assets folder under realtime_od/android/app/ SRC /main/, and put the models and labels in the Models and labels subfolders respectively. Your directory should look like this:
We use the mask model as an example, the model position is: Models /mask/model.nb and the label position is: Labels /mask_label_list. So you need to assign a value to the MainActivity:
Disable compression
During APK generation, our model will be compressed, so we need to modify the build.gradle configuration file to disable the compression of assets folder.
Native Android support
It would be too much work to write a Dart call code for Paddle Lite for Flutter support, so we might as well just modify it based on the official Demo. On the Android side, we directly use the code in the official Demo and register the Channel in the MainActivity.
Since a process safety mechanism has been added to Flutter, we use a MethodResultWrapper to ensure that the main process returns a result. In new Flutter you need to use configureFlutterEngine instead of onCreate to register components.
Using live images
Let’s provide Flutter with live video from its webcam! Dart already has a number of packages available for Flutter:
Also, make sure that the minimum Android SDK version of the project is at least 21. In official provide Demo, image input is using a Bitmap image, but we got from the plug-in format is android. The graphics. ImageFormat. YUV_420_888, in most below we (Predictor for the corresponding transformation, The use of the conversion code has been authorized by contacting the original author. We used RenderScript to do efficient calculations and avoid too much latency.
Display real-time images and annotate them
A lot of work has been done on the Android side. Let’s do some work on the Flutter side.
Dart and object_Detector. dart you can see that we called the methods provided on the Android side, loadModel and detectObject. At the same time in the DrawObjects class to provide the function of marking the target, the code is relatively simple, using the predicted results for drawing.
Let’s run it!
The mask model provided by qunyou is used here. There are only two lines in the label file, which are wearing mask and not wearing mask respectively. We tested the PaddlePaddle official sample image on an Android 9 device.
It can be seen from the log that the time Paddle Lite predicts is close to 700 ms.
Change the model and optimize the solution
How to use other models
We’ve adapted YOLO V3 by referring to groupmate’s solution (see link), with major changes in Predictor’s model input and MainActivity initialization. Because other models are used officially, the Shape input is different from ours. Ours is 320.
If you need to use another model, modify the input at the same time:
And at the output:
Dart also needs to be modified in the callout function:
How much faster
In fact, our model is not fast enough, and with the right model, the prediction time can be shortened even further. Depending on your needs, Paddle Lite supports a number of mainstream models that you can choose from.
Trouble Shooting
Problems recorded include those encountered during Flutter development and those encountered during Paddle Lite use:
1. Methods marked with @UiThread must be executed on the main thread.
This is because Flutter introduces process security and cannot return result directly in the child process. Instead, result should be returned in the main process. There are many solutions available online, and ours is also from GitHub.
Error: Incompatible type: MainActivity cannot be converted to FlutterEngine
Chances are you’re reading an old version of the tutorial, so please refer to the official documentation for native Android. We specified V2 during native Android development.
3. A library error occurred in Paddle Lite
At first I thought it was an official problem, but I can solve it by compiling the library manually. I’ve built in a problem-free library for ARM64.
4. Other issues
Official document -> GitHub -> Search engine -> Paddle Lite official QQ Group (696965088), generally can be resolved. 🙂
Refer to the link
Paddle Lite Github address:
Github.com/PaddlePaddl…
Chinese version of Paddle Lite
Paddle – lite. Readthedocs. IO/useful/latest/I…
Paddle Lite Deployment:
www.ralphlu.top/article/202…
Real-time Object Detection with Flutter, TensorFlow Lite and Yolo -Part 1:
Blog. Francium. Tech/real – time – o…
Writing Custom platform-specific code:
Flutter. Dev/docs/develo…
PaddleDetection address:
Github.com/PaddlePaddl…
Official website address:
www.paddlepaddle.org.cn
Open source framework project address:
Making: github.com/PaddlePaddl…
Gitee: gitee.com/paddlepaddl…