The author | Li Tianyi
Abstract: HiLens Kit has been basically developed and can be deployed to HiLens Kit. The model is YOLOv3 model based on DarkNet53, the weight is data set based on COCO2014 training, and the detection of lane lines is realized based on the traditional method of OpenCV. It can be jointly developed and deployed through ModelArts AI Gallery and HiLens Kit end-cloud of the whole process.
Click on the portal to see the final video effect →→(PS: Please ignore the background music)!
Main process introduction:
- (Optional, ignore can also, depending on the quality of the camera, for large camera distortion need to calculate the distortion matrix and distortion coefficient of the camera, to correct the picture) picture correction;
- The region of interest is intercepted and only the image region containing lane line information is processed.
- Apply perspective transform to the region of interest;
- For lane lines with different colors, lane lines under different lighting conditions and lane lines with different definitions, different gradient thresholds are used according to different color Spaces, and different color thresholds are processed. The binary graph of lane lines is obtained by merging each processing method.
- Extract pixels belonging to lane lines in binary graph;
- Histogram statistics of the pixels of binary images were carried out, and the peak points on the left and right sides were counted as the starting point coordinates of the left and right lane lines for curve fitting.
- Quadratic polynomial was used to fit the pixels of left and right lane lines respectively (for pixels with large noise, filtering can be carried out, or random sampling consistency algorithm can be used for curve fitting).
- Calculate lane curvature and deviation position of vehicle relative to lane center;
- Effect display (feasible domain display, curvature and position display).
- Detect the status of other vehicles on the road during driving, display the vehicle category and confidence, and detect vehicles through YOLOv3, and then perform perspective transformation between the coordinates of the returned vehicle detection frame and the current coordinates to obtain the approximate distance as the distance dis between vehicles.
Description:
The main framework of this Demo has been basically developed based on HiLens Kit. The model is YOLOv3 model based on DarkNet53, the weight is data set based on COCO2014 training, and the detection of lane lines is realized based on the traditional method of OpenCV, so the calculation is large and the overall speed is slow.
About Deployment and Training — End Cloud Collaboration based on ModelArts and HiLens Kit:
1) Considering the deployment, we have tested the deployment to HiLens Kit. By the way, we can also run simulation tests in HiLens Stuido, especially in HiLens Studio, which is more convenient and does not require hardware support. You only need to install the Pillow library on HiLens Kit or HiLens Studio. How to install the third-party library on HiLens Kit and HiLens Studio is very simple, please refer to:
Kit: in HiLens bbs.huaweicloud.com/forum/threa…
In HiLens Studio: bbs.huaweicloud.com/forum/threa…
2) if you want to practice or optimization, by no hardware (GPU), it is recommended to use ModelArts, one-stop development, seamlessly to HiLens Kit oh, introductions of ModelArts may refer to: www.huaweicloud.com/product/mod…
Here are three algorithms that can be deployed to HiLens Kit inference. In the latest AI Gallery (the original AI market), there are algorithms, models, data sets, etc. It is very rich, you can explore it yourself. At the same time, you can also share your algorithm to other developers, developers can subscribe to create training use, very convenient, compared to GitHub, not only provides source code, but also provides hardware resources for training, powerful Tesla V100 32GB version oh.
(1) YOLOv3_Darknet53, YOLOv3_Darknet53, is a classic target detection network, YOLOv4, YOLOv5, AI Gallery also released YOLOv5, but the PyTorch framework does not support modal conversion. So I gave up for the time being.
About YOLOv3_Darknet53 using method, the algorithm is interface is introduced in detail, here is not here, you can refer to use, pay attention to the part model transformation, please refer to this blog post model transformation part: bbs.huaweicloud.com/blogs/19987…
Algorithm referral link for: marketplace.huaweicloud.com/markets/aih…
(2) YOLOv3_Resnet18(GPU), if we just want to do vehicle detection or to simplify the model and improve the speed, we can choose YOLOv3 whose trunk network is Resnet18. The network is lighter and the speed will be faster.
To the same link: marketplace.huaweicloud.com/markets/aih…
(3) YOLOv3_Resnet18(Ascend 910) is not much different from the previous version (2). It is mainly changed from GPU to Ascend 910 training.
Links to: marketplace.huaweicloud.com/markets/aih…
Ok, the algorithm part is solved, what about the data? AI Gallery of ModelArts has also thought of the data set, providing a free subscription, download to your own OBS import can be used (note that OBS costs a certain amount of money), here I also provide open source data set 5164 photos of people and cars, for everyone to use oh, on the link: Marketplace.huaweicloud.com/markets/aih…
Now that you have the data sets and algorithms, it’s time to train. It can be done very well on ModelArts, and very quickly with the powerful Tesla V100 32GB or Ascend 910.
The above part of the overall operation process can be completed by referring to the following blog post:Bbs.huaweicloud.com/blogs/19987…
Operation effect:
It takes about 300ms to complete the execution of a frame as shown in the video, i.e. an FPS of about 3, which is not slow and needs to be optimized. I think there is a lot of room for improvement, such as:
(1) the pre-processing and post-processing is very time-consuming, real reasoning is very fast, can accelerate from this aspect, do not know whether to open multithreading to improve, there should be;
(2) should be able to better play the dedicated hardware, remember there seems to be DVPP and AIPP, can be used to do the operation of gamut conversion;
(3) The post-processing analysis of YOLOv3 is more troublesome. The COCO 80 class is used here, but the person and CAR classes can be used only, or a little more. At the same time, NMS takes a lot of time.
(4) Python is used here. If C++ is used, the execution efficiency will be much higher.
(5) The speed improvement should be optimized according to the performance of the hardware, and the time-consuming part should be analyzed to maximize the performance of the hardware. It needs to be optimized in a targeted and customized way. I personally have a very shallow understanding, and I hope to have the guidance of students familiar with it.
Remark:
- The model and code used in this Demo are all from the open source code. It seems to be a project involved in one of Mr. Ng’s courses, but I can’t find the source. If you know, please reply below for your advice, thank you.
- The accuracy of the Demo model and running speed is not guaranteed, only for learning exchange use.
- For other matters not covered, please advise us.
- The code will be open source, so stay tuned. Thanks.
Share this article from huawei cloud community “Demo share | when autopilot ModelArts, ModelArts AI Gallery and HiLens Kit development”, the original author: Tianyi_Li.
Click to follow, the first time to learn about Huawei cloud fresh technology ~