Introduction: In the past year, Ali Sports technology team has been continuously exploring the aspect of end intelligence, especially realizing the implementation of practice and business empowerment in the sports health scenario, which is the AI sports project. AI sports projects practice the concept of sports digitalization, provide important support for the upturn of the sports population, take the first step in the intelligent sports field of Ali sports terminal, and bring more interesting and novel gameplay for users. Since its launch, the project has attracted extensive attention.

The author | the | acoustic source ali technology to the public

A background

In the past year, Ali Sports technology team has continuously explored the aspect of end intelligence. In particular, it has realized practical implementation and business empowerment in the sports health scenario, which is AI sports project. AI sports projects practice the concept of sports digitalization, provide important support for the upturn of the sports population, take the first step in the intelligent sports field of Ali sports terminal, and bring more interesting and novel gameplay for users. Since its launch, the project has attracted extensive attention.

In 2020, sports at home are becoming a new trend as traditional offline activities are restricted due to the COVID-19 pandemic. Based on alibaba’s strong technology precipitation, Ali Sports team responds to the urgent needs of online sports and develops intelligent sports based on AI recognition, providing users with a simple and fun new way of home sports. All you need is a mobile phone and a space of 3-4 square meters. When exercising, users open le Dynamics APP, fix the phone on the side of the field, set the Angle of the phone appropriately, adjust the distance between the body and the phone according to the automatic voice prompt of the application, until the body is completely located in the recognition frame, then they can start exercising.

Two – end intelligence practice

After one year’s exploration and improvement, Ali Sports has established a systematic client sports intelligence system, from the verification DEMO to the AI sports intelligence platform that includes a variety of actions and supports ability transfer. End intelligent motion system based on the depth of the ali inference engine in the mobile terminal to reasoning, identify human body posture and movement, the analysis of the human body posture, movement, and movement Angle, gives the real-time feedback and action to correct, through capacity modular combination, is now supporting a dozen movement action and dozens of gameplay, realize the organic integration of sports and AI, Let the user’s online sports become easy and full of fun.

3. Technical Support

The basic technical idea of intelligent terminal motion is to use MNN inference engine for inference and attitude recognition. namely

  • Real-time detection of human contours in images and videos, positioning of 14 key bone points, including head, shoulder, foot and other key joint sites.
  • Based on these key points of information, connecting points into lines, lines to form action, can analyze human posture, action Angle and movement trajectory.
  • Through action posture matching, the user movement is detected to realize the timing and counting of the action. At the same time, real-time detection and analysis of the degree of standardization of actions, give state feedback, correct user actions, realize interaction, improve interactive experience.

In traditional sports mode, users can get real-time reminders and help from on-site auxiliary personnel (coaches, examiners or relatives and friends) in time. In the terminal intelligent motion mode, users can only interact with mobile apps when doing actions. Interaction ability and recognition level are affected by a series of factors such as reasoning model ability, complexity of motion scene and recognition algorithm of motion matching. In the process of exploring and landing intelligent motion ability of terminal, some new problems or difficulties will be encountered, such as man-machine orientation matching, bone point recognition missing point, point error recognition, two-dimensional distortion, user movement, mobile phone shaking, scene noise and so on. These issues are not repeated, but only a few representative issues are selected to share:

  • It is the basis of intelligent motion ability to judge the effectiveness of motion and design the key algorithm to improve the accuracy of motion matching.
  • On the premise of ensuring the identification effect, effective measures should be taken to reduce resource consumption of mobile terminals to improve user experience, mainly in the form of power consumption and heat consumption.
  • Adopt a more flexible approach to reduce the manpower and time consumption of mobile testing, improve the efficiency of development and testing, and provide strong support for team delivery assurance.

Improve identification accuracy

The most intuitive and basic feeling that intelligent movement brings to users is the accuracy of movement counting. If the action recognition and counting is not correct, the enthusiasm of users to use the APP will be diminished and the participation will not be high. To this end, we must first solve the problem of accurate counting.

The basic principle of intelligent motion counting is to decompose a complete action into several small steps, and then trigger recognition and judgment for each step. After traversing all the steps, the validity of the whole action is confirmed. If valid, the count increases by 1; If not, repeat the process. In short, intelligent motion recognition and counting is a state machine. A motion action is discretized and abstracted into N state machines, {S (0), S (1), S (2)… ,s(n-1)}, the state machine detects in a certain sequence, all detection means that the user has completed the action, add 1 to the count; If a state is not detected, the corresponding feedback information is triggered to reset the state machine and enter a new cycle. Each state machine corresponds to a certain trigger condition, and an action matching result can be obtained through the cyclic matching detection of real-time bone point coordinate and state.

It is not difficult to see that the motion recognition accuracy is closely related to the motion matching algorithm, and the better the algorithm matching effect, the higher the recognition accuracy. In order to improve the accuracy of action recognition, factors affecting the matching algorithm can be selected as the entry point and breakthrough point, bone point, state machine, matching, etc. Corresponding solutions are as follows:

  • Improve bone point stability and ensure the accuracy of state matching results.
  • Select the motion with stable, recognizable and representative bone points as the state machine.
  • The frame rate should cover all state machines for an action.

Examples are given below.

Bone point recognition accuracy has an important influence on motion matching. As shown in the following figure, the test subject has an error in identifying bone points on his left arm. If you go straight to the match, you will obviously get the wrong result. In view of this situation, we should make good use of the user’s historical action information to correct the action matching algorithm.

In another case, the user has completed all actions of a certain action, such as the jacks in the figure below. Due to the low sampling frame rate, it is impossible to capture and identify all poses in the process of all jacks, resulting in the failure of matching of a certain state and the final matching error of jacks. For low frame rate problems, you can start with both the model and the input source. For the model, the simplified model is adopted to reduce the inference time without affecting the accuracy of action recognition. For different terminal devices, input sources with different resolutions are used to reduce the time consuming of original data processing.

Reduced performance consumption

Affected by physical conditions, the computing power and storage space of mobile phones are limited. In addition, deep learning reasoning itself contains a large number of operations and consumes large resources. If deep learning reasoning is carried out directly on the terminal and the resource consumption of the mobile terminal’s own services (such as camera, video recording and animation effects) is taken into account, the CPU and memory cost will increase significantly, which is intuitively reflected in the obvious heating of the mobile phone and the rapid power consumption. When smart motion is implemented on end intelligence, special consideration should be given to reducing performance consumption, which is crucial for improving user experience.

To reduce the overall performance consumption, we should trace back to the source and start from reducing the consumption of single frames. Single frame processing can be divided into three stages: pre-inference, inference and inference.

These three stages play different roles. In the pre-reasoning stage, format conversion is mainly completed, and the stream data obtained by the camera is converted to the data format required for reasoning, such as YUV format and RGBA format. The inference stage mainly completes the calculation and output of bone point coordinates. On the input frame data, through the inference engine, execute a series of algorithms, output inference results, such as attitude recognition is the RGBA data of the input picture into bone point coordinate data. In the post-reasoning stage, display is mainly completed, rendering operation and business related operations, such as UI display and animation effect display.

Accordingly, the above three stages can be optimized respectively. Among them, the optimization in the reasoning process is responsible for MNN, which will not be discussed here. For the data conversion in the pre-reasoning stage, unnecessary intermediate conversion links should be reduced, and the camera stream data should be directly converted into the required format. If reasoning uses RGBA raw data, convert the camera stream data directly to RGBA format. For the post-reasoning stage, the appropriate rendering scheme should be selected according to the bearing platform to reduce the rendering consumption. For iOS platforms, Metal can be directly used for rendering improvement.

Improve test efficiency

AI intelligent movement is a bold attempt of Ali Sports team in sports digitization. In application development, especially in the testing process, we invest considerable manpower, equipment and time to constantly improve application functions, optimize application performance and improve user experience. In addition, the effect test of AI motion recognition is greatly affected by environmental factors, such as light, background, distance, and the size of the image of the figure in the camera. This puts the testing method to the test.

Take the traditional testing scheme as an example: it is usually a real person, field and real-time action, and the tester records the results manually and then analyzes them afterwards, as shown in the figure below.

It is not hard to imagine that the AI intelligent motion run by the mobile phone has a different brand, model, system version and performance parameters, AI intelligent motion of the use of the user may be in different environment, if adopts the traditional test method, test coverage on different factors, the testers, test time, put forward the challenge, and consistent with the test accuracy is not certain. Specific reasons are as follows:

  • High labor cost: a test requires the cooperation of more than one student, time-consuming and exhausting.
  • Single test environment: unable to cope with complex and diverse online environments.
  • The test results are difficult to quantify. The accuracy of the model, efficiency of the algorithm, dynamic matching accuracy, accuracy improvement and performance consumption can not be quantitatively evaluated.
  • The problem is difficult to locate. After analysis and investigation, the problem of online customer complaints cannot be repeated.

Traditional testing methods cannot be sustained. In order to overcome the above difficulties, ali Sports Technical team developed a set of AI motion automatic testing tools, which are specially used to solve AI intelligent project testing problems, realize fast positioning and regression of online problems, and achieve quantitative evaluation of the accuracy of model algorithm.

The solution of automatic test tool is: batch parsing video sets, simulating real scenes, obtaining bone point data, testing service results, and automatically generating test reports. The specific technical scheme is shown in the figure below:

After adopting the new testing tool, the labor cost is significantly reduced and the testing efficiency is improved. The specific test results are as follows:

It should be noted that the effect of the test tool is related to the number of test samples. The richer the samples, the better the test accuracy.

Iv Business Results

Ali Sports Intelligent Sports now supports dozens of sports movements, has developed rich AI training courses, and at the same time, through the modular combination of sports ability, supports the continuous expansion of new movements in the future.

Since the birth of AI intelligent motion, le dynamic APP launched in succession straight arm jumping jacks, push-ups and other upper limb movements, hip bridge, lower limb movements such as squats, and rope skipping, jumping jacks, and other forms of body movement and so on the many kinds of sports, makes the users can not restricted by time and place, with friends anytime and anywhere to participate in the AI movement, raised the APP users of attractive and interesting. In addition, the AI training course innovatively introduces star resources and promotes the continuous “star training partner” class for 52 weeks, 7 days a week, with stars to drive users to develop sports habits, happy sports, and love sports. Ali Sports team will also continue to create more sports games according to user needs, enrich product functions, and form ali Sports intelligent unique business brand and innovative product features.

The original link

This article is the original content of Aliyun and shall not be reproduced without permission.