Make a New Year gesture red envelope, from now on red envelope will not be white whoring "can experience" (3)

PK creative Spring Festival, I am participating in the “Spring Festival creative submission contest”, please see: Spring Festival creative submission Contest

The effect

experience	Send page	Get the page	New Year greeting posture recognition page

First, give red envelopes

This article addresses

Two, get a red envelope

This article addresses

Three, New Year gesture to get red envelopes

Need to make New Year’s greetings to the camera action can receive

Train of thought
- Use wechat # CameraFrameListener to capture every frame of the camera
- Send to the back end
- The back end sends to the human key point failure service and returns to the key point after failure
- More key point to judge whether it is a New Year greeting posture
- If yes, grab the red envelope
- If not, continue to identify

code

Small program

<view> Camera <view style="">
            <camera device-position="front" flash="off" frame-size="small" @error="error" @initdone="xiangji"
                    style="width: 100%; height: 100vh;"></camera></view> Indicate whether the posture is correct <view v-show="gesture_error" class="background_image"
            style="z-index: 999; margin-top: 170rpx; width: 530rpx; margin-left: 110rpx;">
            <u-alert type="warning" :show-icon="true" :title="gesture_title" :center="true"></u-alert>< / view > border < imageclass="background_image" src="https://img.yeting.wang/new_year/7.png"></image> Pop box after receiving red envelope < U-modal :show="res_lingqushow" :confirmText="redPacketReceive.buttonContext"
            @confirm="resBut(redPacketReceive.buttonMethod)">
            <view class="slot-content">
                    {{redPacketReceive.message}}
            </view>
    </u-modal>
</view>

xiangji() {
    this.listener = this.ctx.onCameraFrame((frame) = > {
        // To avoid going too fast, decide whether to deal with it
        if (this.flag == true) {
            this.flag = false
            console.log(frame.data instanceof ArrayBuffer, frame.width, frame.height)
            / / the images change my into [[r, g, b, a], [r, g, b, a], [r, g, b, a]], to the python into images
            // The front-end UPNG can also be converted to pictures, but the real machine is very slow
            let data = new Uint8ClampedArray(frame.data);
            let list = new Array
            for (var i = 0; i < data.length; i += 4) {
                    list.push([data[i], data[i + 1], data[i + 2], data[i + 3]])}// Send to the server
            uni.$u.http.post('/redPacket/infer', {
                    imgArray: list,
                    width: frame.width,
                    height: frame.height,
                    redPacketUserId: this.redPacketUserId,
                    redPacketId: this.redPacketId,
            }).then(data= > {
                    console.log(data)
                   // Determine if the posture fails
                    if (data.gestureStatus) {
                            // Stop monitoring. A result dialog box is displayed
                            this.listener.stop()
                            this.gesture_error = false
                            this.redPacketReceive = data
                            this.res_lingqushow = true
                    } else {
                            // Do not identify the top prompt, continue to identify
                            this.flag = true
                            this.gesture_error = true
                            this.gesture_title = data.message
                            console.log("New Year greeting posture not recognized.");
                    }
            }).catch(err= > {
                    console.log("err:" + err)
                    setTimeout(() = > {
                            this.flag = true
                    }, 3000); })}})this.listener.start({
            success: function(res) {
                    console.log("Start listening."); }}); },Copy the code

The back-end

// The code is messy, but it works
@Override
publicResult<? > infer(UserBo userBo, RedPacketInferVo redPacketInferVo) {long s = System.currentTimeMillis();
    // Call the identifying end
    String post = HttpUtil.post(inferUrl + "/infer/array_buffer", JSONUtil.toJsonStr(redPacketInferVo));
    long e = System.currentTimeMillis();
    System.out.println("Use time:" + (e - s));
    // Convert the result
    JSONObject jsonObject = JSONUtil.parseObj(post);
    if (new Integer(20000).equals(jsonObject.getInt("code"))) {
        // Data can return more than one person
        JSONArray dataArray = jsonObject.getJSONArray("data");
        if(dataArray ! =null && dataArray.size() > 0) {
            // Take your shoulders, elbows and wrists
            JSONObject data = dataArray.getJSONObject(0);
            JSONObject left_shoulder = data.getJSONObject("left_shoulder");
            Double left_shoulder_x = left_shoulder.getDouble("x");
            Double left_shoulder_y = left_shoulder.getDouble("y");
            JSONObject right_shoulder = data.getJSONObject("right_shoulder");
            Double right_shoulder_x = right_shoulder.getDouble("x");
            Double right_shoulder_y = right_shoulder.getDouble("y");
            JSONObject left_elbow = data.getJSONObject("left_elbow");
            Double left_elbow_x = left_elbow.getDouble("x");
            Double left_elbow_y = left_elbow.getDouble("y");
            JSONObject right_elbow = data.getJSONObject("right_elbow");
            Double right_elbow_x = right_elbow.getDouble("x");
            Double right_elbow_y = right_elbow.getDouble("y");
            JSONObject left_wrist = data.getJSONObject("left_wrist");
            Double left_wrist_x = left_wrist.getDouble("x");
            Double left_wrist_y = left_wrist.getDouble("y");
            JSONObject right_wrist = data.getJSONObject("right_wrist");
            Double right_wrist_x = right_wrist.getDouble("x");
            Double right_wrist_y = right_wrist.getDouble("y");
            // Read posture
            if (Math.abs(left_shoulder_y - right_shoulder_y) < 20) {
                if (Math.abs(left_shoulder_x - right_shoulder_x) > 70) {
                    if (Math.abs(left_wrist_x - right_wrist_x) < 70) {
                        if (Math.abs(left_wrist_y - right_wrist_y) < 20) {
                            if (right_elbow_y > right_shoulder_y || left_elbow_y > left_shoulder_y) {
                                if (right_wrist_y < right_elbow_y || left_wrist_y < left_elbow_y) {
                                    if ((right_wrist_x > right_elbow_x && right_wrist_x > right_shoulder_x)
                                            || (left_wrist_x < left_elbow_x && left_wrist_x < left_shoulder_x)) {
                                        System.out.println("Good posture.");
                                        // Call the red envelope interface if the gesture is correct
                                        return receive(userBo, new RedPacketVo()
                                                .setRedPacketId(redPacketInferVo.getRedPacketId())
                                                .setUserId(redPacketInferVo.getRedPacketUserId())
                                        );
                                    } else {
                                        System.out.println("My hands are not in the middle."); }}else {
                                    System.out.println("Hands over elbows."); }}else {
                                System.out.println("Elbows less than shoulders."); }}else {
                            System.out.println("Abnormal wrist height."); }}else {
                        System.out.println("Abnormal wrist width."); }}else {
                    System.out.println("Abnormal shoulder width."); }}else {
                System.out.println("Abnormal shoulder height."); }}}return Result.success(new RedPacketReceiveDto()
            .setRedPacketId(redPacketInferVo.getRedPacketId())
            .setGestureStatus(false)
            .setStatus(false)
            .setMessage("New Year greeting posture not recognized.")); }Copy the code

The human body recognition

import json
import time

from django.http import HttpResponse

# Create your views here.
from django.views.decorators.csrf import csrf_exempt
import det_keypoint_unite_infer2 as keypoint

# Base64 Image recognition
@csrf_exempt
def base64(request) :
    res = {}
    if request.method == 'GET':
        res['message'] = keypoint.main()
        return HttpResponse(json.dumps(res))
    else:
        json_str = request.body
        body = json.loads(json_str)
        img = body.get("img")
        data = keypoint.base64(img)
        res['message'] = 'success'
        res['code'] = 20000
        res['data'] = data
        return HttpResponse(json.dumps(res))

# [[r,g,b,a],[r,g,b,a],[r,g,b,a]
@csrf_exempt
def array_buffer(request) :
    res = {}
    if request.method == 'GET':
        res['message'] = keypoint.main()
        return HttpResponse(json.dumps(res))
    else:
        print(time.time())
        json_str = request.body
        body = json.loads(json_str)
        img_array = body['imgArray']
        width = body['width']
        height = body['height']
        data = keypoint.array_buffer(img_array, width, height)
        print(time.time())
        res['message'] = 'success'
        res['code'] = 20000
        res['data'] = data
        return HttpResponse(json.dumps(res))
Copy the code

# Prediction with given data
def predict_with_given_det(image, det_res, keypoint_detector, keypoint_batch_size, det_threshold, keypoint_threshold, run_benchmark) :
    rec_images, records, det_rects = keypoint_detector.get_person_from_rect(
        image, det_res, det_threshold)
    keypoint_vector = []
    score_vector = []
    rect_vector = det_rects
    batch_loop_cnt = math.ceil(float(len(rec_images)) / keypoint_batch_size)

    for i in range(batch_loop_cnt):
        start_index = i * keypoint_batch_size
        end_index = min((i + 1) * keypoint_batch_size, len(rec_images))
        batch_images = rec_images[start_index:end_index]
        batch_records = np.array(records[start_index:end_index])
        if run_benchmark:
            # warmup
            keypoint_result = keypoint_detector.predict(
                batch_images, keypoint_threshold, repeats=10, add_timer=False)
            # run benchmark
            keypoint_result = keypoint_detector.predict(
                batch_images, keypoint_threshold, repeats=10, add_timer=True)
        else:
            keypoint_result = keypoint_detector.predict(batch_images,
                                                        keypoint_threshold)
        orgkeypoints, scores = translate_to_ori_images(keypoint_result,
                                                       batch_records)
        keypoint_vector.append(orgkeypoints)
        score_vector.append(scores)

    keypoint_res = {}
    keypoint_res['keypoint'] = [
        np.vstack(keypoint_vector).tolist(), np.vstack(score_vector).tolist()
    ] if len(keypoint_vector) > 0 else [[], []]
    keypoint_res['bbox'] = rect_vector
    return keypoint_res


# Top-down joint forecasting
def topdown_unite_predict(detector,
                          topdown_keypoint_detector,
                          image_list,
                          keypoint_batch_size=1,
                          save_res=False) :
    det_timer = detector.get_timer()
    for i, img_file in enumerate(image_list):
        det_timer.preprocess_time_s.start()
        image, _ = img_file
        det_timer.preprocess_time_s.end()
        # Predict pedestrians
        print("Pedestrian start time: {}".format(time.time()))
        results = detector.predict([image])

        print("Pedestrian end time: {}".format(time.time()))
        # Determine if pedestrians are predicted
        if results['boxes_num'] = =0:
            continue
        # Forecast key points
        print("Key point start time: {}".format(time.time()))
        keypoint_res = predict_with_given_det(
            image, results, topdown_keypoint_detector, keypoint_batch_size, 0.5.0.5.False)
        print("Key end time: {}".format(time.time()))
        # draw_pose(img_file, keypoint_res)
        skeletons, scores = keypoint_res['keypoint']
        skeletons = np.array(skeletons)
        kpt_nums = 17
        if len(skeletons) > 0:
            kpt_nums = skeletons.shape[1]
        if kpt_nums == 17:  # plot coco keypoint
            EDGES = [(0.1), (0.2), (1.3), (2.4), (3.5), (4.6), (5.7), (6.8),
                     (7.9), (8.10), (5.11), (6.12), (11.13), (12.14),
                     (13.15), (14.16), (11.12)]
        else:  # plot mpii keypoint
            EDGES = [(0.1), (1.2), (3.4), (4.5), (2.6), (3.6), (6.7), (7.8),
                     (8.9), (10.11), (11.12), (13.14), (14.15), (8.12),
                     (8.13)]
        NUM_EDGES = len(EDGES)
        # (int(np.mean(skeletons[j][i, 0])), int(np.mean(skeletons[j][i, 1])))
        res = []
        for j in range(len(skeletons)):
            res.append({})
        skeleton_index = {
            0: 'nose'.1: 'left_eye'.2: 'right_eye'.3: 'left_ear'.4: 'right_ear'.5: 'left_shoulder'.6: 'right_shoulder'.7: 'left_elbow'.8: 'right_elbow'.9: 'left_wrist'.10: 'right_wrist'.11: 'left_crotch'.12: 'right_crotch'.13: 'left_knee'.14: 'right_knee'.15: 'left_ankle'.16: 'right_ankle'
        }
        # I is the bone and j is the person
        for i in range(NUM_EDGES):
            for j in range(len(skeletons)):
                skeleton_res = res[j]
                skeleton_res[skeleton_index[i]] = {
                    'x': skeletons[j][i, 0].'y': skeletons[j][i, 1]}return res

# Task identification model path
det_model_dir = ' '
keypoint_model_dir = ' '
Failed type OF GPU CPU
device = 'GPU'
# Pedestrian detection model
# Take the pedestrian detection model path
pred_config = PredictConfig(det_model_dir)
# Detector type: Lightweight detection model PicoDet
detector_func = 'Detector'
if pred_config.arch == 'PicoDet':
    detector_func = 'DetectorPicoDet'
# Load model
detector = eval(detector_func)(pred_config,
                               det_model_dir,
                               device=device)
# Key point detection model
# Select key points to detect model path
pred_config = PredictConfig_KeyPoint(keypoint_model_dir)
assert KEYPOINT_SUPPORT_MODELS[
           pred_config.
               arch] == 'keypoint_topdown'.'Detection-Keypoint unite inference only supports topdown models.'
Load the keypoint detection model
topdown_keypoint_detector = KeyPoint_Detector(
    pred_config,
    keypoint_model_dir,
    device=device)

# predict base64
def base64(base64_image) :
    img_list = [decode_base64_image(base64_image, {})]
    # Top-down joint forecasting
    keypoint_res = topdown_unite_predict(detector, topdown_keypoint_detector, img_list)
    return keypoint_res

# predict array_buffer
def array_buffer(img_array, width, height) :
    img_list = [decode_array_buffer_image(img_array, width, height, {})]
    # Top-down joint forecasting
    keypoint_res = topdown_unite_predict(detector, topdown_keypoint_detector, img_list)
    return keypoint_res
Copy the code

conclusion

Technical points: red envelope cent algorithm, grab red envelope to prevent multiple, prevent multiple money, identify New Year gesture

Learning points: distributed lock, allocation of red envelope amount, small program arrat_buffer processing

mo4tech.com (Moment For Technology) is a global community with thousands techies from across the global hang out!Passionate technologists, be it gadget freaks, tech enthusiasts, coders, technopreneurs, or CIOs, you would find them all here.

Make a New Year gesture red envelope, from now on red envelope will not be white whoring “can experience” (3)

The effect

First, give red envelopes

Two, get a red envelope

Three, New Year gesture to get red envelopes

conclusion

Make a New Year gesture red envelope, from now on red envelope will not be white whoring “can experience” (3)

The effect

First, give red envelopes

Two, get a red envelope

Three, New Year gesture to get red envelopes

conclusion

Related Posts

Win10 environment quick installation superset construction and use

ArrayList source Analysis (2)

Spring Boot 2.X Combat –Spring Security (Token) login and registration