@[TOC]

preface

After staying up late to finish the project lab report, I Huterox was back. So today is actually a step up from the last blog post (Mediapipe). (No one really thought this blog was written to express their love to a girl, joking that it was all dry stuff!) So the same inside there are a lot of principles in today’s blog are written in that blog, how to play are very clear. So now all I have to do is wrap up the original code THAT I wrote last time, and add some functionality.

Demand effect

This is a direct look at the following effect to know

The project structure

Media resources

Here I want to say is that media resources, because this Demo is relatively simple I did not upload Github, also do not want to upload Baidu cloud disk. That’s it. That’s the pictureSo if you want to take a screenshot of it, it’s going to be 0,1,2,3,4 from top to bottom. JPG and notice that it’s JPG and 1280 x 720 and you can do that yourself

HandDetecter tools

Gestures to find

I’m going to talk a little bit about what each of these tools does and what they have. First of all, gesture search is the call operator

Check finger standing up

And then since we need to track our index finger, and our middle finger, we need to get those index fingers that stick up. So this check is also very simple, we said in the beginning of the blog has explained the principle, I will not repeat here. And the thing to illustrate here is the argument that we return.His logic is that the first thing is to check if your hand is on the camera. If so, it checks to see if the finger is standing up from thumb to pinkie, and then writes the coordinates of the current finger directly into the list. Note here that we only allow one hand to draw on one screen. Here you can see that the return is direct, so each time you get the latest hand that appears on the screen.

Flag is whether it’s up or not x is the x coordinate and y is the y coordinate

Check that a finger stands up

This is our CheckHandUp() return list, because we check the finger stand up in order, so we can know the stand up of each thumb directly from the index. And our order is the same whether we’re left or right.

Window

Display window

This is the window that shows us as a whole, which is this interface:Here are a fewHere is the call

UI Media Resources

So this is the UI resource that we talked about earlier.Windows PrintUI () is called.

Controller

Gestures rules

So this is what we do with our gestures, first index and middle finger together is the pattern now, no drawing. With only one index finger sticking out go draw

The core logic is to figure out where the finger is, see if it’s sticking out, and then roughly where the finger is.

drawing

There are two methods, one is to draw the picture on the artboard, the other is to display it in our picture.

Drawing palette

This is an important logical part. First of all, because we need to record the last drawing graph, we cannot directly save the picture returned by our camera, because the background is refreshed in real time and cannot be saved, so we need a static picture.And I’m going to merge the image and this method is going to merge the image inside the Window class just to make sure it doesn’t look weird.

The complete code

The basic project explanation section is OK, so let’s start with the code

ConterCenter.py

from Utils.HandDetecter import HandDetected
from Utils.SourseDetecter import SourseDetecter
from Utils.Window import Window
import numpy
class Controller(object) :
    def __init__(self) :
        self.HandDetected = HandDetected()
        self.window_w = 1300
        self.window_h = 720
        self.Window = Window(width=self.window_w,height=self.window_h)
        self.SourseDetecter = SourseDetecter()
        self.ID = 0
        self.PenColor = (0.0.255) # default color
        self.PenPostion = (0.0)
        self.Canvas = numpy.zeros((self.window_h,self.window_w,3),numpy.uint8)
        self.PenSize = 15

    def ControllerFingerTrack(self,result,img) :
        # We track the index and middle fingers
        postions = self.HandDetected.CheckHandUp(result)
        if(postions):
            indexFinger = postions[self.HandDetected.INDEXFINGER]
            middleFinger = postions[self.HandDetected.MIDDLEFINGER]

            if(indexFinger.get('flag') and middleFinger.get('flag') ):
                # index finger, middle finger stand up to check if the function is selected
                indexFinger_x ,indexFinger_y =int( indexFinger.get("x")*self.window_w),int (indexFinger.get("y")*self.window_h )
                middleFinger_x,middleFinger_y = int( middleFinger.get("x")*self.window_w),int(middleFinger.get("y")*self.window_h )

                center_x = int((indexFinger_x+middleFinger_x)/2)
                center_y = int((indexFinger_y+middleFinger_y)/2)

                self.Window.cv.circle(img, (center_x,center_y), 35, self.PenColor, self.Window.cv.FILLED)

                # Start specific functional modules
                if(0<=center_y and center_y<=125) :if(center_x>=125 and center_x<=260):
                        self.ID = 1
                        self.PenSize = 15
                        self.PenColor=(0.0.255)
                    if(center_x>=365 and center_x<=505):
                        self.ID = 2
                        self.PenSize = 15
                        self.PenColor = (255.0.0)
                    if(center_x>=615 and center_x<=755):
                        self.ID = 3
                        self.PenSize=15
                        self.PenColor=(0.255.0)
                    if(center_x>=1065 and center_x<=1205):
                        self.ID = 4
                        self.PenSize= 30
                        self.PenColor=(0.0.0)

        return img


    def DrawPenController(self,result,img) :
        # Start drawing
        postions = self.HandDetected.CheckHandUp(result)
        h, w, c = img.shape
        if (postions):
            indexFinger = postions[self.HandDetected.INDEXFINGER]
            middleFinger = postions[self.HandDetected.MIDDLEFINGER]


            if (indexFinger.get('flag') and not middleFinger.get('flag')):
                indexFinger_x, indexFinger_y = int(indexFinger.get("x") * self.window_w), int(
                    indexFinger.get("y") * self.window_h)
                self.Window.cv.circle(img, (indexFinger_x, indexFinger_y), self.PenSize, self.PenColor, self.Window.cv.FILLED)

                if (self.PenPostion == (0.0)) : self.PenPostion = (indexFinger_x,indexFinger_y) self.Window.cv.line(self.Canvas,self.PenPostion,(indexFinger_x,indexFinger_y),self.PenColor,self.PenSize) self.PenPostion = (indexFinger_x,indexFinger_y)else:
                self.PenPostion = (0.0)

        return img


    def ShowDrawWay(self,img) :
        h, w, c = img.shape
        self.Canvas = self.Window.cv.resize(self.Canvas, (w, h))
        img = self.Window.addImg(img, self.Canvas)
        # img = self. Window. CV. AddWeighted (img, 0.5, the self. Canvas, 0.5, 0)
        return img

    def Main(self) :

        while True:

            flag,img =  self.Window.cap.read()
            img = self.Window.cv.flip(img, 1)
            result,img = self.HandDetected.findHands(img,True)


            img = self.ControllerFingerTrack(result,img)

            img = self.DrawPenController(result,img)
            img = self.ShowDrawWay(img)
            img = self.Window.PrintUi(img, self.ID)

            self.Window.show("Canvas",img)
            if(self.Window.cv.waitKey(1) = =ord("q")) :return


if __name__=="__main__":
    Controller = Controller()
    Controller.Main()
Copy the code

HandDetecter.py

import mediapipe as mp
import cv2
import os
from Utils.SourseDetecter import SourseDetecter
class HandDetected(object) :
    def __init__(self) :
        self.mpHand = mp.solutions.hands
        self.Hand = self.mpHand.Hands()
        self.mphanddraw = mp.solutions.drawing_utils
        self.THUMB = 0
        self.INDEXFINGER=1
        self.MIDDLEFINGER=2
        self.RIGHTFINGER=3
        self.PINKY=4


    def findHands(self,img,flag=False,name="Hands") :
        You just need to pass the image from the camera into this function
        rgbimg = cv2.cvtColor(img,cv2.COLOR_BGR2RGB)
        result = self.Hand.process(rgbimg)
        if(flag):
            self.MarkHands(result,img,name)
        return result,img

    def GetHandPoints(self,result) :
        pass

    def FindUPFings(self,result,ID) :
        The method is suitable for viewing the condition of one finger at a time, not suitable for tracking multiple fingers at the same time
        if(result.multi_hand_landmarks):
            fingers = self.CheckHandUp(result)
            if(fingers[ID].get('flag')) :return fingers[ID]

    def CheckHandUp(self,result) :

        TipsId = [4.8.12.16.20]  # Coordinates of the fixed point
        hands_data = result.multi_hand_landmarks
        if (hands_data):
            for handlist in hands_data:
                fingers = []
                # Judge the thumb switch
                if(handlist.landmark[TipsId[0] -2].x < handlist.landmark[TipsId[0] +1].x):
                    if handlist.landmark[TipsId[0]].x >  handlist.landmark[TipsId[0] -1].x:
                        data = {"flag":0."x":int(handlist.landmark[TipsId[0]].x),"y":handlist.landmark[TipsId[0]].y}
                        fingers.append(data)
                    else:
                        data = {"flag": 1."x": int(handlist.landmark[TipsId[0]].x), "y": handlist.landmark[TipsId[0]].y}
                        fingers.append(data)
                else:
                    if handlist.landmark[TipsId[0]].x < handlist.landmark[TipsId[0] - 1].x:
                        data = {"flag": 0."x": handlist.landmark[TipsId[0]].x, "y": handlist.landmark[TipsId[0]].y}
                        fingers.append(data)
                    else:
                        data = {"flag": 1."x": handlist.landmark[TipsId[0]].x, "y": handlist.landmark[TipsId[0]].y}
                        fingers.append(data)
                # Judge other fingers
                for id in range(1.5) :if(handlist.landmark[TipsId[id]].y > handlist.landmark[TipsId[id] -2].y):
                        data = {"flag": 0."x": handlist.landmark[TipsId[id]].x, "y": handlist.landmark[TipsId[id]].y}
                        fingers.append(data)
                    else:
                        data = {"flag": 1."x": handlist.landmark[TipsId[id]].x,
                                "y": handlist.landmark[TipsId[id]].y}
                        fingers.append(data)


                return fingers
        else:
            return None


    def MarkHands(self,result,img,name) :
        # Whether to label the detected hand and then return the marked picture
        hands_data = result.multi_hand_landmarks
        if (hands_data):

            for handlm in hands_data:
                h, w, c = img.shape
                self.mphanddraw.draw_landmarks(img, handlm, self.mpHand.HAND_CONNECTIONS, )



    def __str__(self) :
        print("HandDetected for Identify Hands")

if __name__=="__main__":
    pass
Copy the code

SourseDetecter.py

import os
import cv2
import time

class SourseDetecter(object) :
    def __init__(self) :
        pass

    @staticmethod
    def GetUiImgs(path) :
        if(os.path.exists(path)):
            imgs = os.listdir(path)
            imgFiles = []
            for img in imgs:

                imgFile = cv2.imread(f"{path}/{img}")

                imgFiles.append(imgFile)

            return imgFiles

        else:

            raise FileNotFoundError("Not is File")
if __name__=="__main__":

    root_path = os.path.abspath(os.path.dirname(os.path.dirname(__file__)))+"\ \"
    img2 = SourseDetecter.GetUiImgs(root_path+"Media") [0]

Copy the code

Window.py

import cv2
import os
from Utils.SourseDetecter import SourseDetecter
class Window(object) :
    def __init__(self,sourse=0,width = 640,height = 480) :

        self.cv = cv2
        self.cap = self.cv.VideoCapture(sourse)
        self.cap.set(3,width)
        self.cap.set(4,height)
        self.SourseUi=[]
        root_path = os.path.abspath(os.path.dirname(os.path.dirname(__file__))) + "\ \"
        try:
            self.SourseUi = SourseDetecter.GetUiImgs(root_path+"Media")
        except Exception as e:
            self.SourseUi = []
            print(e)


    def PrintUi(self,img,ID) :
        if (self.SourseUi):
            hui, wui, cui = self.SourseUi[ID].shape

            img[0:hui,0:wui] = self.SourseUi[ID]
            return img
        else:
            return None

    def show(self,name,img) :

        self.cv.imshow(name,img)


    def addImg(self,img1,img2) :
        # merge two images,img2 is the image to be merged
        imgGray = self.cv.cvtColor(img2,self.cv.COLOR_BGR2GRAY) # Grayscale the canvas image to reduce the difference after merging
        _,imgInv = self.cv.threshold(imgGray,50.255,self.cv.THRESH_BINARY_INV)
        imgInv = self.cv.cvtColor(imgInv,self.cv.COLOR_GRAY2BGR) # Re-conversion after processing will color images

        img1 = self.cv.bitwise_and(img1,imgInv)
        img1 = self.cv.bitwise_or(img1,img2)



        return img1

    def release(self) :
        self.cap.release()
        self.cv.destroyAllWindows()

    def __str__(self) :
        print("For Show img")


Copy the code

conclusion

Happy New Year to you all in advance