“This is the 30th day of my participation in the Gwen Challenge in November. See details: The Last Gwen Challenge in 2021”
Introduction to Face Detection
Face detection is the task of determining the position and size of a face in an image, and is often the basis for building face processing applications (for example, expression recognition, drowsiness detection, gender classification, face recognition or human-computer interaction, etc.), because such applications need to take the position and size of a detected face as input. Therefore, automatic face detection plays an important role and is one of the most studied subjects in artificial intelligence.
Face detection is not difficult for humans, but often involves many challenges for computers (such as appearance changes, scale shifts, rotation, facial expression changes, occlusion or lighting conditions, etc.). In this section, we will use popular face detection techniques (including machine learning and deep learning methods) provided by OpenCV to detect faces in images.
Face detection using OpenCV
OpenCV provides two face detection methods, including:
- Face detector based on Haar cascade
- Face detector based on deep learning
Face detector based on Haar cascade
OpenCV provides a face detection algorithm based on Viola and Jones proposed object detection framework. In addition, the object detection framework can also be used to detect other objects, such as license plate numbers or cat faces. In this section, we will use this framework to detect faces.
OpenCV provides four cascade classifiers for face detection. You can download these cascade classifiers from OpenCV official:
- haarcascade_frontalface_alt.xml (FA1)
- haarcascade_frontalface_alt2.xml (FA2)
- haarcascade_frontalface_alt_tree.xml (FAT)
- haarcascade_frontalface_default.xml (FD)
Can use different data sets to evaluate the performance of these cascade classifier, in general these classifications appliances have similar accuracy, therefore, the next in order to simplify the procedure, we use only the two of them (more specifically, only loading FA2 cascade classifier and FD), you can also choose to other interested classifier to test its performance. OpenCV provides the cv2.cascadeclassifier () function to load classifiers from files:
Load the cascade classifier
# first line of code
cas_alt2 = cv2.CascadeClassifier("haarcascade_frontalface_alt2.xml")
cas_default = cv2.CascadeClassifier("haarcascade_frontalface_default.xml")
Copy the codeNext, you can use the cv2. CascadeClassifier. DetectMultiScale () function performs testing:
# Second line of code, face detection is as simple as that
faces_alt2 = cas_alt2.detectMultiScale(gray)
faces_default = cas_default.detectMultiScale(gray)
Copy the codeCv2. CascadeClassifier. DetectMultiScale () function to detect object and returns them as rectangular list. To visualize, we finally write the show_detection() function:
def show_detection(image, faces) :
""" Mark each detected person with a rectangle drawn on their face. """
for (x, y, w, h) in faces:
cv2.rectangle(image, (x, y), (x + w, y + h), (255.0.0), 5)
return image
Call the show_detection() function to identify the detected face
img_faces_alt2 = show_detection(img.copy(), faces_alt2)
img_faces_default = show_detection(img.copy(), faces_default)
Copy the codeThis is the cv2.face.getFaceshaar () function provided by OpenCV:
# one line of code based on Haar cascade face detector, learn a send a
retval, faces_haar_alt2 = cv2.face.getFacesHAAR(img, "haarcascade_frontalface_alt2.xml")
retval, faces_haar_default = cv2.face.getFacesHAAR(img, "haarcascade_frontalface_default.xml")
Copy the codeCv2. CascadeClassifier. DetectMultiScale () needs to gray image as input, and cv2. Face. GetFacesHAAR () needs to BGR images as input. In addition, the cv2. CascadeClassifier. DetectMultiScale () to detect the human face output for rectangular list, for example, if detected two face, the output form is as follows:
[[809 494 152 152] [168 503 188 188]]
Copy the codeThe cv2.face.getFaceshaar () function returns the detected face in the following format:
[[[809 493 151 151]] [[167 503 189 189]]Copy the codeTherefore, if the cv2.face.getFaceshaar () function is used for detection, the np.squeeze() function is called to eliminate redundant dimensions when the detection box is drawn:
faces_haar_alt2 = np.squeeze(faces_haar_alt2)
faces_haar_default = np.squeeze(faces_haar_default)
Copy the codeFrom the loaded image, to detect and draw the complete code of the face is as follows:
# visualization function
def show_img_with_matplotlib(color_img, title, pos) :
img_RGB = color_img[:, :, ::-1]
ax = plt.subplot(2.2, pos)
plt.imshow(img_RGB)
plt.title(title, fontsize=8)
plt.axis('off')
# load image
img = cv2.imread("test_face_detection.jpg")
# Convert BGR image to grayscale image
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
Load classifiers
cas_alt2 = cv2.CascadeClassifier("haarcascade_frontalface_alt2.xml")
cas_default = cv2.CascadeClassifier("haarcascade_frontalface_default.xml")
# Face detection
faces_alt2 = cas_alt2.detectMultiScale(gray)
faces_default = cas_default.detectMultiScale(gray)
retval, faces_haar_alt2 = cv2.face.getFacesHAAR(img, "haarcascade_frontalface_alt2.xml")
faces_haar_alt2 = np.squeeze(faces_haar_alt2)
retval, faces_haar_default = cv2.face.getFacesHAAR(img, "haarcascade_frontalface_default.xml")
faces_haar_default = np.squeeze(faces_haar_default)
# Draw face detection box
img_faces_alt2 = show_detection(img.copy(), faces_alt2)
img_faces_default = show_detection(img.copy(), faces_default)
img_faces_haar_alt2 = show_detection(img.copy(), faces_haar_alt2)
img_faces_haar_default = show_detection(img.copy(), faces_haar_default)
# visualization
show_img_with_matplotlib(img_faces_alt2, "detectMultiScale(frontalface_alt2): " + str(len(faces_alt2)), 1)
show_img_with_matplotlib(img_faces_default, "detectMultiScale(frontalface_default): " + str(len(faces_default)), 2)
show_img_with_matplotlib(img_faces_haar_alt2, "getFacesHAAR(frontalface_alt2): " + str(len(faces_haar_alt2)), 3)
show_img_with_matplotlib(img_faces_haar_default, "getFacesHAAR(frontalface_default): " + str(len(faces_haar_default)), 4)
plt.show()
Copy the codeThe output of the program is shown below:
As shown in the figure above, faces can be detected differently by using different hierarchical classifiers based on haar features. Finally, to be sure, cv2. CascadeClassifier. DetectMultiScale minSize and maxSize parameters () function, To set the minimum size (objects smaller than minSize will not be detected) and the maximum size (objects larger than maxSize will not be detected), and the cv2.face.getFaceshaar () function does not provide this parameter.
Face detector based on deep learning
In addition to the face detector based on Haar cascade, OpenCV provides a face detector based on deep learning, OpenCV Deep Neural Networks (DNN) face Detector is based on the Single Shot MultiBox Detector (SSD) framework using resNET-10 network.
The DNN module in OpenCV uses popular deep learning frameworks (such as Caffe, TensorFlow, Torch, and Darknet) to implement forward computation (i.e., the reasoning stage) through pre-trained deep networks. This means that we can use the pre-trained network to perform full forward calculations and use the output to make predictions in our application without spending hours training the network.
Next, our pre-trained deep learning face detector model performs face detection, using the following two models:
- Face detector (FP16) : floating point 16 version of Caffe’s implementation. To use this detector, first download the model file (deploy.prototxt) and configuration file (res10_300x300_SSD_iter_140000_fp16.caffemodel).
- UINT8: An 8-bit quantized version of TensorFlow’s implementation. To use this detector, first download the model file (opencv_face_detector_UINT8.pb) and configuration file (opencv_face_DETECtor_UINT8.pb).
Next, I will introduce how to use the pre-trained deep learning face detector model to detect faces. The first step is also to load the pre-trained model:
# Load pretrained model, Caffe implementation version
net = cv2.dnn.readNetFromCaffe("deploy.prototxt"."res10_300x300_ssd_iter_140000_fp16.caffemodel")
# Load pretrained model, version of Tensorflow implementation
# net = cv2.dnn.readNetFromTensorflow("opencv_face_detector_uint8.pb", "opencv_face_detector.pbtxt")
Copy the codeTo obtain the best accuracy, it is necessary to perform (104, 177, 123) channel mean subtraction for blue, green, and red channels respectively and adjust the image to a 300 x 300 BGR image. This can be done in OpenCV by using the cv2.dnn.blobFromImage() function:
blob = cv2.dnn.blobFromImage(image, 1.0, (300.300),104..117..123.].False.False)
Copy the codeThe next step is to set the BLOB as the input to get the result and perform a forward calculation over the entire network to calculate the output:
Set bloB as input and get detection results
net.setInput(blob)
detections = net.forward()
Copy the codeThe last step is to iteratively detect and draw the results, visualizing them only when the corresponding confidence is greater than the minimum threshold:
# Iterate over all detection results
for i in range(0, detections.shape[2) :Get the confidence of the current detection result
confidence = detections[0.0, i, 2]
If the confidence is greater than the minimum confidence, visualize it
if confidence > 0.7:
detected_faces += 1
Get the coordinates of the current detection result
box = detections[0.0, i, 3:7] * np.array([w, h, w, h])
(startX, startY, endX, endY) = box.astype('int')
Draw detection results and confidence
text = "{:.3f}%".format(confidence * 100)
y = startY -10 if startY - 10 > 10 else startY + 10
cv2.rectangle(image, (startX, startY), (endX, endY), (255.0.0), 3)
cv2.putText(image, text, (startX, y), cv2.FONT_HERSHEY_SIMPLEX, 0.9, (0.0.255), 2)
# visualization
show_img_with_matplotlib(image, "DNN face detector: " + str(detected_faces), 1)
plt.show()
Copy the codeThe detection results of face detector based on deep learning are shown as follows:
As can be seen from the figure above, all three frontal faces can be detected with high confidence by the two deep learning models.