Douyin beauty effect open source implementation, from AI to beauty process

Beauty and short videos

Beauty-related apps can be said to be essential software on mobile phones, such as Douyin and Kuaishou. No one will dare to upload the “photo fraud” and videos without beauty-related effects. Many people have been curious about how the beauty APP was developed. This article is about how to change lip color in real time on Android. Other functions such as whitening and blush are based on similar principles

The following lip color modification effect is to achieve the function

Beauty principle

The fundamentals of beauty are deep learning plus computer graphics. Deep learning is used for face detection and key point detection. Computer graphics are used to polish skin, thin faces and apply makeup. Generally, OpenGLES is used on Android and Metal is used on IOS. Due to the many and complex concepts of computer graphics, Canvas of Android is used instead in this paper.

Face detection & face key points

Face detection refers to the detection of the face in the picture or video stream, and locate the face in the picture.
Face key point detection is the key point positioning of the facial features and the contour of the face, in general, it is immediately after the face detection.

We will use TengineKit to achieve real-time big red lip effect.

TengineKit

Free mobile terminal real-time face 212 keypoints SDK. Is an easy to integrate face detection and face keypoint SDK. It can run on a variety of phones with very low latency.

Github.com/OAID/Tengin…

Achieve lipstick effect

Configuration Gradle

Build. Gradle in Project

repositories { ... mavenCentral() ... } allprojects { repositories { ... mavenCentral() ... }}Copy the code

Build. gradle added to main Module

    dependencies {
        ...
        implementation 'com. Tengine. Android: tenginekit: 1.0.5'. }Copy the code

Configuration manifests

    <uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />
    <uses-permission android:name="android.permission.INTERNET"/>

    <uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE"/>
    <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"/>
    <uses-permission android:name="android.permission.READ_PHONE_STATE"/>
Copy the code

Compared with the previous part using the camera for effect, this paper uses GIF images to replace the video stream input by the camera. If you want to use the camera for implementation, please refer to:

With open source 212 face key code zhuanlan.zhihu.com/p/161038093 Android human face in real-time

Process the image stream from Gif

First we initialize TengineKit:

The normal processing mode is selected
Open face detection and face key point functions
Set the image stream format to RGBA
Set the width and height of the input image stream. Here is the preview width and height of the GIF image
Set the width and height of the output picture stream. Here is the width and height of GifImageView, which is the same as GIF, so the width and height of GIF is used instead

    com.tenginekit.Face.init(getBaseContext(),
        AndroidConfig.create()
                .setNormalMode()
                .openFunc(AndroidConfig.Func.Detect)
                .openFunc(AndroidConfig.Func.Landmark)
                .setInputImageFormat(AndroidConfig.ImageFormat.RGBA)
                .setInputImageSize(facingGif.getGifWidth(), facingGif.getGifHeight())
                .setOutputImageSize(facingGif.getGifWidth(), facingGif.getGifHeight())
    );
Copy the code

Get the shape of the lips from the key points

    Path getMouthLandmarks(FaceLandmarkInfo fi){
        Path outPath = new Path();
        outPath.moveTo(fi.landmarks.get(180).X,fi.landmarks.get(180).Y);
        for(int i = 180; i < 189; i++){
            outPath.lineTo(
                    fi.landmarks.get(i).X,
                    fi.landmarks.get(i).Y
            );
        }
        for(int i = 204; i >= 196; i--){
            outPath.lineTo(
                    fi.landmarks.get(i).X,
                    fi.landmarks.get(i).Y
            );
        }

        outPath.close();

        Path inPath = new Path();
        inPath.moveTo(fi.landmarks.get(180).X,fi.landmarks.get(180).Y);

        for(int i = 195; i >= 188; i--){
            inPath.lineTo(
                    fi.landmarks.get(i).X,
                    fi.landmarks.get(i).Y
            );
        }
        for(int i = 204; i <= 211; i++){
            inPath.lineTo(
                    fi.landmarks.get(i).X,
                    fi.landmarks.get(i).Y
            );
        }

        outPath.op(inPath, Path.Op.DIFFERENCE);
        return  outPath;
    }
Copy the code

Color your lips

    public static void drawLipPerfect(Canvas canvas, Path lipPath, int color, int alpha) {
        //most 70% alpha
        if (alpha > 80) {
            alpha = (int) (alpha * 0.9 f + 0.5 f);
        }

        alpha = (int) (Color.alpha(color) * ((float) alpha / 255)) < <24;
        color = alphaColor(color, alpha);
        final PointF position = new PointF();
        float blur_radius = 5;

        Bitmap mask = createMask(lipPath, color, blur_radius, position);

        Paint paint = new Paint(Paint.ANTI_ALIAS_FLAG);
        paint.setXfermode(new PorterDuffXfermode(PorterDuff.Mode.SRC_OVER));
        canvas.drawBitmap(mask, position.x, position.y, paint);
    }
Copy the code

This code comes from github.com/DingProg/Ma…

Apply colours to a drawing

The bitmap sent is RGB_565 and needs to be converted to the standard RGBA format

    facingGif.setOnFrameAvailable(new GifImageView.OnFrameAvailable() {
        @Override
        public Bitmap onFrameAvailable(Bitmap bitmap) {
            // bitmap RGB_565

            Bitmap out_bitmap = Bitmap.createBitmap(
                    facingGif.getGifWidth(),
                    facingGif.getGifHeight(),
                    Bitmap.Config.ARGB_8888);

            Canvas canvas = new Canvas(out_bitmap);

            canvas.drawBitmap(bitmap, 0.0.null);
            bitmap.recycle();

            byte[] bytes = bitmap2Bytes(out_bitmap);
            Face.FaceDetect faceDetect = com.tenginekit.Face.detect(bytes);
            if(faceDetect.getFaceCount() > 0){
                faceLandmarks = faceDetect.landmark2d();
                if(faceLandmarks ! =null) {for (int i = 0; i < faceLandmarks.size(); i++) {
                        Path m_p = getMouthLandmarks(faceLandmarks.get(i));
                        LipDraw.drawLipPerfect(canvas, m_p, Color.WHITE, 100); }}}returnout_bitmap; }});Copy the code

Effect of contrast

advice

If you are interested, you can build on the current project. A lot of the code in this article also comes from this

Github.com/DingProg/Ma…

Further want to try commercial grade beauty effect can refer to

Github.com/CainKernel/…

reference