This is the 14th day of my participation in Gwen Challenge

Image segmentation

Image segmentation is the technology and process of dividing an image into several specific and unique regions and proposing objects of interest. It is the key step from image processing to image analysis. The existing image segmentation methods can be divided into the following categories: thresholding based segmentation method, region based segmentation method, edge based segmentation method and specific theory based segmentation method. From a mathematical point of view, image segmentation is the process of dividing a digital image into non-intersecting areas. The process of image segmentation is also a marking process, that is, the pixels belonging to the same region are assigned the same number.

Grabcut

GraphCut requires the user to provide an accurate seed of the foreground background, and when the seed provided cannot cover all the distributions, the accuracy of segmentation is inevitably affected. To solve this problem, Microsoft Lab proposed a faster and more efficient GrabCut segmentation algorithm. GrabCut requires the user to provide a rectangle with a foreground inside and a background outside. GrabCut steps are as follows:

1: The pixels outside the rectangle are used as the background, and the pixels inside the rectangle are used as the foreground. The two groups are used to train the background GMM and the foreground GMM(here GMM refers to the Gaussian mixture model);

2: Two trained GMM are used to calculate the probability that each pixel belongs to background and foreground, and then calculate the Data item in the energy function E. The calculation method of Smoothness item in the energy function is roughly the same as GraphCut.

3: a segmentation of the image is obtained by optimizing the energy function;

4: Train foreground GMM and background GMM using foreground Pixels and background Pixels in segmentation results in 3;

5: Repeat 2, 3, and 4 until the segmentation result converges (there are no more big changes).

It can be seen from the above steps that GrabCut is a cyclic algorithm whose cyclic purpose is to exhaust Maximization (EM). Since the user-supplied rectangle also has some background pixels in it, the seed is not completely correct. Fortunately, the GMM model does not require all training data to be correct. Even if some classification is incorrect, the final result can be correct through THE EM step. GrabCut takes advantage of this GMM feature. It is worth noting that GMM has the problem of falling into local optimization that cannot be solved, so GrabCut has the same problem.

Algorithm paper address: www.microsoft.com/en-us/resea…

API

public static void grabCut(Mat img, Mat mask, Rect rect, Mat bgdModel, Mat fgdModel, int iterCount, int mode)
Copy the code

  • Parameter 1: img, the input image to be segmented. Must be 8-bit three-channel.

  • Parameter 2: mask, input/output 8-bit single-channel mask image. The pixel values in the image and their meanings are as follows.

    Sign a value meaning
    GC_BGD 0 A pixel that is clearly the background
    GC_FGD 1 A pixel that is clearly the foreground (object)
    GC_PR_BGD 2 A possible background pixel
    GC_PR_FGD 3 A possible foreground pixel

  • Parameter 3: RECT, which contains the ROI region of the subdivided object. Areas outside of ROI are marked as bits of “conspicuous background”. This parameter is used only when the seventh mode parameter is GC_INIT_WITH_RECT.

  • Parameter 4: bgdModel, temporary object of foreground model. When working with the same image, do not modify it.

  • Parameter 5: fgdModel, background model temporary object. When working with the same image, do not modify it.

  • Parameter 6: iterCount, number of iterations. Note that you can refine the result by making further calls with mode == GC_INIT_WITH_MASK or mode == GC_EVAL.

  • Parameter 7: mode, segmentation mode flag bit.

    Sign a value meaning
    GC_INIT_WITH_RECT 0 Initialize the state and mask using the supplied rectangle. After that, the algorithm is updated iteratively
    GC_INIT_WITH_MASK 1 Initialize the state using the supplied mask. GC_INIT_WITH_RECT and GC_INIT_WITH_MASK can be used together. Then, all pixels except ROI are automatically initialized using GC_BGD.
    GC_EVAL 2 Indicates that the algorithm should be restored
    GC_EVAL_FREEZE_MODEL 3 GrabCut algorithm is only run under fixed model (single iteration)

operation

Grabcut * author: yidong * 2020/11/21 */
class GrabcutActivity : AppCompatActivity() {

    private val mBinding: ActivityGrabcutBinding by lazy {
        ActivityGrabcutBinding.inflate(layoutInflater)
    }

    private lateinit var mRgb: Mat

    override fun onCreate(savedInstanceState: Bundle?). {
        super.onCreate(savedInstanceState)
        setContentView(mBinding.root)

        val bgr = Utils.loadResource(this, R.drawable.lena)
        mRgb = Mat()
        Imgproc.cvtColor(bgr, mRgb, Imgproc.COLOR_BGR2RGB)
        mBinding.ivLena.showMat(mRgb)
        GlobalScope.launch(Dispatchers.IO) {
            doGrabCut()
        }
    }

    private fun doGrabCut(a) {
        val rectMat = Mat()
        mRgb.copyTo(rectMat)
        val rect = Rect(80.30.340.390)
        Imgproc.rectangle(rectMat, rect, Scalar.all(255.0), 2)
        GlobalScope.launch(Dispatchers.Main) {
            mBinding.ivLena.showMat(rectMat)
        }
        val bgdModel = Mat.zeros(1.65, CvType.CV_64FC1)
        val fgdModel = Mat.zeros(1.65, CvType.CV_64FC1)
        val mask = Mat.zeros(mRgb.size(), CvType.CV_8UC1)
        Imgproc.grabCut(mRgb, mask, rect, bgdModel, fgdModel, 5, Imgproc.GC_INIT_WITH_RECT)

        val result = Mat()
        for (i in 0 until mask.rows()) {
            for (j in 0 until mask.cols()) {
                val value = mask.get(i, j)[0].toInt()
                if (value == 1 || value == 3) {
                    mask.put(i, j, 255.0)}else {
                    mask.put(i, j, 0.0)
                }
            }
        }
        Core.bitwise_and(mRgb, mRgb, result, mask)
        GlobalScope.launch(Dispatchers.Main) {
            mBinding.ivResult.showMat(result)
        }
    }

    override fun onDestroy(a) {
        mRgb.release()
        super.onDestroy()
    }
}
Copy the code

The effect

prospects

background

The source code

Github.com/onlyloveyd/…