Make writing a habit together! This is my first day to participate in the “Gold Digging Day New Plan · April More text challenge”, click to see the details of the activity.

1. Calculation of matching cost

Matching cost is used to measure the correlation between candidate pixels and matching pixels. The smaller the cost, the greater the correlation between two pixels and the higher the probability of having the same name. The so-called point with the same name is the corresponding point in the left and right images, and the point in the red box in the figure below is the point with the same name.

It should be noted that no matter whether the two pixels in the two pictures have the same name or not, the cost can be matched. It is just that the cost of the two pixels that do not have the same name is high. In this way, it will traverse and search relevant pixel points, calculate their surrogate value, and then find a pixel point with the lowest cost as the same name. In fact, before each pixel searches for a point with the same name, it will specify a parallax search range. The parallax search range is limited to D, and a THREE-DIMENSIONAL matrix C with a size of W×H×D (W is the image width, H is the image height) is used to store the matching generation value of each pixel within the parallax range under each parallax. Matrix C is often called DSI (Disparity Space Image). Note the words marked in red above, matrix C is used to store the matching cost, which is searched in the parallax range D class, and then stores the matching cost of each pixel (each pixel seat is marked (x,y)) under each parallax (D). It can be clearly seen that matrix C is a three-dimensional matrix, combined with the following figure for further understanding.

At this point, is there a question? So how do we calculate this matching cost? In fact, there are many kinds of matching cost calculation, traditional methods have AD (gray absolute difference), NCC (normalized correlation coefficient) and so on; In computer vision, there are CT (Census transform), MI (mutual information method) and so on. These different matching cost calculation methods have their own advantages and disadvantages, the effect of different data is not the same, therefore, we choose the appropriate matching cost calculation method in stereo matching is very critical. The matching cost calculation method is as follows: Matching cost calculation

 

2. Cost aggregation

Said to cost is used to measure the candidate pixels and match the correlation between pixels, matching the price is often through two pixel neighborhood within a certain window to calculate the pixel size, only consider the local information, it is calculated easily affected by factors such as noise, this will lead to real point to the value of the same name is not the youngest, would be wrong to find the same point. What does this have to do with cost aggregation? As you might have guessed, the fundamental purpose of cost aggregation is to make the substitution value accurately reflect the correlation between pixels, in plain English, to make the matching cost calculation better. We can intuitively feel the effect of using cost aggregation on the picture effect through the following set of pictures.

It can be seen that the effect of cost aggregation is still obvious, and the aggregated image is obviously closer to the standard image (Ground Truth). Cost aggregation is the relationship established between adjacent pixels, to certain criteria, such as adjacent pixels should have continuous parallax value, to optimize the cost matrix, the optimization tends to be global, “new generation of value for each pixel in a parallax will according to its adjacent pixels in the same generation under the parallax value or near parallax value to recalculate, get new DSI, Let me write it in terms of the matrix S. (The dimensions of matrix S and matrix C are consistent) How to understand the continuous parallax marked in red above? The following is about parallax continuity and parallax discontinuity. Parallax continuity means that the parallax difference of pixels within the local range is very small (within 1 pixel), which is a continuous trend of change. Non-continuous refers to the large difference (more than 1 pixel) in the local pixel parallax, which is a sudden change trend. This local scope is usually a rectangular window (e.g., 3×3, 5×5). Since parallax and depth are actually equivalent to some extent (Z is the apparent depth and DIff is the parallax), parallax continuity expresses the continuity of the distance between the target surface and the camera in space. If the target is imprinted on a continuous surface image, parallax within the imaging range is also continuous. If the target has foreground and background in the image, part of the foreground and part of the background will belong to the foreground and part to the background in the local window. The distance between foreground and background may be very different from the camera, and the parallax will also be very different, that is, discontinuous. The continuous region and discontinuous region of parallax can be intuitively felt in the following figure.

Common cost aggregation methods include scan line method, dynamic programming method and path clustering in SGM algorithm.

3. Parallax calculation

Parallax calculation is relatively simple, which is to determine the optimal parallax value of each pixel through the matrix S after cost aggregation, generally using the winner-takes-all algorithm (WTA). The figure below shows the generational values of all parallax of a pixel. The WTA algorithm selects the parallax corresponding to the smallest cost as the optimal parallax.

 

Parallax optimization

Through the parallax calculation in Step 3, the parallax of each pixel can be obtained, that is, a parallax map can be obtained. The purpose of parallax optimization is to optimize the parallax graph obtained in Step 3 to further improve the quality of the parallax graph. The left-right consistency check algorithm is generally used to eliminate the false parallax caused by occlusion and noise. Isolated outliers are eliminated by eliminating small connected regions algorithm. Smoothing algorithms such as median filter and bilateral filter are used to smooth the parallax graph. In addition, some effective methods to improve the quality of parallax maps, such as robust plane fitting, brightness consistency constraint, local consistency constraint, are often used.

 

5, summary

The above four steps are not necessary for all algorithms. The steps of local matching algorithm generally include matching cost calculation, cost aggregation and parallax calculation, while the steps of global algorithm include matching cost calculation, parallax calculation and parallax optimization. Semi-global Matching (SGM) has four steps.

Refer to the article: ethanli.blog.csdn.net/article/det…