Distortion correction

The last article described the mathematical principle of binocular vision, but the whole article was based on the pinhole camera model to explain, but in fact for better imaging effect we use lens camera, the principle is similar between the two, but lens camera will produce perspective distortion, also called perspective distortion.

The above picture shows the shooting result of a binocular camera. The shooting target is the calibration plate used for camera calibration. It can be obviously observed that the edge of the calibration plate is bent under the action of perspective distortion.

Polar correction

In addition to the problem of distortion, there is actually a point problem. We can see it more clearly by drawing some parallel lines. You will find that the coordinates of the same point (the point pair of the image taken on the left and right) in the direction of YYY axis in the respective image (or projection) coordinate system are different.

That is to say, to a point in the left, wanted to seek the same in the right spot, need to traverse the entire image, but if we can make the same point of YYY axis coordinates are the same, we just need to traverse a line in the right picture, not only lower time complexity, and need to match the location of the less so the result is more accurate.

The so-called traversal matching point of the same name, is to use such as (5×5)(5\times5)(5×5) or other size matrix, with each pixel as the center of the matrix to traverse, through some formula to calculate the correlation of two (5×5)(5\times5)(5×5) matrix, to judge whether they are the same name point.

The reason why the direction coordinates of YYY axis of the same name are different is that we cannot guarantee that ZZZ axis of the left and right camera coordinate system is parallel.

Here, the camera coordinate system is defined. In our binocular analogy, XXX axis is forward on the right side of vision, YYY axis is forward under vision, and ZZZ axis is forward in front of vision.

As shown in the figure above, the polar line is the intersection line between the polar plane and the imaging plane composed of the photographed point, the origin of the left camera coordinate system and the origin of the right camera coordinate system. The correction method is to create the camera coordinate system parallel to ZZZ axis for the left and right cameras respectively. Points on the imaging plane are mapped to the virtual image plane through some mathematical formula.

Code implementation

Distortion correction and pole line correction are implemented with stereoRectify. Let’s see the effect. It is not difficult to find that the edge of the calibration plate is no longer a curve, and the coordinates of YYY axis of the same name also remain consistent.

Before calibration, we need to calibrate the binocular camera to obtain the camera parameters. This is done using OpenCV Python, or if you’re used to OpenCV C++.

calibration

First, look at the functional signature of binocular calibration.

def stereoCalibrate(objectPoints, imagePoints1, imagePoints2, cameraMatrix1, distCoeffs1, cameraMatrix2, distCoeffs2, imageSize [, R[, T[, E[, F[, flags[, criteria]]]]]]) -> (
    retval,
    cameraMatrix1,
    distCoeffs1,
    cameraMatrix2,
    distCoeffs2,
    R, T, E, F
)
Copy the code

Let’s explain what these parameters and return values mean one by one:

parameter

  1. ObjectPoints: World coordinates of checkerboard corners, where the length of each grid side is 1 unit.
w, h = 9.6 # The number of corners in each direction of the checkerboard
def obj_points() :
    points = []
    for x in range(w):
        for y in range(h):
            points.append([x, y, 0])
    return [points]*len(imgs)

There is also a more concise way of writing this, which is equivalent to the above
def obj_ponits() :
    points = np.zeros((np.prod((w, h)), 3), np.float32)
    points[:, :2] = np.indices((w, h)).T.reshape(-1.2)
    return [points]*len(imgs)
Copy the code
  1. ImagePoints: Image coordinates of checkerboard corners, which need to be obtained by corner detection.
def detect_corners(img) :
    ok, corners = cv2.findChessboardCorners(img, self.board_size)
    assert ok
    cv2.cornerSubPix(img, corners, (11.11), (-1, -1), (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_COUNT, 30.0.01))
    return corners
img_points = [detect_corners(img).reshape(-1.2) for img in imgs]
Copy the code
  1. CameraMatrix: Camera internal parameter matrix, also known as KKK matrix mentioned in the previous article, needs to be initialized by objectPoints and imagePoints. KKK obtained here is only an initial value, which is not accurate and needs many iterations.
K = cv2.initCameraMatrix2D(obj_ponits, img_points, img_size, 0)
Copy the code

  1. DistCoeffs: Distortion parameter that requires an initial value for subsequent iterations or can be null.

  2. ImageSize: Size of the image (W,h)(w, h)(w,h), for example (640,480)(640, 480)(640,480) (640,480).

  3. R, T, E, F: outgoing parameters without transmission.

  4. Flags:

    • CALIB_FIX_INTRINSIC: Fixes the incoming cameraMatrix and distCoeffs and computs only R, T, E, and F.
    • CALIB_USE_INTRINSIC_GUESS: Start the iteration with the initial values of the cameraMatrix and distCoeffs passed in.
    • CALIB_FIX_PRINCIPAL_POINT: Fixes the primary point during iteration.
    • CALIB_FIX_FOCAL_LENGTH: Fixed focal length during iteration.
    • CALIB_FIX_ASPECT_RATIO: Fixes fxF_XFX and FYF_Yfy ratios.
    • CALIB_SAME_FOCAL_LENGTH: Forces the focal length of two cameras to be the same.
    • CALIB_ZERO_TANGENT_DIST: Tangential distortion remains zero.
    • CALIB_RATIONAL_MODEL: Uses a more accurate distortion model.
    • CALIB_FIX_K* : The corresponding distortion parameter is not changed during iteration.

  5. Criteria: Termination conditions for an iteration.

criteria = (
    cv2.TERM_CRITERIA_EPS |    # Achieve precision
    cv2.TERM_CRITERIA_COUNT,   Reach the number of iterations
    100.# number of iterations
    1e-5                       # precision
)
Copy the code

The return value

  1. Retval: error (root mean square not used in this article)
  2. CameraMatrix: KKK matrix after calculation.
  3. DistCoeffs: indicates the calculated distortion parameter.
  4. R: Rotation matrix between cameras.
  5. T: Translation matrix between cameras.
  6. E: Essential matrix.
  7. F: Fundamental matrix is not used in this paper.

Here, it is assumed that cameraMatrix1 describes the left camera and cameraMatrix2 describes the right camera, then R and T returned by the function are the transformation from the left camera coordinate system to the right camera coordinate system, that is to say, the result obtained after R and T transformation of the left camera coordinate system, Is the coordinate of the point in the right camera coordinate system.

correction

Let’s first look at the functional signature of stereoscopic correction.

def stereoRectify(cameraMatrix1, distCoeffs1, cameraMatrix2, distCoeffs2, imageSize, R, T [, R1[, R2[, P1[, P2[, Q[, flags[, alpha[, newImageSize]]]]]]]]) -> (
    R1,
    R2,
    P1,
    P2,
    Q,
    validPixROI1,
    validPixROI2
)
Copy the code

The arguments and return values are explained one by one, so I won’t repeat them.

parameter

  1. Flags:
    • CALIB_ZERO_DISPARITY: Make two images have the same image coordinates after calibration.
  2. Alpha:
    • 0: The corrected image is changed without black areas.
    • 1: Keep all the pixels taken, there will be black edges.

The return value

  1. R: rotation matrix.
  2. P: projection matrix.
  3. Q: Disparity depth mapping matrix (Disparity to depth mapping matrix is not used in this paper).
  4. ValidPixROI: Valid region, if alpha = 0, the valid region is all pixels.

Let’s clarify the meaning of R, PR, PR and P matrix.

The rotation matrix RRR is not the rotation matrix from the world coordinate system to the camera coordinate system, but the rotation matrix from the real camera coordinate system to the virtual camera coordinate system.

Projection matrix PPP is the projection matrix from the virtual camera coordinate system to the virtual projection coordinate system.

End

The above two functions calculate the four matrices K, D, R, PK, D, R, PK, D, R, R, P, among which K, DK, DK, D are used for distortion correction, and R, PR, PR, P are used for pole line correction.

This paper just said the use of the function (I do not know how the specific algorithm is run), the description of the parameters of the function is actually official documentation, if there is not professional or inaccurate place please point out.

This article code address: github.com/suqinglee/C…