This is the seventh day of my participation in the First Challenge 2022. For details: First Challenge 2022.

Preface:

In iOS, we will often see CVPixelBufferRef. The most common scenario is that when the Camera is collected, the returned data has a CMSampleBufferRef. Each CMSampleBufferRef contains a CVPixelBufferRef, as well as a CVPixelBufferRef (which contains all compressed image information) in the data returned from the hard decoding of the video. After understanding CVPixelBufferRef, we will be able to master and use CVPixelBufferRef;

In this article, we are mainly familiar with the use of CVPixelBuffer;

CVPixelBuffer profile

CVPixelBuffer: The core video pixel buffer is an image buffer that holds pixels in main memory. CVPixelBuffer can be used by applications that generate frames, compress or uncompress video, or use Core Image.

CVPixelBufferRef: is the pixel buffer type, a reference to the CVPixelBuffer object. The pixel buffer type is based on the image buffer type; Pixel buffer implements memory storage of image buffer.

/ *! @typedef CVPixelBufferRef @abstract Based on the image buffer type. The pixel buffer implements the memory storage for an image buffer. */ typedef CVImageBufferRef CVPixelBufferRef;Copy the code

Since CVPixelBufferRef is an object in C, there is no ARC memory management and the developer must manage the reference count and control the object life cycle.

First of all, we will first understand the basic methods of CVPixelBufferRef, such as: CVPixelBufferRef creation, holding, release;

create

There are four methods of creation, the first three of which are commonly used:

  • CVPixelBufferCreate()
  • CVPixelBufferCreateWithBytes()
  • CVPixelBufferCreateWithPlanarBytes()
  • CVPixelBufferCreateWithIOSurface()

Normal creation method

/ *! CVPixelBufferCreate @abstract Call to create a single PixelBuffer for a given size and pixel format pixelFormatType. @discussion Creates a single PixelBuffer for a given size and pixelFormatType. It allocates the necessary memory based on the pixel dimensions, pixelFormatType and extended pixels described in the pixelBufferAttributes. Not all parameters of the pixelBufferAttributes will be used here. @param width Width of the PixelBuffer in pixels. @param height Height of the PixelBuffer in pixels. @param pixelFormatType Pixel format indentified by its respective OSType. @param pixelBufferAttributes A dictionary with additional attributes for a pixel buffer. This parameter is optional. See BufferAttributeKeys for more details. @param pixelBufferOut The new pixel buffer will be returned here @result returns kCVReturnSuccess on success. */ CV_EXPORT CVReturn CVPixelBufferCreate( CFAllocatorRef CV_NULLABLE allocator, size_t width, size_t height, OSType pixelFormatType, CFDictionaryRef CV_NULLABLE pixelBufferAttributes, CV_RETURNS_RETAINED_PARAMETER CVPixelBufferRef CV_NULLABLE * CV_NONNULL pixelBufferOut) __OSX_AVAILABLE_STARTING(__MAC_10_4,__IPHONE_4_0);Copy the code

Creates a pixel buffer with data specified by memory location

/ *! Create a pixel buffer for the given size and pixel format, Contains the memory location specified data @ function CVPixelBufferCreateWithBytes @ the abstract the Call to create a single PixelBuffer for a given size and  pixelFormatType based on a passed in piece of memory. @discussion Creates a single PixelBuffer for a given size and pixelFormatType. Not all parameters of the pixelBufferAttributes will be used here. It requires a release callback function that will be called, when the PixelBuffer gets destroyed so that the owner of the pixels can free the memory. @param width Width of the PixelBuffer in pixels @param height Height of the PixelBuffer in pixels @param pixelFormatType Pixel format indentified by its respective OSType. @param baseAddress Address of the memory storing the pixels. @param bytesPerRow Row bytes of the pixel storage memory. @param releaseCallback CVPixelBufferReleaseBytePointerCallback function that gets called when the PixelBuffer gets destroyed. @param releaseRefCon User data identifying the PixelBuffer for the release callback. @param pixelBufferAttributes A dictionary with additional attributes for a a pixel buffer. This parameter is optional. See PixelBufferAttributes for more details. @param pixelBufferOut The new pixel buffer will be returned here @result returns kCVReturnSuccess on success. */ CV_EXPORT CVReturn CVPixelBufferCreateWithBytes( CFAllocatorRef CV_NULLABLE allocator, size_t width, size_t height, OSType pixelFormatType, void * CV_NONNULL baseAddress, size_t bytesPerRow, CVPixelBufferReleaseBytesCallback CV_NULLABLE releaseCallback, void * CV_NULLABLE releaseRefCon, CFDictionaryRef CV_NULLABLE pixelBufferAttributes, CV_RETURNS_RETAINED_PARAMETER CVPixelBufferRef CV_NULLABLE * CV_NONNULL pixelBufferOut) __OSX_AVAILABLE_STARTING(__MAC_10_4,__IPHONE_4_0);Copy the code

Create a CVPixelBuffer in flat format

/ *! @function CVPixelBufferCreateWithPlanarBytes @abstract Call to create a single PixelBuffer in planar format for a given size and pixelFormatType based on a passed in piece of memory. @discussion Creates a single PixelBuffer for a given size  and pixelFormatType. Not all parameters of the pixelBufferAttributes will be used here. It requires a release callback function that will be called, when the PixelBuffer gets destroyed so that the owner of the pixels can free the memory. @param width Width of the PixelBuffer in pixels @param height Height of the PixelBuffer in pixels @param pixelFormatType Pixel format indentified by its respective OSType. @param dataPtr Pass a pointer to a plane descriptor block, or NULL. @param dataSize pass size if planes are contiguous, NULL if not. @param numberOfPlanes Number of planes. @param planeBaseAddress Array of base addresses for the planes. @param planeWidth Array of plane widths. @param planeHeight Array of plane heights. @param planeBytesPerRow Array of plane bytesPerRow values. @param releaseCallback CVPixelBufferReleaseBytePointerCallback function that gets called when the PixelBuffer gets destroyed. @param releaseRefCon User data identifying the PixelBuffer for the release callback. @param pixelBufferAttributes A dictionary with additional attributes for a a pixel buffer. This parameter is optional. See PixelBufferAttributes for more details. @param pixelBufferOut The new pixel buffer will be returned here @result returns kCVReturnSuccess on success. */ CV_EXPORT CVReturn CVPixelBufferCreateWithPlanarBytes( CFAllocatorRef CV_NULLABLE allocator, size_t width, size_t height, OSType pixelFormatType, void * CV_NULLABLE dataPtr, // pass a pointer to a plane descriptor block, or NULL size_t dataSize, // pass size if planes are contiguous, NULL if not size_t numberOfPlanes, void * CV_NULLABLE planeBaseAddress[CV_NONNULL ], size_t planeWidth[CV_NONNULL ], size_t planeHeight[CV_NONNULL ], size_t planeBytesPerRow[CV_NONNULL ], CVPixelBufferReleasePlanarBytesCallback CV_NULLABLE releaseCallback, void * CV_NULLABLE releaseRefCon, CFDictionaryRef CV_NULLABLE pixelBufferAttributes, CV_RETURNS_RETAINED_PARAMETER CVPixelBufferRef CV_NULLABLE * CV_NONNULL pixelBufferOut) __OSX_AVAILABLE_STARTING(__MAC_10_4,__IPHONE_4_0);Copy the code

Modify the

Before using CPU access to the pixel data must call CVPixelBufferLockBaseAddress, then call CVPixelBufferUnlockBaseAddress. This is not necessary when using a GPU to access pixel data.

Fill the PixelBuffer extension

/ *! Fill extension PixelBuffer @ function CVPixelBufferFillExtendedPixels @ the abstract Fills the extended pixels of the PixelBuffer. This function replicates edge pixels to fill the entire extended region of the image. @param pixelBuffer Target PixelBuffer. */ CV_EXPORT CVReturn CVPixelBufferFillExtendedPixels( CVPixelBufferRef CV_NONNULL pixelBuffer ) __OSX_AVAILABLE_STARTING(__MAC_10_4,__IPHONE_4_0);Copy the code

Lock the PixelBuffer address

/ *! Locking PixelBuffer address @ function CVPixelBufferLockBaseAddress @ the abstract Description the Locks the BaseAddress of the PixelBuffer to  ensure that the memory is accessible. @discussion This API ensures that the CVPixelBuffer is accessible in system memory. This should only be called if the base address is going to be used and the pixel data will be accessed by the CPU. @param pixelBuffer Target PixelBuffer. @param lockFlags See CVPixelBufferLockFlags. @result kCVReturnSuccess if the  lock succeeded, or error code on failure */ CV_EXPORT CVReturn CVPixelBufferLockBaseAddress( CVPixelBufferRef CV_NONNULL pixelBuffer, CVPixelBufferLockFlags lockFlags ) __OSX_AVAILABLE_STARTING(__MAC_10_4,__IPHONE_4_0);Copy the code

Unlock the PixelBuffer address

/ *! @function CVPixelBufferUnlockBaseAddress @abstract Description Unlocks the BaseAddress of the PixelBuffer. @param pixelBuffer Target PixelBuffer. @param unlockFlags See CVPixelBufferLockFlags. @result kCVReturnSuccess if the unlock succeeded, or error code on failure */ CV_EXPORT CVReturn CVPixelBufferUnlockBaseAddress( CVPixelBufferRef CV_NONNULL pixelBuffer, CVPixelBufferLockFlags unlockFlags ) __OSX_AVAILABLE_STARTING(__MAC_10_4,__IPHONE_4_0);Copy the code

Memory management (reserved or freed)

Since CVPixelBufferRef is an object in C, there is no ARC memory management. It is up to the developer to manage the reference count and control the life cycle of the object.

hold

/ *! Reference count +1 @function CVPixelBufferRetain @abstract Retains a CVPixelBuffer object @discussion Equivalent to CFRetain, but NULL safe @param buffer A CVPixelBuffer object that you want to retain. @result A CVPixelBuffer object that is the same as the passed in buffer. */ CV_EXPORT CVPixelBufferRef CV_NULLABLE CVPixelBufferRetain( CVPixelBufferRef CV_NULLABLE texture ) __OSX_AVAILABLE_STARTING(__MAC_10_4,__IPHONE_4_0);Copy the code

The release of

/ *! 1 @function CVPixelBufferRelease @abstract Releases a CVPixelBuffer object @Discussion Equivalent to CFRelease, but NULL safe @param buffer A CVPixelBuffer object that you want to release. */ CV_EXPORT void CVPixelBufferRelease( CV_RELEASES_ARGUMENT CVPixelBufferRef CV_NULLABLE texture ) __OSX_AVAILABLE_STARTING(__MAC_10_4,__IPHONE_4_0);Copy the code

check

CVPixelBufferGetHeight Returns the height of the pixel buffer. CVPixelBufferGetWidth returns the width of the pixel buffer

Here more methods, not one example, you can view the official document

Pixel format type kCVPixelFormatType

RGB :

kCVPixelFormatType_32BGRA = 'BGRA' kCVPixelFormatType_32BGRA = 'BGRA', kCVPixelFormatType_32ABGR = 'ABGR',   kCVPixelFormatType_32RGBA = 'RGBA',

NV12 :

kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange = '420v', kCVPixelFormatType_420YpCbCr8BiPlanarFullRange = '420f',

YUV420P : kCVPixelFormatType_420YpCbCr8Planar = 'y420',

KCVPixelFormatType_ {| sequence length}} {color space {Planar | BiPlanar} {VideoRange | FullRange}Copy the code

As can be seen from these types, both YUV format and RGB format can be created as CVPixelBuffer;

So after we get CVPixelBufferRef, if we want to display it, we can use CVPixelBufferRef to turn it into A UIImage, or we can display it by drawing a texture.

If you want to know more about kCVPixelFormatType, you need to know more about YUV format and RGB format, so in the next article, we will explain the two formats in detail, and then look at kCVPixelFormatType. That will deepen our understanding of it.

Next chapter: “iOS CVelBuffer (Middle)”