The wheat fields, the sky, the water, the horses, the trees… What color will it be?

Black and white and color

Seems to divide The Times in two

I see here the confusion of prosperity, the dazzling, the rapid ebb and flow of color.

Open a black and white photograph

I can see the marks of generations, a beauty that never dies

I can see but I can’t see what you can see

What seemed a casual glance seemed a startling one in my heart

Daunting

But do not live quietly fantasy

The wheat fields, the sky, the water, the horses, the trees…

What color will it be?

In the moment of the freeze frame

I would give you colors,

Take you to experience this bustling scene ~

Black and white image coloring model principle

This application is based on Centerm AI heterogeneous computing architecture CANN and is built by using black and white image coloring model. It is suitable for Atlas 200 DK and Atlas 300I and other related inference products.

Objects (including background color) (grayscale image represented by L channel) are extracted by convolution operation, and then classified by convolution, so as to try to give the color expectation of each pixel of grayscale image (AB channel). Then, resize the value of ab channel to the width and height of the original picture and superposition it with L channel to convert it into RGB image to get a color image. The input of the model is the data of channel L (224,224,1), and the output is the data of channel ab (56,56,2).

We can also use open source model test scripts to understand the input and output of the original model, as well as the calculation process before input (also known as pre-processing) and the processing process after output (also known as post-processing).

Preprocessing process: RGB format read into Lab, resize to 224224, extract L channel, subtract the mean (-50)

Post-processing: the result of reasoning ab channel, resize to 224224, and input L merged into Lab, converted to RGB, saved as JPEG image

Model original link: github.com/richzhang/c…

Prepare model

Ascend Tensor Compiler (ATC) can replace the network models of open source frameworks (such as Caffe, TensorFlow, etc.) and the single-operator description files defined by Ascend IR with offline models supported by the Ascend AI processor. In the process of model transformation, operator scheduling optimization, weight data rearrangement and memory utilization optimization can be realized, and model pretreatment can be completed without equipment.

Using ATC to convert the black and white image coloring model to offline model supported by centerm AI processor:

Atc – input_shape = “data_l: 1,1,224,224” – weight = “. / colorization. Caffemodel “- input_format = NCHW –output=”./colorization” –soc_version=Ascend310–framework=0 –model=”./colorization.prototxt”

Application development

This application adopts the modular design as shown in the following figure, and completes the inference output of a picture through coordination and cooperation between modules. The reasoning of the video is the same, you can directly refer to the source code in the open source warehouse. I won’t repeat it here.

The main function points of each module are as follows:

  • Run management resource requests

Used to initialize internal system resources, fixed call flow.

  • Load the model file and build the output memory

To load offline model data from files, users need to manage the memory of model running by themselves. Basic model information including buffer size of model input and output data can be obtained according to the model loaded in memory. The model output memory is constructed from the basic information of the model to prepare for the following model inference.

  • Data preprocessing

Receive the image data transmitted from the outside world, preprocess the image data read in, and then build the input data of the model.

  • Model reasoning

Model inference is performed based on the constructed model input data.

  • Analytic reasoning result

Based on the model output, the reasoning results of the model are analyzed. Use OpencV to save the converted color image data into picture files.

Preprocessed image

Image preprocessing part, using OpencV imread interface to read the image, read out is BGR format.

The model input is 224×224, so the image read needs to be resized to 224×224. After Resize, the data is normalized.

As mentioned in the algorithm design section, the model uses Lab color space, so it is necessary to convert BGR format data into Lab format data. The model uses L channel data to predict possible AB spatial data, so L channel data should be separated from Lab data. Then subtract the mean value from the obtained data to obtain the input data required by the model.

Model reasoning

In the reasoning part of the model, the key processes are described as follows:

1. Call aclmdlCreateDesc to create a data type that describes basic model information.

modelDesc_ = aclmdlCreateDesc();

2. Call the aclmdlGetDesc interface to obtain basic model information according to the model ID returned during model loading. aclError ret =aclmdlGetDesc(modelDesc_, modelId_);

3. Prepare input and output data for model reasoning;

4. Perform model reasoning and call aclmdlExecute interface; aclError ret =aclmdlExecute(modelId_, input_, output_);

5. Obtain the results of model reasoning for subsequent processing.

6. Release memory. Call the aclrtFree interface to free memory on the Device. Releases data of related data types. After the model reasoning is complete, the aclDestroyDataBuffer and aclmdlDestroyDataset interfaces need to be called in time to release the data describing the model input, and the aclDestroyDataBuffer interface needs to be called first. Then call the aclmdlDestroyDataset interface. If there are multiple inputs and outputs, call the aclDestroyDataBuffer interface multiple times.

Post-processing part

Image post-processing part, using OpencV. The predicted ab spatial data is deduced from the model.

Firstly, the obtained data is resized back to the original image size, and then combined with the L-channel data of the original image, namely, the complete Lab image is obtained.

The Lab image can be converted back to BGR format to save as A JPEG image, and the colored image can be obtained.

Black and white image coloring is an art, but also a technology

Centerm AI “Black and White Image Coloring” application is now online, just a click, can be colored!

Click on the online experiment and learn the coding process through jupyter online experiment.

In this case the address: www.hiascend.com/zh/develope…

In this era, a new color can represent a new mood, a new perspective.

Monotonous black and white seems to have been insufficient to meet our needs ~

Let AI technology take you to interpret different beauty ~

At the same time, click online experiment, you can also experience how the application is developed with CANN;

The source code of the application in Gitee and Github are open source, click on the above Github, Gitee direct source repository, we can be based on the source code for secondary development.

More application cases and source code:

The application case address: www.hiascend.com/developer/c…

Source code address: gitee.com/ascend/samp…

Example illustration of black and white image coloring

Open the website can not only experience a second of coloring joy ~~

You can also access the open source, learn as you go, and play on your own Ascend devices!

The persistence of technology is changing the world, and we can’t change the world, but we have to experience all the good things

Simple and fun, this web app is great for coloring old photos, black and white, and all kinds of monochromatic photos!

Click to follow, the first time to learn about Huawei cloud fresh technology ~