A list,

Visible and infrared image fusion is a branch of image fusion. It is used in many fields, such as visible and infrared image fusion tracking (RGB-T tracking), face recognition, human ear recognition, SLAM, defense fields. Therefore, it has been an active research field for many years.

In terms of methods, it mainly includes methods based on spatial domain and transformation domain.

The method based on spatial domain refers to the method that directly operates the source image in the spatial domain to obtain the fusion image, mainly including pixel-based, block-based and region-based methods.

The method based on transform domain refers to the process of first transforming the source image to a certain transform domain, then performing image fusion in the transform domain (generally in the form of coefficients), and finally obtaining the fused image by using inverse transformation. Commonly used transformations include multi-scale transformation (such as wavelet transform), compressed sensing, sparse representation, etc.

In recent years, with the development of deep learning, deep learning technology has also been introduced into the field of visible and infrared image fusion. Some deep learning models, including CNN and GAN, have been applied to the field of visible and infrared image fusion and achieved good results.

However, since the research direction of visible and infrared image fusion appeared, there has been a major problem: the lack of benchmark. This is mainly reflected in the following aspects:

First, there is no uniform data set. Although some data sets are relatively common, there is no standard or consensus, so inconsistencies in test images are common in the literature. Most of the time, It is Joe who uses picture A to test performance, and Joe who uses picture B to test performance.

Second, there is no unified evaluation index. Different from other fields with ground truth such as target tracking, there is generally no ground truth in the field of image fusion, resulting in no standard answer for the evaluation of fusion results. Therefore, the performance evaluation in the field of image fusion is generally classified into two parts: qualitative evaluation and quantitative evaluation.

Qualitative evaluation is also called subjective evaluation, which is to observe the quality of fused images by human eyes. Although qualitative evaluation is very important, it is obviously too subjective, after all, “everyone is difficult to adjust”, and everyone’s evaluation standards are different. Quantitative evaluation refers to the use of some evaluation metrics to evaluate the fusion image. However, as there is no standard answer mentioned above, there is no unified evaluation index at present. In fact, so far, researchers have devised no fewer than 30 different indicators to evaluate the quality of the fused images. However, there is no best indicator among these indicators. The evaluation effect of indicators is very dependent on the test image, and there are contradictory phenomena among indicators. Therefore, it is common practice in the literature to present the results of only a few sets of indicators that are favorable to one’s own algorithm.

Due to the above two reasons, in the field of visible and infrared image fusion, there is no unified benchmark to objectively and comprehensively evaluate the algorithm performance. At present, the literature is basically “every woman sells her melon and every man boasts himself”. Considering these situations, as well as the widespread existence of benchmarks (such as OTB, VOT, etc.) in target tracking and other fields, we believe that it is very necessary to make benchmarks in the field of visible and infrared image fusion.

Ii. Source code

%% section I: Read source images(Read source image)
clear all
set(0.'defaultfigurecolor'.'w') 
DistortFlag = 0; %input('Is there distortion of infrared image? :\n'); [I1gray, I2gray, I1rgb, I2rgb, F1, f2, path] = cp_readImage; . % (0.'F:\'['I' num2str(infrared) '.jpg'], ['V' num2str(visible) '.jpg']); %% section II: Resize images based on the minimum imaclosege height height = size(I1gray,1); [I1, I2, scale] = cp_resizeImage(I1gray,I2gray,height); %% section III: Registrate iteratively & Coarse matching clc; I1_itea = I1; iterationNum =1;
iteration = 0;
Runtime = 0;
maxRMSE = 4*ceil(size(I2,1) /300);
AffineTrans = zeros([3 3 iterationNum]);
while  iteration < iterationNum
    fprintf('\n%d(th) iteration of registration... \n',iteration);
    [P1,P2, Rt,corner12] = cp_registration(I1_itea,I2, 20, maxRMSE,iteration, 1.0.6.1    ,I2gray);
                        % cp_registration(I1,    I2, theta,maxRMSE,iteration,zoom+,zoom-,Lc,showflag,I2gray)
    Runtime = Rt + Runtime
    [I1_itea,affmat] = cp_getAffine(I1_itea,I2,P1,P2); % [v1,u1]==[v2,u2]
    iteration = iteration+1; AffineTrans(:,:,iteration) = affmat.T; End % Points of I1gray after resize P1 = [P1 ones([length(P1))1])];
[pos_cor1,~] = find(corner12(:,1) = =0);
for iteration = iteration:- 1:2
    P1 = P1 / AffineTrans(:,:,iteration- 1);
    cor12 = [corner12(1:pos_cor1- 1.1:2) ones(pos_cor1- 1.1)] / AffineTrans(:,:,iteration- 1);
    P1(:,1:2) = P1(:,1:2) ./ P1(:,3);
    P1(:,3) = ones(length(P1),1);
    corner12(1:pos_cor1- 1.1:2) = cor12(:,1:2) ./ cor12(:,3);
    corner12(1:pos_cor1- 1.3) = ones(pos_cor1- 1.1);
end
P1 = P1(:,1:2);
corner12 = corner12(:,1:2);
% Correct matches in the source images
P1(:,2) = size(I1gray,1) / 2 + scale(1) * ( P1(:,2)-size(I1,1) /2);
P1(:,1) = size(I1gray,2) / 2 + scale(1) * ( P1(:,1)-size(I1,2) /2);
corner12(1:pos_cor1- 1.2) = size(I1gray,1) / 2 + scale(1) * ( corner12(1:pos_cor1- 1.2)-size(I1,1) /2);
corner12(1:pos_cor1- 1.1) = size(I1gray,2) / 2 + scale(1) * ( corner12(1:pos_cor1- 1.1)-size(I1,2) /2);
P2(:,2) = size(I2gray,1) / 2 + scale(2) * ( P2(:,2)-size(I2,1) /2);
P2(:,1) = size(I2gray,2) / 2 + scale(2) * ( P2(:,1)-size(I2,2) /2);
corner12(pos_cor1+1:end,2) = size(I2gray,1) / 2 + scale(2) * ( corner12(pos_cor1+1:end,2)-size(I2,1) /2);
corner12(pos_cor1+1:end,1) = size(I2gray,2) / 2 + scale(2) * ( corner12(pos_cor1+1:end,1)-size(I2,2) /2);
%% section IV: Fine matching
P3 = cp_subpixelFine(P1,P2); % Fine matching
%% section V: Show visual registration result
[~,affmat] = cp_getAffine(I1gray,I2gray,P1,P3);
Imosaic = cp_graymosaic(I1gray, I2gray, affmat);
figure, subplot(121),imshow(Imosaic); subplot(122),imshow(cp_rgbmosaic(I1rgb,I2rgb,affmat));
cp_showResult(I1rgb,I2rgb,I1gray,I2gray,affmat,3); % checkborder image
cp_showMatch(I1rgb,I2rgb,P1,P2,[],'Before Subpixel Fining');
cp_showMatch(I1rgb,I2rgb,P1,P3,[],'After Subpixel Fineing');
% imwrite(cp_rgbmosaic(I1rgb,I2rgb,affmat),['D:\' f1(1:end4 -) '_Mosaic.jpg']);
%% Obtain reference transformation matrix manually
Copy the code

3. Operation results





Fourth, note

2014a