Article from arXiv by Mingliang Xu et al., Heart of Machine

Mobile phone QR code is too common, change or not good. Why not customize one you like? Recently, zhengzhou University, Zhejiang University, Microsoft Research Asia, Beijing University of Aeronautics and Astronautics researchers released a paper, proposed a method to design personalized art style TWO-DIMENSIONAL code, through three steps to customize unique two-dimensional code, and can maintain the success rate of scanning.
With the continuous popularization of the Internet and smart mobile devices, TWO-DIMENSIONAL code (Quick Response Code) has become one of the most widely used information carriers in the world. The average QR code doesn’t look good, with monotonous black/white coding blocks that the human eye can’t read. Recently, two-dimensional code visual beautification in the academic and industry rise a wave of upsurge. As shown in Figure 1, the current methods can be divided into four categories: I) embedded [1] — [5], embedding images to cover additional bits in the TWO-DIMENSIONAL code; Ii) Deformation type [5], change the shape/color of two-dimensional code module, such as transforming the square into a circle, triangle or star; Iii) Manual [5], from manual design and rendering; Iv) Fusion [6] — [12], image fusion into two-dimensional code. Among them, the fusion approach has the best visual effect and the most eye-catching.

Figure 1:4 existing methods to beautify the two-dimensional code.



Although the existing fusion type method can improve the visual quality of TWO-DIMENSIONAL code, there are still the following aspects to be improved: (1) Diversified and personalized, the mainstream method by changing the fusion image to generate different appearance of the art TWO-DIMENSIONAL code, but in fact, users want to unique fusion image (such as LOGO, personal photos or trademarks) to generate diversified and personalized art two-dimensional code; (2) Artistic quality. On the one hand, most methods define “beauty” as “more similar to the fusion image”, that is, the ideal result is that the original image is not “dominated” by artistic elements. On the other hand, the existing fusion type two-dimensional code usually directly and mechanically combines the image with the black/white coding module, resulting in the immovable and monotonous appearance of the coding module, even if the image fusion is very beautiful (See Figure 4(a)-(d)). (3) Robustness. It is not easy to integrate artistic elements into TWO-DIMENSIONAL code without affecting readability. Most existing methods lack error correction mechanism to ensure the robustness of results, which leads to the two-dimensional code being decoded incorrectly.

Figure 2 :(a)(b) the method described in this paper directly beautifies the baseline art qr code and generates the art style qr code. (c)(d) If the traditional fusion type approach is used to generate art-style results, beautiful images must be combined with black/white coding modules. But basically, the results of (c)(d) are still classified as baseline art QR codes which, although incorporating art images, are still not true art style QR codes. Compared with steps (c) and (d), the proposed method has two advantages :(1) the generation steps are reduced. (2) Further integrate modules and fusion images into one style to enhance visual appeal (See Fig.4 (e)-(L)).



Solving these problems without sacrificing other features is a big challenge. Fortunately, we found an effective solution by migrating the network based on CNN style. This paper proposes a method to automatically generate robust art style QR Code — SEE QR Code. As for question (1), as shown in Figure 3, our SEE QR Code is a style-oriented artistic QR Code. Users can generate QR codes of various artistic styles by embedding a single image. Therefore, users can make personalized choices according to their own preferences and needs. For question (2), our SEE QR Code should be called a work of art, not just a stylization of the original image; In addition, the method can directly stylize the baseline art QR code (See FIG. 2(a)(b)), and simultaneously give artistic elements to the coding module and the fusion image to enhance its visual appeal (See FIG. 4(e)-(L)). For problem (3), we designed an error correction mechanism to quantify and balance two competing elements, visual quality and readability, to achieve robust results.

Figure 3: Some examples of SEE QR codes that look like works of art; And users can generate different artistic styles of beautification images based on a single background image.

Figure 4 :(a)-(d) coding modules in the traditional method results (see figure 2). (e)-(l) the coding module in the result of our method. It can be seen that the code module in SEE QR code also has attractive artistic elements.

The above existing work only focuses on the first phase of two-dimensional code beautification, which is to generate aesthetic two-dimensional codes by changing the embedded image (see Figure 2(a)(d)). In contrast, our approach primarily beautifies the baseline aesthetic of the QR code (see Figure 2(b)). To sum up, our main contributions are:

  • We propose an aesthetic two-dimensional code of art style — SEE two-dimensional code, which is superior to existing methods in the following aspects: diversity and individuality, aesthetic quality and robustness.
  • We designed an efficient algorithm that prioritized module arrangements in the baseline art QR code to minimize visual contrast between black/white coded modules and fused images.
  • We adjusted the style migration network to accommodate the baseline art QR code as the content target, which not only effectively avoided the visual impact of coding blocks like noise, but also reduced the number of error blocks caused by the network.
  • We propose an error correction mechanism based on iterative updates, and ensure the robustness of qr codes by balancing two competing elements: visual quality and readability.
The structure of the whole system is shown below. In Stage A, we combine the two-dimensional code of pictures and coded information to generate the artistic two-dimensional code. Then, the style to be transferred is combined with the previously generated art TWO-DIMENSIONAL code, and error correction is made with the extracted information encoding in Stage C to improve the decoding robustness.



Figure 6: Method overview. Stage A: Two-dimensional code of generative art; Stage B: Style transfer; Stage C: Error correction.

Each stage of the system and the experimental results will be briefly introduced later, so we need to understand the symbolic meaning described in Table 1:

Stage A: Two-dimensional code of generative art

A. Methodology overview

As shown in Figure 6, our method is divided into three steps: Stage A, Stage B, and Stage C. In Stage A, we generated an optimized art QR code Q_a. In Stage B, we stylize Q_a using an adjusted neural style transfer network and output a non-robust art style QR code Q_b. Finally, in Stage C, we repaired the readability of Q_b through error correction mechanism to obtain a robust result Q_c. These three basic steps are described below.

Figure 7: Flowchart for Stage A. We set the priority of the variable module according to the gray distribution of the fused image to minimize the visual contrast between the fused image and the noise-like black and white module, and finally output the baseline art QR code Q_a.

Based on the coding rules of RS error correction code, the TWO-DIMENSIONAL code is expressed as a square coding module. [13] proves that we can use gauss-Jordan elimination to modify the color of modules (i.e., black or white) within a limited range without considering machine readability. Mainstream studies ([7] [8] [10]) usually use the method in [13] to select variable modules by considering local visual features of fused images (such as saliency map, edge map or region of interest). In Stage A, we propose an efficient strategy to set the priority of variable modules, that is, select according to the global features of the mixed image I, and finally output the baseline art QR code Q_a, which minimizes the visual contrast between image I and noise-like black and white modules.

As for the gray scale of the fused image I^g, the gray scale value of each pixel is in the interval [0, 255], and the gray scale value of the black and white modules is the constant value 0 and 255 respectively. Therefore, we consider that visual contrast is minimal when the color of the module is closest to the corresponding region in image I. In other words, assigning high weights to the blackest/whitest regions in image I corresponding to the black and white modules will greatly optimize Q_a’s visual perception. Based on this, Stage A is divided into two steps: calculation of weight matrix and image fusion.



Stage B: Style migration

The style transfer architecture used in this paper is shown in Figure 8 below. We generally use the framework proposed by J. Johnson et al. [16] for Yan. For style migration, we input the target image A_C and style image a_S of the style to be migrated, and the output A hat should combine the content of A_C and style features of A_S.

Figure 8: Flowchart for Stage B. We roughly follow the style transfer system proposed in [16], and further adjust the content reconstruction layer and features of the loss network φ, so that black and white squares like noise are no longer used to enhance the style and aesthetic features of the TWO-DIMENSIONAL code.

In Stage B, the baseline art QR code Q_a has very dense black and white coding modules as the content that needs to be migrated, which requires us to solve two problems: 1) For stronger robustness, the number of error modules generated should be minimized; 2) For the sake of visual quality, it is very important to avoid the visual impact of noise-like modules. Therefore, in order to solve these two problems, we made further modifications to the network architecture of content reconstruction layer and style reconstruction layer as follows:

Figure 9: Similar to the framework proposed by L. A. Gatys[14] and J. Johnson[16] et al., we reconstructed style features from pre-trained VGG-16 loss networks, The above shows the reconstruction results of style characteristics of RELU1_2, RELU2_2, RELU3_3 and RELU4_3 layers, respectively. In addition, we find that the low-level reconstruction style features are very similar to the dense coding modules of the art QR code.

Table 2: Adjustment of feature reconstruction layer.



Stage C: Error correction

Although we significantly optimized the problem of weak robustness in Stage B, there are still a small number of error-modules in Q_b. Therefore, at Stage C, we designed an error correction mechanism to detect and correct the error module of Q_b by balancing robustness and visual quality, so as to generate a robust result Q_c.



B robustness evaluation of coding module

Figure 10 :(a) in the sampling stage, the collected pixels may be affected by factors that are not directly related (such as image scaling, tilt Angle, etc.) and lead to errors, while the essential influencing factor is the size of the coding module. (b) In the thresholding stage, sampled pixels may be incorrectly threshed by factors that are not directly related (e.g. illumination, light color, etc.), while the essential influencing factor is the color of the coding module.



C error correction mechanism

Figure 11: Flowchart for Stage C. (a) Detect and calibrate non-robust modules iteratively, and update thresholds until each module becomes robust. (b) Preprocess non-robust modules by creating coding points. (c) Convert grayscale robust art style QR code to RGB space.

Figure 12 :(a) the specific step of figure 11 (b) preprocessing non-robust modules in Q_b, where the local average color is calculated by a simple method as the color of C_k. (b) The specific steps of modifying a grayscale non-robust coding module. (c) Failed to pass step (a) processing generated Q_c, that is, correction points in Q_c may have very serious visual distortion.

Figure 13: Some examples of two locators. One is a standard locator consisting of black and white, and the other is a color locator after style migration and error correction.



The experiment

FIG. 14COMPARISON of experimental results between Q_A and TS (based on two-stage generation results in [10]). Our results focus on the global features of the background image I, and the black and white coding module tries to allocate the deeper/shallower region of the image I to minimize the visual contrast between the module and the image I.

Figure 17: The initial style migration system [16] with our adjusted results. As a result of the initial style transfer, irregular color changes usually occur in large areas, which can be very visual and easily lead to decoding failures. Our adjustments significantly improve issues such as decoding robustness and visual quality.

Table 4: Qr code scanning success rate on different mobile devices.



Thesis: Stylize Aesthetic QR Code









Links to papers: arxiv.org/pdf/1803.01…

Abstract: With the continuous development of smart mobile devices, the use of TWO-DIMENSIONAL code is more and more widespread. Existing research attempts to beautify the appearance of QR codes and has developed a range of related technologies. However, there is still room for improvement in such studies as visual diversity, aesthetic quality, flexibility, generality and robustness. In order to solve these problems, this paper proposes a new method to beautify the TWO-DIMENSIONAL code, SEE (Stylize aEsthEtic), which can automatically generate robust artistic style two-dimensional code in only three steps. Specifically, as the first step, we propose a method to generate optimized baseline art QR codes that reduce the visual contrast between noise-like black and white modules and fused images. Second step, to obtain the TWO-DIMENSIONAL code of art style, we adjust an appropriate neural style transfer network to add some abstract art elements to the baseline art two-dimensional code. Third, we designed an error correction mechanism to ensure robust performance by balancing two competing elements: visual quality and readability. A large number of experiments have proved that SEE QR codes maintain high quality in terms of appearance and robustness, while providing users with more personalized choices.