Recently, the author has been studying the technology of front-end visualization and construction. Recently, I also encountered a very interesting topic, which is to automatically extract picture information based on design draft to intelligently generate code. Of course, this article will not introduce many obscure technical concepts. I will introduce how to realize some interesting functions by using Canvas image recognition technology from several practical application scenarios. Finally, I will summarize some thoughts on intelligence and planning of low code direction, hoping to inspire you.

Canvas image recognition technology

Those who are familiar with the front end may be familiar with Canvas. Next, I will take you to realize the following application scenarios to deeply understand the canvas image recognition technology.

  • Based on the image dynamic generation site color and gradient
  • Create a one-click website color scheme based on images/sketches
  • Image recognition technology scheme

Based on the image dynamic generation site color and gradient

Perhaps some friends will ask, based on the picture dynamic generation of site color and gradient, what can it solve the problem? What are the application scenarios? Here the author gives a few practical application examples.

Netease cloud music we may not be unfamiliar, careful friends may be able to observe the sitebannerPart of the background is not good andbannerForm a good unity?

We will find that the background of each image in rotation is based on the current image color gradient or blur, to achieve perfect alignment with the image in rotation. There are many similar examples, such as the background of photo websites, the background of picture cards, all use similar technology.

Realize the principle of

We know that canvas objects have three methods:

  • createImageData()Create a new, blankImageDataobject
  • getImageData()returnImageDataObject that copies pixel data for the specified rectangle on the canvas
  • PutImageData () puts the ImageData(from the specified ImageData object) back onto the canvas

To analyze ImageData, we need to use the second method above, getimagedata.imagedata object, which is not an image, defines a part of the canvas (rectangle) and holds information about each pixel within that rectangle. For each pixel in the ImageData object, there are four bits of information, the RGBA value:

  • R – Red (0-255)
  • G – Green (0-255)
  • B – Blue (0-255)
  • A-alpha channel (0-255; 0 is transparent, 255 is fully visible)

The color/alpha information exists as an array and is stored in the Data property of the ImageData object.

With the above technical basis, it is completely possible for us to extract the color information of the picture, and analyze the main color of the picture. So our implementation process is as follows:

The reference code is as follows:

img.onload = function () {
    ctx.drawImage(img, 0.0)
    img.style.display = 'none'
    // Get pixel data
    let data = context.getImageData(0.0, img.width, img.height).data
    // imagedata. data is a one-dimensional array of the type of Uint8ClampedArray. Each of the four array elements represents the RGBA information of a pixel point, and the value of each element is between 0 and 255
    let r = 0,
        g = 0,
        b = 0
        
    // Take the average of all pixels
    for (let row = 0; row < img.height; row++) {
        for (let col = 0; col < img.width; col++) {
            r += data[(img.width * row + col) * 4]
            g += data[(img.width * row + col) * 4 + 1]
            b += data[(img.width * row + col) * 4 + 2]}}// Calculate the average value
    r /= img.width * img.height
    g /= img.width * img.height
    b /= img.width * img.height

    // round the result
    r = Math.round(r)
    g = Math.round(g)
    b = Math.round(b)
    
    // Set the gradient for the background
    bgBox.style.backgroundImage = `linear-gradient(rgb(${r}), rgb(${g}), rgb(${b}) `;
  }
Copy the code

It is worth noting that, depending on the range scenario, we can also use other algorithms such as:

  • Average value algorithm (to obtain the main tone)
  • Median segmentation (to get the primary color of a PNG image)
  • Complementary color calculation method

Create a one-click website color scheme based on images/sketches

Now that we’ve introduced the canvas color scheme, let’s take it a step further and explore how to create a one-click website color scheme based on images/artwork.

In fact, based on the above example we can fully achieve a set of website color generation tools, here in order to save time, the author recommends a more powerful plug-in, to help us achieve similar functions.

That’s right, ColorThief. It supports browsers and Node environments, so as a front end, it’s very easy to use it and get the color scheme of the art/artwork.

Github Portal: Online generation of image color scheme library

A simple usage example is as follows:

import ColorThief from './node_modules/colorthief/dist/color-thief.mjs'

const colorThief = new ColorThief();
const img = document.querySelector('img');

if (img.complete) {
  colorThief.getColor(img);
} else {
  image.addEventListener('load'.function() {
    colorThief.getColor(img);
  });
}
Copy the code

The library also has many detailed apis, such as control of the quality of production, granularity and so on, which we can use to make some smarter tools.

Image recognition technology scheme

Image recognition technology can help technicians to process and analyze images by computer and better identify targets with different patterns. The process and content of image recognition are more, mainly including image pretreatment and image segmentation and other content, it is effective in the application of image processing, but also according to the characteristics of the image of its judgment and matching, so that users can more quickly search the information they want to obtain in the picture.

Those of you who know about neural networks may know that the real solution for image recognition technology is convolutional neural networks (CNNs or ConvNets).

In terms of image recognition technology, convolutional neural network filters unnecessary connections according to the degree of association, thus making the image recognition process more computationally operable. Convolutional neural networks intentionally limit the connections in image recognition, allowing one neuron to only accept input from small segments of previous layers (say 3×3 or 5×5 pixels), avoiding excessive computational burden. As a result, each neuron is responsible for processing only a small part of the image.

Of course, as front end engineers, we probably don’t go that far, but don’t worry, there are a lot of tools out there to help us solve the underlying analysis problems. Imgcook, for example, uses recognition technology to generate HTML code that can be consumed by browsers.

Its working mechanism is as follows:

The underlying recognition technology is also based on the analysis, extraction and transformation of image information elements to achieve the purpose of intelligent arrangement. There are also open source libraries that can help us achieve a certain level of recognition. We can build more intelligent tools for development based on these solutions.

Here I put forward a picture recognition library gocr. js, for your reference.

Gocr.js is a pure JavaScript version of the GOCR (open source OCR optical recognition program) project, using Emscripten for automatic conversion. This is a simple OCR (Optical Character recognition) program that can scan images from text to text.

The library is also very simple to use, we just need to import the library, enter the following code:

var string = GOCR(image);
alert(string);
Copy the code

The demo is as follows:

Intelligent thinking

In recent years, lowcode and Nocode platforms have developed rapidly at home and abroad, and the basic construction can no longer meet the needs of science and technology enterprises. Intelligent/automated construction platforms keep emerging. In the previous article, the author shared 10 top lowcode development platforms abroad in 2021, and also introduced many excellent lowcode platforms abroad, many of which have a lot of practical implementation of intelligence. The author simplified it as follows:

Recently, the h5-Dooring visual editor has been continuously iterated. The data source has been basically built, and more intelligent direction will be followed. The first version of visual large screen V6.

The domestic Lowcode platform still has a long way to go, look forward to working together 💪!

More low code/visualization related technology sharing and implementation, welcome wechat search interesting front-end learning exploration.