Illustration Research: Using deep learning to solve the problem of simplified illustration generation from freehand sketches

Illustration is an important part of the image processing challenge. Not only are the sketches sparse and noisy, but there are also huge differences in painting styles. In addition, real-world usage scenarios require very precise results. Illustration research focuses on solving these problems and helping illustrators in the content creation process. This paper mainly introduces the work of the two papers. Edgar Simo-Serra, assistant professor at Waseda University in Tokyo, Japan. The project is supported by the Japan Science and Technology Agency (JST).

1.Sketch Simplification

Essay: Learning to Simplify:Fully Convolutional Networks for Rough Sketch Cleanup

Sketch Simplification

Presented at SIGGRAPH, class A conference on Computer Graphics and Multimedia, 2016

Abstract

This paper presents an overall framework for training a sketch simplification network that can turn challenging sketches into clear line drawings. The clumsy approach is to extend a simplified network with a discriminator network, train the two networks jointly so that the discriminator network can tell whether a line graph is real training data or the output of the simplified network, and in turn try to fool the network. This approach has two main advantages :(2) because the discriminator network learns its structure on the line diagram, it can encourage the output sketch of the simplified network to look more like the training sketch. (2) we can use the additional unsupervised data to train the network, by adding not mutually corresponding rough sketches and gantt chart, improving the quality of sketch a simplified, because of the difference of architecture, this article method when faced with a similar confrontation training method in the stability of the training and the use of unsupervised training data shows advantage ability. Although we use the same structure for reasoning, this paper still shows that the performance of its structure is significantly better than that of the existing state-of-the art method for training models in sketch simplification tasks. In addition, this paper proposes a method to optimize the performance of a single graph, which takes advantage of the extra computation time cost to improve the accuracy. Finally, this paper shows that by using the same structure, the network can be trained to perform inverse transformation problems, converting simplified line drawings to pencil sketches, which cannot be achieved by using standard Mean squared error loss.

Model structure

Detailed table of network structure



Compare that to a business application

Computation time



Results of different scenarios

Compare with the current state-of-the-art method

limitations

conclusion

This paper presents a novel automated end-to-end system that can obtain rough rasterized sketches and output high quality vector simplifications. The model in this paper is based on push-down convolution operations to improve efficiency and is able to handle very challenging pencil and paper scan images from a variety of sources. In addition, the fully convolutional structure proposed in this paper is optimized for simplifying tasks and can process images of any resolution. This paper also carefully designs a new data set for simplifying this task, which, combined with the learning method in this paper, can be used to teach our model to simplify the sketch. The method in this paper is fully automatic and does not require user intervention. The results show that the performance of the proposed method exceeds that of the current state-of-the-art sketching simplification method, although it does not share the serious limitation that only vector images can be processed within the computation time of less than 1s. We also confirmed with user studies that the proposed model can achieve better performance than currently commercialized vector software. This approach is an important step in being able to integrate sketch simplification into an artist’s daily workflow.

2.Mastering Sketching

Abstract

This paper presents a novel technique to simplify sketching by learning a series of convolution operators. In contrast to existing methods that require vector input, this article allows more general and challenging rough raster sketches to be entered, such as rasterized sketches obtained from scanned pencil sketches. This article transforms the rough sketch into a simplified version, and then changes the vectorization. These operations are all automated and do not involve user interaction. The model in this paper adopts full convolutional neural network, which is different from most existing neural networks in that it can process images with any size and aspect ratio as input, and output simplified sketches with the same dimension as the input image. In order to teach our model to simplify, this paper presents a new data set whose rough and simplified sketches are paired. By using our proposed data set efficiently and using the convolution operator, we can train our sketch simplification model. The method in this paper overcomes many limitations of existing models, such as input vectorized images and long calculation time. At the same time, this article shows that significant simplifications can be achieved for many different test cases. Finally, a user study is used to verify the structure, in which the structure outperforms similar methods in performance and achieves state-of-the art performance in rasterized image sketching simplification.



Introduction to data set



Methods adopted in this paper





Discriminator network structure



The result comparison



The user study results

discuss

conclusion

This paper introduces adversity-enhancement for structural prediction and application of its path sketch simplification tasks and its inverse problems, such as pencil graph generation. We demonstrate that Loss enhancement of the standard model by monitoring against Loss results in a more realistic structured output. In addition, it is necessary to allow the same structure of unsupervised data enhancement for structured prediction tasks when the cost of obtaining additional annotation training data is very high. The proposed method has proved that it is impossible to generate pencil graphs from clear sketches using standard losses and is widely applicable to most structured prediction problems.

Experiment 3.