The author | Jasmeet Bhatia

Compile | KK4SBB


This article provides an overview of the performance of several popular kerAS-supported deep learning frameworks, including Tensorflow, CNTK, MXNet, and Theano.Jasmeet Bhatia is a data and ARTIFICIAL intelligence architect at Microsoft. The opinions expressed in this article are personal.

If anyone is questioning Keras’s standing in the data scientist and engineer community right now, ask him to look at professional AI and cloud computing players’ support for Keras. The latest official release of Keras already supports Google’s Tensorflow and Microsoft’s CNTK deep learning library, in addition to general libraries such as Theano. Last year, Amazon Cloud announced support for another powerful tool, MXNet.

Just a few weeks ago, a new version of MXNet was also compatible with Keras. However, as of now, MXNet only seems to support Keras v1.2.2, not the latest version 2.0.5.

While Keras models can be deployed as long as the product uses a supported back-end service, developers and architects must keep in mind that Keras is a high-level interface for different deep learning frameworks and does not allow arbitrary adjustment of model parameters from external libraries.

Therefore, if you want to fine-tune all the parameters provided by the backend framework, the authors recommend that you go straight to the specific framework instead of wrapping it with Keras. But over time, Keras’s features will definitely improve. But we have to say that Keras is still a gem in the early stages of development of deep learning projects, allowing data scientists and engineers to quickly generate and test complex deep learning models.

Keras also allows developers to quickly compare and test the relative performance of several deep learning frameworks. The configuration file for Keras has a parameter that specifies what backend framework to use. As a result, you can write one piece of code and run it once on Tensorflow, CNTK, and Theano without changing the code.

As for MXNet, since it only supports Keras v1.2.2 right now, you need to modify the code slightly to run it. Obviously, these individual frameworks can be further tuned to take advantage of the various features in the development library, but Keras provides an opportunity to compare the underlying performance of these frameworks.

Several people have written about the relative performance of the Keras backend support, and with each new release of Keras or the backend framework, we see a huge improvement in performance.

So let’s see what the latest versions of Keras and the various back-end frameworks can achieve.


First, introduce the hardware configuration of this experiment.

All testing was done on an Azure NC6 virtual machine with an NVidia Tesla K80 GPU. The vm image file is Data Science Virtual Machine (DSVM) Azure. The image file comes preloaded with Keras, Tensorflow, Theano, MXNet, and other data science tools. For testing purposes, all applications were upgraded to the latest version, using Keras v1.2.2 for MXNet. Details on Azure DSVM can be found at the link


configuration

Due to the different dependencies of each framework, the authors tested three different configurations, as follows:

Although all the frameworks’ development teams claim to be working on new versions with improved performance that can be used for research projects, actual products tend to use stable versions. As a result, all backend frameworks for this test used the latest stable versions.

The performance test

To compare the performance of each framework, the authors use the following five deep learning models. To ensure the fairness of the experiment, all the models are from the sample code of the Keras project on Github.

The author posted all the test code on his Github page.

It is worth noting that two of the tests did not include the MXNet model. Because MXNet does not support the function of the latest version of Keras, it needs to change a lot of code to run, so it is excluded from this experiment. The other three experiments support MXNet with only a few changes, mainly in the naming of the functions.


Experiment 1: CIFAR10 CNN

  • Model type: Volume computer network

  • Dataset/task name: CIFAR10 image dataset

  • Goal: Categorize images into 10 categories

Tensorflow is slightly faster than MXNet in completing each epoch. CNTK was slightly ahead at iteration 25 in terms of accuracy/convergence rate, but after iteration 50 all frameworks were nearly identical in accuracy.

Performance comparison chart of different frameworks


Experiment 2: MNIST CNN

  • Model type: Volume computer network

  • Dataset/Task name: MNIST handwritten digital dataset

  • The goal: To recognize handwritten numbers in photographs

In this group of experiments, the training speed of Tensorflow is slightly faster than that of Theano and far better than that of CNTK, but the accuracy and convergence speed of all frameworks are equally.



Experiment 3: MNIST MLP

  • Model type: multilayer perceptron/neural network

  • Dataset/Task name: MNIST handwritten digital dataset

  • The goal: To recognize handwritten numbers in photographs

This experiment also tested the standard deep neural network model on the MNIST data set. CNTK, Tensorflow and Theano had roughly the same speed (2.5-2.7s/epoch), while MXNet was unique and reached the speed of 1.4s/epoch. MXNet also performs slightly better in terms of accuracy and convergence speed.

Experiment 4: MNIST RNN

  • Model type: Hierarchical cyclic neural network

  • Dataset/Task name: MNIST handwritten digital dataset

  • The goal: To recognize handwritten numbers in photographs

In this set of experiments, the training speed of CNTK and MXNet was relatively close (162-164s/epoch), while Tensorflow was slightly slower than them, only 179s/epoch. Theano’s performance on the RNN model is pretty poor.

Experiment 5: BABI RNN

  • Model type: Recurrent neural network

  • Data set/task name: bAbi project

  • Objective: To train two recurrent neural network models based on description and problem. The generated vectors are used to answer a series of bAbi tasks.

MXNet did not participate in this experiment. The performance of Tensorflow and Theano is similar, and CNTK is 50% faster at 9.5s/epoch.

conclusion

Performance comparison of different frames in each group of experiments

  • Tensorflow performed well in each group of CNN model experiments, but performed poorly in RNN model.

  • CNTK performs far better than Tensorflow and Theano in BAbi RNN and MNIST RNN experiments, but not as well as Tensorflow in CNN experiments.

  • MXNet performs slightly better than CNTK and Tensorflow in RNN tests, and outperforms all other frameworks in MLP tests. However, due to the limitations of Keras V2 version, I could not participate in the other two groups of comparison experiments, but this situation will be resolved soon.

  • Theano performs slightly better than Tensorflow and CNTK in MLP experiments.

conclusion

Based on the results of the experiment, all frameworks have their own areas of expertise, and no one framework can completely crush the others. However, participants in the open source community continue to develop and maintain these products, improving their performance and expanding their capabilities to make use and deployment easier for users. However, performance is only one aspect of the deep learning model, and users need to consider multiple factors such as effectiveness and convenience when using it in practice.

译 文 : Search fastest Keras Deep Learning backend