By Rachel Allen and Michael Li

We ranked 23 open source deep learning libraries for data science. The ranking is based on three equally weighted metrics: activity on Github, activity on Stack Overflow and Google search results.

The ranking results

Here’s a ranking of 23 open source deep learning libraries for data science, measured by activity on Github, activity on Stack Overflow, and Google search results. The table shows the standardized score, with a value of 1 representing one standard deviation above the mean (mean =0). Caffe, for example, is one standard deviation above the Github activity average, while Deeplearning4j is close to the average.

The methods are as follows

Results and discussion

The rankings are based on three equally weighted metrics: Github (stars and branches), Stack Overflow (tabs and questions), and Google results (overall and quarterly growth rates). These are obtained using the available apis. Making a comprehensive list of deep learning toolkits is tricky — in the end, we came up with five different lists that we thought were representative (see the methodology below for details). After calculating standardized scores for each metric, we can see which packages stand out in each category. Complete list here (https://github.com/thedataincubator/data-science-blogs/blob/master/output/DL_libraries_final_Rankings.csv), The raw data here (https://github.com/thedataincubator/data-science-blogs/blob/master/output/deep_learning_data.csv).

TensorFlow leads the way with the largest active community

Across all measures, TensorFlow was at least two standard deviations above the average. TensorFlow has nearly three times as many Github branches and more than six times as many Stack Overflow problems as Caffe, the next most popular framework. Originally opened source by the Google Brain team in 2015, TensorFlow has gained momentum over older libraries such as Theano (# 4) and Torch (# 8) to move to the top of our list. Although TensorFlow comes with a Python API that runs on a C++ engine, several libraries on this list can use TensorFlow as a back end, providing their own interfaces. These libraries include Keras (# 2, soon to be part of core TensorFlow) and Sonnet (# 6). TensorFlow’s popularity may be due to its combination of a general-purpose deep learning framework, flexible interfaces, clean-looking computational graphics visualizations, and Google’s vast developer and community resources.

Caffe has not been replaced by Caffe2

Caffe, which came in third on the list, has more activity on Github than all of its competitors (except TensorFlow). Caffe has traditionally been considered more specialized than Tensorflow, initially focusing on image processing, object recognition and pre-trained convolutional neural networks. Facebook launched Caffe2 (# 11) in April 2017 and Caffe2 is already in the top half of the deep learning library. Caffe2 is a more lightweight, modular and expandable Caffe that includes a circulating neural network. Caffe and Caffe2 are separate code bases, so data scientists can continue to use the original Caffe. However, some migration tools such as Caffe Translator provide a means to use Caffe2 to drive the existing Caffe model.

Keras is the most popular deep learning front end

Keras (no. 2) is the highest-ranked non-framework library. Keras can be used as a front end to TensorFlow (# 1), Theano (# 4), MXNet (# 7), CNTK (# 9), or Deeplearning4J (# 14). Keras outperformed average on all three measures. Keras’s popularity may be attributed to its simplicity and ease of use. Keras allows users to prototype quickly at the cost of a little less flexibility and control that comes with working directly with the framework. Keras is popular with data scientists who use deep learning on data sets. Keras is growing and gaining popularity thanks to R Studio’s recent release of a KerAS-oriented interface that uses R.

Even without a lot of industry support, Theano is still at the top

Of all the new deep learning frameworks, Theano (# 4) is the oldest library on this list. Theano pioneered the use of computational graphs and remains popular throughout the deep learning and machine learning research community. Theano is really just a numerical computation library for Python, but can be used with advanced deep learning packages like Lasagne (bit 15). Google supports TensorFlow (# 1) and Keras (# 2), Facebook supports PyTorch (# 5) and Caffe2 (# 11), while MXNet (# 7) is the official deep learning framework for Amazon Web Services (AWS), Microsoft designs and maintains CNTK (# 9), Though not officially endorsed by any of the tech industry’s giants, Theano remains popular.

Sonnet is the fastest growing library

In early 2017, Google’s DeepMind publicly released the code for Sonnet (# 6), a high-level object-oriented library based on TensorFlow. This quarter saw a 272% increase in the number of pages returned from Google search results for Sonnet compared to the previous quarter, the largest increase of any library we ranked. Although Google acquired DeepMind, a British AI company, in 2014, DeepMind and Google Brain still have largely separate teams. DeepMind focuses on strong ARTIFICIAL intelligence, and Sonnet helps users build neural networks based on specific AI ideas and research.

Python is the preferred language for deep learning interfaces

PyTorch (No. 5), a framework that uses Python for its single interface, is the second fastest growing library on this list. Google search results for PyTorch increased 236% compared to last quarter. Of the 23 open source deep learning frameworks and packaging libraries we ranked, only three do not use Python interfaces: Dlib (No. 10), MatConvNet (No. 20), and OpenNN (No. 23). The C++ and R interfaces are used in 7 and 6 of the 23 libraries, respectively. While the data science community is almost unanimous in its use of Python, there are many options for deep learning libraries.

limit

As with any analysis, we made decisions along the way. All source code and data are on our Github page. The complete list of deep learning libraries comes from several sources.

Of course, some older libraries will have higher metrics and therefore rank higher. The only metric that takes this into account is Google’s quarterly search growth rate.

The data raises several conundrums:

  • Neural Designer and Wolfram Mathematica are proprietary and have been removed

  • CNTK is also called Microsoft Cognition Toolkit, but we just use the original CTNK name

  • Nervana Neon was changed to Nervana Neon

  • Paddlepaddle instead of paddle

  • Some libraries are clearly derivatives of others, such as Caffe and Caffe2. If these libraries have a unique Github code base, we decide to treat them separately.

methods

All the source code and data on our making page (https://github.com/thedataincubator/data-science-blogs).

We started by making a list of 23 open source deep learning libraries from five different sources, and then collected metrics for all libraries to come up with rankings. The Github data is based on stars and branches, the Stack Overflow data is based on tags and questions containing package names, and the Google results are based on the total number of Google results over the past five years and the quarterly results growth rate over the past three months compared to the previous three months.

Some other things to watch out for:

  • Several libraries are common terms (caffe, Chainer and Lasagne), so the search terms used to determine the number of Google results include the library name and the term “deep learning.”

  • Any missing Stack Overflow counts are converted to zero.

  • The counts were normalized, with a mean of 0 and a deviation of 1, then averaged to get Github and Stack Overflow scores, and finally combined with the search results to get the total score.

  • Do a manual check to confirm the location of Github’s codebase.

All data was downloaded on September 14, 2017.