Column series

Machine Learning: An Overview for Beginners (I) : juejin.cn/post/684490… What is monte Carlo tree search: juejin.cn/post/684490… On the limitations of deep learning juejin.cn/post/684490…

The tools required

According to the needs of specific scenarios (scientific research/engineering), as well as the size of the team, the ability of the developers and other issues, it is necessary to choose the appropriate language and framework and auxiliary tools for development. This not only reduces the difficulty of development, but also improves the efficiency of development. Due to the rapid development of frameworks, many standards and characteristics are constantly changing, and few people can use a variety of frameworks for development at the same time, so after a large number of data collection, the author of various frameworks for the current characteristics of a concise overview and comparison. For a more detailed description, click on the application materials at the end of this article.

Language tool selection

1. Python:

  1. Completely open source and free
  2. There are many Python based frameworks: TensorFlow(Google), PyTorch(Facebook), Caffe2(Facebook/UC Barkely), PaddlePaddle(Badiu), Keras, CNTK(Microsoft), Theano, MXNet (Amazon)
  3. The framework is open source and free, with many samples, easy to deploy and an active community
  4. Language widely used, fast
  5. There are many other type libraries that are easy to use together

2.MATLAB:

  1. Commercial software, more expensive (permanent MATLAB+NN kit: ￥22,500)
  2. Toolbox packages powerful functions, many functions
  3. In some cases, it may take longer to understand functions in depth
  4. The Python framework has relatively few users
  5. Less portable than Python
  6. Algorithm, simulation research advantages, easy to use

3.Jupyter Notebook(IPython):

  1. Support for multiple languages: Python, Julia, R, Scala
  2. Easy to share
  3. Interactive programming for instant results

Photo credit: Jupyter Notebook

4.C++:

  1. High computational speed
  2. Maintenance and development costs are relatively high, it is a low-level language
  3. When you need to develop new algorithms and other modules, it is a required language

In addition, there are also scientific computing language Julia, Scala for big data processing, scripting language Lua and other languages and tools for data processing and analysis

Frame pros and cons and comparison

Photo source: Zhihu/Stanford CS231N courseware

0.scikit-learn

Development team: Contributors to the GitHub open source community

Python is a python library for data mining and analysis. It is an easy-to-use, statistical machine learning library. Based on NumPy, SciPy, matplotlib, free open source.

Open source: BSD 2.0

1.Tensorflow

Development team: Google

Introduction: Is Google’s second generation of artificial intelligence learning system, based on Theano framework for more optimization and improvement

Open source license: Apache 2.0

Advantage:

  1. Tensorflow has an active community, and it is easy to find relevant models and problems. Various supporting tools for the overall framework are mature
  2. Tensorflow is a mature solution for production deployments, from mobile devices to servers, that are easier to deploy than other frameworks
  3. Tensorflow provides API development primarily to python, but also to Java, Go, C++, Haskell, and Rust.
  4. Tensorflow itself also has high-level frameworks such as Keras on top of it for efficient development, as well as the deployment benefits that Tensorflow brings later
  5. According to the summary in CS231N, Tensorflow is very powerful in processing RNN and massively parallel deep learning
  6. There is Tensorflow Serving that loads the model directly to provide RPC interface services
  7. After the completion of the definition of the flow chart, it can be automatically deduced, saving a lot of work

Disadvantage:

  1. Tensorflow is a difficult language to get started with, and many developers report that it is a difficult language to write
  2. Not suitable for quick idea testing
  3. The proportion of Tensorflow in the framework currently used in all papers is still relatively low
  4. There are many places belong to black box operation, it is difficult to understand the principle of processing, debugging is difficult
  5. The functionality has expanded so rapidly that the actual training speed is now far from ideal, with one paper on ArXiv (reference [2]) showing the worst performance of any framework

2.PyTorch

Development team: Facebook and other companies and universities

Open source: BSD 2.0

Advantage:

  1. Tensorflow is easier to use than Tensorflow. It is written in Python style rather than in your own language, making development more efficient
  2. The intermediate process is more transparent and easy to debug and learn
  3. For most people (except experts), writing code is faster to train than Tensorflow due to ability issues
  4. More suitable for small team use and development
  5. The source code of the framework itself is relatively clear and readable, and much more understandable than Tensorflow

Disadvantage:

  1. In most cases, the speed capability is inferior to that of TensorFlow, such as AlexNet and other classical neural networks. For details, see the performance capability of various frameworks in various modes in the paper [2].
  2. Still in Beta, the underlying and framework bugs are more common than TensorFlow
  3. The community is not as large as TensorFlow’s, with the exception of people migrating from the original Torch

3. Caffe2

Team: Facebook

Open source license: Apache 2.0

Facebook developed Caffe2 for product use, while PyTorch is generally used for research and idea implementation. At the same time, Caffe2 is more similar to TensorFlow. Different from PyTorch, It uses static diagrams while Torch uses dynamic diagrams. Therefore, it is not as flexible as Torch and focuses more on speed performance.

Advantage:

  1. Speed is fast
  2. Caffe only supports C++, and supports both C++ and python apis
  3. It does a good job on mobile, probably better than TensorFlow, but there is no evidence yet. However, it is better than PyTorch in industrial applications and distributed training, focusing more on mobile performance.
  4. Caffe’s model can be converted to Caffe2
  5. Facebook and PyTorch are both part of Facebook and are in the process of establishing a communication agreement to share the back end.
  6. You can simply call the pre-trained Model Zoo Model

Disadvantage:

  1. The community is not as mature as TensorFlow
  2. The training is not as flexible as PyTorch

4. MXNet

Development team: Tianqi Chen et al., now funded and supported by Amazon

Advantage:

  1. Support languages: Python, R, MATLAB, Julia, Go, Scala, Javascript
  2. Focus on the core of deep learning, good optimization, fast training speed. Because MXNet has focused on performance optimization rather than functionality extensions like TensorFlow, it has a performance advantage.
  3. Amazon’s cloud service support, now Amazon’s AWS cloud computing service has default MXNet as a learning framework
  4. MXNet’s execution steps are more flexible than TensorFlow’s

Disadvantage:

  1. Low community activity
  2. Average newbie friendliness, poor documentation (but improving with Amazon support)

5. PaddlePaddle

Development team: Baidu

Open source license: Apache 2.0

Advantage:

  1. Easy to use, especially for existing models, simply changing parameters and data
  2. The Chinese document is very friendly
  3. Main program source code written in C++, relatively easy to read
  4. Seems particularly suited to NLP (natural language processing) related problems

Disadvantage:

  1. Fewer documents
  2. It is not suitable to develop new algorithms or features because secondary development needs to be written from the underlying C++

6. Keras

Development team: GitHub/Keras team

Open Source license: MIT

Advantage:

  1. Quick to start, friendly to novice, simple model development efficiency is high
  2. You can choose the underlying framework flexibly: currently support TensorFlow, CNTK, Theano
  3. At the same time with scalability and ease of use, you can increase the required algorithm and other modules
  4. Layer as a concept to encapsulate, easy to understand

Disadvantage:

  1. Performance is good, but not a strength
  2. Flexibility requires familiarity with TensorFlow or one of the other underlying languages, as well as a certain amount of Keras source code, so there are upfront costs to implementing flexibility
  3. Only supports the python

7. CNTK

Development team: Microsoft

Open source agreement: Boost Software License is used for C++ part, and other protocols are used for data set part

Summary: GitHub’s Star and PR show that the project is not less active, but its domestic popularity and usage are much lower than other frameworks

Advantage:

  1. In all kinds of comparative performance, in the middle of the position, often second only to MXNet

Disadvantage:

  1. The number of domestic users is small and the relevant information is few
  2. Novice is not friendly, not easy to install, it is recommended to use Docker isolation installation

8. deeplearning4j

Development team: Eclipse

Open source license: Apache 2.0

Advantage:

  1. Use Java as the development language and enjoy the cross-platform advantages brought by Java VIRTUAL machines
  2. There are some good examples to learn from
  3. Speed vs. other frameworks when using cuDnn
  4. Support Scala

Disadvantage:

  1. Garbage collection (GC) of memory during training causes performance degradation, which the authors are working on
  2. The number of domestic users is small
  3. Documentation is relatively simple

Reference data

  1. How does PyTorch, released by Facebook on January 18, 2017, compare to TensorFlow and MXNet? https://www.zhihu.com/question/54914188
  2. S.Shi et al., Benchmarking State-of-the-Art Deep Learning Software Tools, https://arxiv.org/pdf/1608.07249v7.pdf
  3. Stanford University, cs231n :Convolutional Neural Networks for Visual Recognition, http://cs231n.stanford.edu/
  4. Drake, etc.,Cs231n Notes Chinese translation, https://zhuanlan.zhihu.com/p/21930884
  5. Lei feng,Framework of contrast, https://www.leiphone.com/news/201709/3T4pwc5UBLtRuKvx.html
  6. Zhihu,Keras, Theano, pytorch, caffe2 which is better and how should you try to study?, https://www.zhihu.com/question/59274399
  7. Zhihu,How to evaluate TensorFlow and other deep learning systems, https://weibo.com/p/1001603907610737775666
  8. Zhihu,How do you evaluate Caffe2, https://www.zhihu.com/question/58698158/answer/158139047
  9. GitHub, chenrudan, Comparison of TensorFlow, MXNet and PaddlePaddle open source libraries, http://chenrudan.github.io/blog/2017/02/25/comparetfmxpd.html
  10. CSDN, Mxnet experience, http://blog.csdn.net/u013713010/article/details/71635814
  11. The heart of the machine,Depth | amazon MXNet player, after the heart of the machine and they chatted, http://tech.huanqiu.com/news/2016-11/9726439.html
  12. Zhihu,How efficient is Deeplearning4j? What is the approximate Mnist case run time?, https://www.zhihu.com/question/49509850
  13. Jane book, DrFish heart,Left-handed programmer, right-handed writer: You have to know Jupyter Notebook,https://www.jianshu.com/p/86117613b7a6