Since its release, PyTorch has been one of the most popular frameworks in 2017 due to its advantages in debugging, compiling, and more. This article looks at PyTorch’s achievements since its first year of open source. PyTorch is also comparable to TensorFlow in some metrics. Is PyTorch the star frame of 2017?


Yann LeCun Twitter

As of today, PyTorch has been publicly available for one year. Over the past year, we have been dedicated to building a flexible deep learning research platform. Thanks to the PyTorch community for their continued contributions and optimizations over the past year.

In this article, we want to summarize PyTorch’s year: progress, news, and community highlights.

community

We are fortunate to have a strong and passionate group of researchers and engineers on the PyTorch team. The core team of engineers and researchers from different countries, companies and universities would not be PyTorch today without their efforts.

Research papers, toolkits, GitHub

PyTorch has only been out for a few days, and the community is already using PyTorch to implement their favorite research papers and post their code on GitHub. Open source code is a major and core tool for today’s researchers.

People came together to create Torchtext, TorchVision and Torchaudio to benefit the masses in different fields of research.

The first PyTorch community toolkit (named Block) from Brandon Amo helps make it easier to work with Block matrices. Locus LABS from CMU later went on to publish the PyTorch toolkit and the implementation of most of its research. The first research paper is code-named “Paying More Attention to Attention” by Sergey Zagoruyko.

Jun-yan Zhu, Taesung Park, Phillip Isola, Alyosha Efros and their team from UC Berkeley released the very popular Cycde-gan and Pix2PIx, which are used for image conversion.

  • See paper: academic | make monet painting pictures: Berkeley image to image translation, a new study

Researchers at Harvard DNLP and Systran began using PyTorch to develop and enhance OpenNMT, which began as an initial re-implementation of Facebook Adam Lerer’s [Lua]Torch code.

  • See also: open source | harvard group NLP open source toolkit OpenNMT machine translation: is up to the level of the production is available

The MagicPony team from Twitter contributed the PyTorch implementation of its super-resolution study example.

Salesforce has released several toolkits, including its highlight, PyTorch-QRNN, a new RNN that is 2 to 17 times faster than cudNn-optimized standard LSTM. James Bradbury and his team are one of the most active and engaging in the PyTorch community.

Researchers from Uber, Northeaster, and Stanford have formed an active probabilistic programming community around their toolkits Pyro and ProbTorch. They are actively developing the Torch. Review core toolkit. The community is very active and growing rapidly, Together with Fritz Obermeyer, Noah Goodman, Jan-Willem Van de Meent, Brooks Paige, Dustin Tran and 22 other attendees, we held the first PyTorch at NIPS 2017 Probabilistic programming conference on bayesitizing the world.

  • See also: Uber and Stanford Open Source Deep Probabilistic Programming language Pyro: Based on PyTorch

Nvidia researchers released three high-quality REPO implementations including Pix2PiX-HD, Sentiment Neuron and FlowNet2. The extensibility analysis of the different data parallelism models in PyTorch will benefit the entire community.

AllenNLP, released by the Allen institute for artificial intelligence, includes multiple NLP advanced models: reference implementations of standard NLP tasks and easy-to-use web demos.

In June, we also won our first Kaggle contest (Team GRT123). They won the 2017 Data Science Cup (about lung cancer detection) and later unveiled their PyTorch implementation.

In terms of visualization, Tzu-Wei Huang implemented the Tensorboard-PyTorch plug-in, and Facebook AI Research released a VisDOM visualization package compatible with PyTorch.

Finally, Facebook AI Research released several projects, such as ParlAI, fairseq-py, VoiceLoop, and FaderNetworks, that implemented advanced models and interface datasets in multiple domains. Because the space is limited, there would not have outstanding project list, detailed list can refer to: https://github.com/soumith?tab=stars.

We would also like to thank those who actively help others in the forum. Thank you very much for your invaluable service.

indicators

By the numbers:

  • There are 87,769 lines of code introducing the Torch on Github.

  • There are 3983 Repositories on Github that mention PyTorch in their name or description.

  • PyTorch Binary has been downloaded over 500,000 times (651916).

  • On the forum, 5400 users posted 21,500 discussions covering 5,200 topics.

  • Since its release, PyTorch has been mentioned in 131 discussions under the/R/Machinelearning theme on Reddit. During the same period, TensorFlow was mentioned 255 times.

See also:

  • Which is better, PyTorch or TensorFlow? What do the first tier developers say

  • TensorFlow open Source one year: This is probably the most complete inventory yet

The indicators

PyTorch is a research focused framework. So the metrics to measure it include PyTorch’s use in machine learning research papers.

  • PyTorch was mentioned in 87 papers presented at the ICLR 2018 Conference, compared to 228 for TensorFlow, 42 for Keras, and 32 for Theano and Matlab.

  • On a monthly basis, arXiv cited The PyTorch framework in 72 papers, TensorFlow in 273, Keras in 100, Caffe in 94 and Theano in 53.

Courses, tutorials and books

We had good API documentation in place when we launched PyTorch, but the tutorials were limited, with only a few Ipython Notebooks, which were useful but not enough.

Sasank Chilamkurthy, undertake the task of the improved tutorial, see this tutorial: http://pytorch.org/tutorials/.


Sean Robertson and Justin Johnson have written new tutorials on the NLP field, as well as tutorials on learning by example. Yunjey Choi has written a tutorial on deploying most models in 30 lines or less. Each new tutorial helps users find their own learning path more quickly with different learning methods.

Goku Mohandas and Delip Rao changed the code in the book they were writing to use PyTorch.

We see some university machine learning courses taught using PyTorch as the primary tool, such as Harvard CS 287. To further facilitate mass learning, we also saw three online courses taught using PyTorch.

Fast. Ai’s “Deep Learning for Coders” is a popular online course. In September, Jeremy and Rachel announced that the next Fast. Ai class would be based almost entirely on PyTorch.

Ritchie Ng, a researcher who studied at both Tsinghua and the National University of Singapore, launched a Udemy course called “Practical Deep Learning with PyTorch”.

Sung Kim from The Hong Kong University of Science and Technology launched PyTorch Zero to All, an online course for a general audience, on YouTube.

  • See also: Four Days in a row! PyTorch, Hong Kong University of Science and Technology

engineering

Last year PyTorch implemented a number of features, including performance on the board, fixing numerous bugs, and more. Last year to complete the task list as bellow: https://github.com/pytorch/pytorch/releases. Here are a few highlights:

High degree of ladder

With the publication of many papers on the implementation of gradient penalty term and the continuous development of second-order gradient method, high ladder degree becomes a necessary hot function. Last August, we implemented a common interface that uses n-th derivatives to speed up convergence in support of higher-ladder functions, and as of this writing, almost all OPS support this interface.

Distributed PyTorch

Last August, we released a small distributed package that uses the very popular MPI-Collective approach. It has multiple backends such as TCP, MPI, Gloo, and NCCL2 to support a variety of CPU/GPU set operations and use cases. This package integrates distributed technologies such as Infiniband and RoCE. Distribution was hard, and we had some bugs in the initial iteration. In subsequent releases, we made some improvements to make the package more stable and more powerful.

Closer to the NumPy

One of the biggest demands of users is that they are familiar with NumPy functionality. Features like Broadcasting and Advanced Indexing are easy and concise, saving users time. We implemented these features and started to bring our API closer to NumPy. Over time, we hope to get closer and closer to NumPy’s API where appropriate.

performance

Performance is an ongoing battle, especially for PyTorch, a dynamic framework that wants to maximize flexibility. Last year, from the core Tensor library to the neural network operator, we improved PyTorch’s performance on board, making it much faster to write microoptimizations on board.

  • We’ve added special AVX and AVX2 internal functions for Tensor calculation;

  • Write faster GPU kernel for common workloads such as cascading and Softmax;

  • Rewrite code for multiple neural network operators, such as nn.Embedding and group convolution.

PyTorch reduces board overhead by 10x

Since PyTorch is a dynamic graph framework, we create a new graph for each iteration of the training cycle. Therefore, the framework overhead must be low, or the workload must be large enough to hide the framework overhead. Last August, DyNet authors (Graham Neubig and his team) showed that DyNet was faster than PyTorch on some small NLP models. This was an interesting challenge, and we started rewriting the PyTorch internals to reduce the framework overhead from 10 subtlety/operator to 1 subtlety.

ATen

Along with the redesign of PyTorch internals, we also built the ATen C++11 library, which now dominates all of PyTorch’s back ends. ATen has an API similar to the PyTorch Python API, making it a C++ library for Tensor calculations. ATen can be built and used independently by PyTorch.

Output model for production: support for ONNX and JIT compilers

A common request we received was to export the PyTorch model to another framework. Users use PyTorch for quick research, and once the model is complete, they want to ship it into a larger project that only requires C++.

So we built tracer, which outputs the PyTorch model as an intermediate representation. Users can run the current PyTorch model more efficiently with subsequent tracers, convert it to ONNX format for output to Caffe2, MXNet, TensorFlow and other frameworks, or load directly into hardware acceleration libraries such as CoreML or TensorRT. This year, we will make more use of the JIT compiler to improve performance.


This article is from PyTorch and compiled by Heart of the Machine.
Pytorch.org/2018/01/19/….