1. Introduction

AutoAugment is the pioneering work of data augmentation strategy search, which puts forward a feasible data augmentation strategy search space, but the search time is extremely long and consumes computational resources. After that, UC Berkeley team put forward PBA algorithm in 2019, which optimized the design of the search space of AutoAugment and used the evolutionary algorithm as the search algorithm to achieve the performance similar to that of AutoAugment by far exceeding the search efficiency of AutoAugment.

Here, PBA searches policy schedule instead of the search policy in AutoAugment. The policy schedule refers to which enhancement policy to use up to a particular epoch during training, while the search policy refers to the set policy that is used throughout the training. A later article will explain why you search for policy Schedule rather than policy.

2. The principle of the PBA

2.1 Searching for a Policy Schedule

Search for policy schedule instead of policy:

  • If a fixed policy is searched, the entire training process needs to be completed in each training. The reason is that the improvement of accuracy brought by data enhancement is usually reflected in the later training period. By searching schedule, the calculation results and model parameters in the earlier period can be reused in the later searching period.
  • Searching fixed policy can obtain the policy that can generate gains in the final performance, but it does not mean that the same policy can generate good gains in the early training stage. Therefore, by searching schedule, the enhancement strategy suitable for the specific epoch training stage can be obtained in different training stages. Easy to search through evolutionary algorithms.
  • Although searching schedule will bring more search space than searching policy, it can make the search process more efficient with proper search algorithm.

2.2 Search Space

  • In terms of operator selection, the type, probability and magnitude of the operator are basically the same as AutoAugment, except for Sample Pairing.

  • Search space calculation: The policy at a certain moment in schedule is 2∗152*152∗15 (each picture allows 2 operations of the same operation with 15 operators to be selected), There are a total of (10∗11)30≈1.75∗1061(10*11)^{30}≈1.75*10^{61}(10∗11)30≈1.75∗1061 possibilities, compared with AutoAugment, the search space 2.9∗10322.9*10^{32}2.9∗1032 is larger.

2.3 How to use the PBA policy at a specific time

A PBA policy takes the form of a List of (operation, prob, mag), for example: [(Rotate, 0.5 8), (Rotate, 0.3, 2), (Color, 0.5, 3), (Color, 0.6, 7),… (TranslateX, 0.3, 4), (TranslateX, 0.4, 6)]

Shuffle the list of 30 operations, and select 0-2 enhancement operators according to the probability.

2.4 PBT training thought

  • Figure A represents the idea of serialization search, which requires several models to be trained successively, and the strategy is updated from the evaluation performance of each model, and then the next model training is performed with the new strategy. AutoAugment belongs to this category and takes a long time.
  • In Figure B, the parallel random/grid search is realized, and the training of several models with different strategies is started at the same time. Finally, the strategy with better training effect is selected.
  • Figure C is the training idea of PBT, which is an evolutionary algorithm. Firstly, different strategies are used to train several models in parallel at the same time. After a period of training, these models are evaluated at the same time. A explore perturbation is carried out after the policy change, and parallel training is continued after some changes (similar to genetic mutations) of the strategy. In this way, the poorly performing model can continue to be optimized with the previously verified parametric strategy, thus realizing the learning of schedule.

2.5 PBA algorithm flow

Take CIFAR10 as an example:

  • A subset of 4000 samples was sampled in the CIFAR10 training set as the training set for the search phase.
  • Sixteen classification network models with the same structure were randomly initialized.
  • Assign an initial data enhancement strategy to each model.
  • A data enhancement strategy was applied to run 16 models and several Epochs in parallel to test performance on validation sets.
  • The parameters and enhancement strategies of the four models with the best performance are used to cover the four models with the worst performance, and a Explore perturbation is carried out for the current enhancement strategy. The perturbation algorithm is shown in the figure below.
  • Repeat the above two steps until 200 epochs are complete.
  • The enhanced policy table of the model with the best performance in the whole training process was extracted as the schedule obtained by the final search.

Perturbation algorithm:

3. Experimental results

3.1 PBA search results

The searched results are shown in the figure below. It can be found that the probability of the searched results being operators is relatively low in the early training period. It can be seen that the model does not need too much data enhancement in the early training period. The second diagram shows that the probability trend of various enhancement operators is relatively consistent.

3.2 Accuracy Effect

In the final training stage, the policy schedule obtained from the study of 200 epochs was evenly stretched to the training of 1800 Epochs to complete the entire training. You can find that the effect is still good, can achieve the same performance under the premise of the speed increased very much.

4. Analysis of advantages and disadvantages

Advantages:

  • Efficiency analysis: AutoAugment requires training 120∗15000120*15000120∗15000 epoches, while PBA requires training only 200∗16200*16200∗16 epoches, and PBA saves more than 500 times the cost of AutoAugment computation.
  • The effect of schedule learning strategy is better than that of operation: augmentation schedule is proved to be more effective in this paper.

Disadvantages:

  • Strong assumption: Paper every time training to use has been one of the data set is a data set of (one over ten), and after training inadequate epoch will schedule to expand to full training phase, rely on the strong assumption is that in small data sets training model as a typical task, the learned data enhancement strategy can effectively migrate to the big data sets and model. Such a strong assumption is not necessarily true.
  • Prior to formal training of the model, specific data enhancement strategy search phases are required.
  • Huge search space: compared with AutoAugment, the search space is larger, increasing the search workload.

Reference:

  1. Arxiv.org/pdf/1905.05…
  2. Arxiv.org/pdf/1805.09…
  3. Github.com/arcelien/pb…
  4. Arxiv.org/pdf/1711.09…