The paper contains 8094 words and is expected to last 30 minutes or longer
Photo credit: pexels.com/pixabay
TensorFlow is a general-purpose high-performance computing library that Google opened source in 2015. The initial focus was to provide high-performance apis for building neural networks (NNs). However, over time and with the rise of the machine learning (ML) community, TensorFlow has evolved into an entire machine learning ecosystem.
TensorFlow has changed dramatically since its inception. TensorFlow 2.0 is currently in beta and has many changes compared to TF1.x. Here are the main changes.
1. The default state is Eager Execution
First, eager execution is the default way to run TF code.
To build a neural network in TF1.x, you need to define an abstract data structure called Graph. Also, if you try to print one of the graph nodes, you will not see the expected value, but you will see the reference to the graph node. In fact, to run the graph, you need to use a wrapper called a Session. Using the session.run () method, you can pass Python data to the graph and actually train the model.
TF 1.x code example
Use eager execution to make that change. TensorFlow code can now run just like normal Python code. This means that actions can be created and evaluated quickly.
Tensorflow 2.0 code example
TensorFlow 2.0 code looks a lot like NumPy code. In fact, TensorFlow and NumPy objects can be easily switched. So don’t worry about placeholders, sessions, feed_dictionaties, etc.
2. Clear API
Many apis, such as tf.gans, tf.app, tf.contrib, tf.flags, are cleaned up or moved to separate repositories.
One of the most important cleanups, however, involves how we build models. There are more than 1 or 2 different ways to build/train ML models in TF1.x.
Tf.slim, tf.layers, tf.contrib.layers, and tf.keras are all apis that can be used to build neural networks. Does not include sequences from TF1.x to API sequences. In most cases, it is not clear which one to choose in each case.
While many apis perform well, they don’t seem to converge to a common approach to development. Furthermore, if you train the model in one API, it’s not easy to reuse the code using the other apis.
In TF2.0, tF.keras is the recommended high-level API.
As you can see, you are currently trying to resolve all possible use cases using KerasAPI.
3. Beginner API
There are few changes to the beginner API in version 2.0 compared to version 1.0. But for now, Keras is the default and recommended advanced API. In short, Keras refers to a set of layers that are used to build a neural network using explicit criteria. When you install TensorFlow using PIP, you typically get the full Keras API and some additional functionality.
model= tf.keras.models.Sequential()
model.add(Flatten(input_shape=
(IMAGE_HEIGHT,IMAGE_WIDTH)))
model.add(Dense(units=32,activation='relu')) model. The add (Dropout (0.3)) model. The add (Dense (units = 32, activation ='relu'))
model.add(Dense(units=10,activation='softmax'))
#Configures the model for training.
#Define the model optimizer, the loss
function and the accuracy metrics
model.compile(optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
model.summary()Copy the codeThe beginner API is called serialization and typically defines a neural network as a set of layers. It has other advantages besides simplicity. Note that the model is defined in terms of a data structure (a set of layers). As a result, the probability of errors due to model definitions is minimized.
4. Keras-Tuner
Keras-tuner is a special library for hyperparameter tuning of Keras models. At the time of writing, the library is in a pre-alpha state, but runs fine on Colab using TF.Keras and TensorFlow 2.0β.
Portal: https://github.com/keras-team/keras-tuner
The concept is very simple. First, you need to define a modeling function that returns the compiled Keras model. The function takes a parameter named HP as input. Using HP, you can define a range of candidate values to sample hyperparameter values.
A simple model is established and three hyperparameters are optimized. To see hidden units, integer values between predefined ranges are sampled. To calculate the loss rate and learning rate, a random selection is made between some given values.
defbuild_model(hp):
# define the hyper parameter ranges for
thelearning rate, dropout and hidden unit
hp_units = hp.Range('units', min_value=32,max_value=128, step=32)
hp_lr = hp.Choice('learning_rate',values=[1e-2, 1e-3, 1e-4])
hp_dropout = hp.Choice('dropout', values = [0.1, 0.2, 0.3])# build a Sequential model
model = keras.Sequential()
model.add(Flatten(input_shape=(IMAGE_HEIGHT,IMAGE_WIDTH)))
model.add(Dense(units=hp_units,activation='relu'))
model.add(Dropout(hp_dropout))
model.add(Dense(units=32, activation='relu'))
model.add(layers.Dense(10,activation='softmax'))
# compile and return the model model.compile(optimizer=keras.optimize
rs.Adam(hp_lr),
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
return model
#create a Random Search tunertuner= RandomSearch(
build_model,
objective='val_accuracy'.# define themetric to be optimized over
max_trials=3,
executions_per_trial=1,
directory='my_logs')
# define the outputlog/checkpoints folder
# start hyper-parameter optmization searchtuner.search(x_train,y_train,
epochs=2,
validation_data=(x_test, y_test))Copy the codeThen create a tuner object. In this case, a random search strategy is implemented. Finally, you can start tuning using search(). It has the same signature as fit().
Finally, you can check the tuner’s conclusions and choose the best model. Note that both the training logs and model checkpoints are saved in the directory folder (my_logs). In addition, the choice of minimizing or maximizing the goal (validation accuracy) is automatically inferred.
5. Advanced API
When you see this type of implementation, you have to go back to object-oriented programming. The Model here is a Python class that extends tf.keras.model. Model subclassing was introduced by Chainer and has a lot to do with how PyTorch defines models.
Model layers in class constructors are defined by model subclassing. Call () is responsible for the definition and execution of forward deduction.
class Model(tf.keras.Model):
def__init__(self):
# Define thelayers here
super(Model,self).__init__()
self.conv1 =Conv2D(filters=8, kernel_size=4, padding="same",
strides=1,input_shape=(IMAGE_HEIGHT,IMAGE_WIDTH,IMAGE_DEPTH))
self.conv2 =Conv2D(filters=16, kernel_size=4, padding="same", strides=1)
self.pool =MaxPool2D(pool_size=2, strides=2, padding="same")
self.flat =Flatten()
elf.probs =Dense(units=N_CLASSES, activation='softmax', name="output")
def call(self,x):
# Define theforward pass
net =self.conv1(x)
net =self.pool(net)
net =self.conv2(net)
net =self.pool(net)
net =self.flat(net)
net =self.probs(net)
return net
defcompute_output_shape(self, input_shape):
# You need tooverride this function if you want to use the subclassed model
# as part of afunctional-style model.
# Otherwise,this method is optional.
shape =tf.TensorShape(input_shape).as_list()
shape[-1] =self.num_classes
returntf.TensorShape(shape)Copy the codeSubclassing has many advantages: simplified model checking; (using breakpoint debugging) stops at a given line and checks the activation or logic of the model.
However, with increased flexibility comes more loopholes.
Subclassing requires more attention and understanding from programmers.
In general, errors in code are more obvious (e.g. model wiring).
6. Define training loop
The easiest way to train models in TF2.0 is to use fit(). Fit () supports both serialized and subclassed models. If model subclassing is used, the only adjustment required is to override the compute_output_shape() classification method. In addition, you should be able to input FIT () with tF.data.dataset or the standard Numpynd-Arrays.
However, if you want to get a clear picture of gradients or loss, you can use gradient bands. This is particularly effective in research.
Using gradient bands, each step of the training process can be manually defined. It can be applied to each basic step of training neural network, such as:
Driving method
· Loss function evaluation
Inverse method
· Gradient descent
You can see intuitively how neural networks are trained. You can also directly output the loss value W.R.T, the model weight, or the gradient vector itself to check.
Gradient zones provide greater flexibility. However, as with subclassing and serialization, the greater the flexibility, the higher the additional cost. In contrast to FIT (), a training loop needs to be defined manually. As a natural consequence, it makes the code more buggy and harder to debug. This is a good compromise, more suited to code engineers (looking for standardized code) than researchers who are interested in developing new products.
Also, using fit() makes it easy to set up the TensorBoard.
7. Set the TensorBoard
Using fit() makes it easy to set up instances of TensorBoard. It also works with Jupyter/Colab Notebook.
Add TensorBoard as a callback function for FIT.
Fit () can be used for serialization apis and subclassing apis.
# Load theTensorBoard notebook extension%load_exttensorboard
# create thetensorboard callbacktensorboard= TensorBoard
(log_dir='logs/{}'.format(time.time()), histogram_freq=1)
# train themodelmodel.fit(x=x_train,
y=y_train,
epochs=2,
validation_data=(x_test, y_test),
callbacks=[tensorboard])
# launchTensorBoard%tensorboard--logdir logsCopy the codeIf you choose to subclass the model and write your own training loop (using a grading band), you also need to manually define the TensorBoard. This includes creating a summary file using tf.summary.create_file_writer() and specifying the variables to visualize.
There are many callback functions available, but the most effective ones are:
· EarlyStopping: As the name suggests, it establishes a rule to stop training when the monitoring volume stops increasing.
· ReduceLROnPlateau: Reduce the learning rate when the measure stops increasing.
· TerminateOnNaN: Terminates training when NaN is lost.
· LambdaCallback: Used to create simple custom online callbacks.
The full directory is available at TensorFlow 2.0 Callbacks.
Portal: https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/callbacks
8. Extraction performance of EagerCode
If the gradient band is selected for model training, the performance will be significantly reduced.
TF code execution is good for understanding, but poor for performance. To avoid this problem, TF2.0 introduced tf.function.
Basically, if you decorate a Python function with tf.function, TensorFlow takes the function and converts it into a TF high performance abstraction.
@tf.functiondeftrain_step(images, labels):
with tf.GradientTape() as tape:
# forward pass
predictions = model(images)
# compute the loss
loss = cross_entropy(tf.one_hot(labels,N_CLASSES), predictions)
# get the gradients w.r.t the model's weights
gradients = tape.gradient(loss,model.trainable_variables)
# perform a gradient descent step
optimizer.apply_gradients(zip(gradients,model.trainable_variables))
# accumulates the training loss and accuracy train_loss(loss)
train_accuracy(labels, predictions)Copy the codeThis means that the function will be marked for JIT editing so that TensorFlow can run it as a graph. As a result, tf1.x (graphics) performance benefits such as node pruning, kernel fusion, etc.
In short, TF2.0 aims to design code into smaller functions. You can then use tf.function to tag the required code for additional performance. Best suited for modifying functions that represent maximum computational bottlenecks. These are usually the forward thrusts of the training loop or model.
Note that you lose some of the benefit of eager execution when you decorate a function with tf.function. That is, you cannot set a breakpoint or use print() in this code snippet.
9. Save and repair the model
Tf1.x lacks consistent standards on how to save/load trained models for production. TF2.0 tries to solve this problem by defining a single API.
TF2.0 saves models in a few ways, but standardizes an abstraction called a SavedModel.
Needless to say here. If you use the TF.keras. Model to create a serialized model or extended class, the class inherits from tF.train. Therefore, the model can be serialized as a SavedModel object.
# serialize your model to a SavedModel object
# It includes the entire graph, all variables
andweightsmodel.save('/tmp/model', save_format='tf')
# load your saved
modelmodel = tf.keras.
models.load_
model('/tmp/model')Copy the codeSavedModel is integrated with the Tensorflow ecosystem. In other words, it can be distributed to many different devices, including mobile phones, edge devices, and servers.
10. Switch to TF-Lite
If you want to assign SavedModel to an embedded device such as Raspberry Pi, Edge TPU, or mobile phone, use the TF Lite converter.
Note that in version 2.0, the TFLite converter does not support frozen GraphDef (typically generated in TF1.x). If you want to convert frozen GraphDefs to run in TF2.0, you can use the TF.pat.v1.tflite converter.
Post-training quantization is generally performed before assignment to embedded devices. To do this using the TFLite converter, set the optimization flag to “OPTIMIZE_FOR_SIZE”. This quantifies the weight of the model from floating point to 8-bit accuracy. Model size can be reduced and delay can be improved without reducing model accuracy.
# create a TF Lite converter
converter
=tf.lite.TFLiteConverter.from_keras_mode
l(model)
# performs model quantization to reduce
the size of themodel and improve latency
converter.optimizations
=[tf.lite.Optimize.OPTIMIZE_FOR_SIZE]
tflite_model = converter.convert()Copy the codePlease note that this is an experimental flag and is subject to change.
11. Convert to tensorflow.js
To close, use the same SavedModel object and convert it to tensorflow.js format. You can then load it using JavaScript and run the model on a browser.
! tensorflowjs_converter\ --input_format=tf_saved_model \ --saved_model_tags=serve \ --output_format=tfjs_graph_model \ /tmp/model \ /tmp/web_modelCopy the codeFirst, you need to install tensorflow.js via PIP. The tensorFlowJs_Converter script is then used to take the training model and convert it into Javascript compatible code. Finally, it can be loaded and inferred in Javascript.
You can also use tesnorflow.js on a browser to train the model.
conclusion
Finally, other features of version 2.0. First, it is very simple to add more layers to a serialized or subclassed model. Although TF covered Conv2D, TransposeConv2D and other layers, it was always unusable. This is especially true for reproductions or research.
The good news is that you can develop custom layers. By following the same Keras API, you can create a class and extend it to tf.keras.layer. In fact, custom activation functions, regularization layers, or measures can be created in a very similar pattern.
Resource portal: https://www.tensorflow.org/tutorials/eager/custom_layers
In addition, existing TensorFlow 1.x code can be converted to TF2.0. To do this, the TF team creates tF_upgrade_v2.
This script does not convert TF1.x code to 2.0 idiomatic code. It basically uses the TF.pat.v1 module for namespace-changing functions. Also, if the old code uses tf.contrib, the conversion will not work. New TF2.0 versions of other libraries or missing functions may be required.
Leave a comment and like it on moments
We share the dry goods of AI learning and development
