(what)

This blog has two main objectives. First, to document the process of learning to use TensorFlow. Second, the parameter data in TensorFlow training data model are visualized.

Specific Operations (how)

Instance to describe

1. Use TensorFlow to build a three-layer neural network (input layer, hidden layer, output layer) model to train the classical MNIST data set, so as to predict handwritten numbers. 2. Use TensorBoard to visualize the Github code address of the parameter data instance in the training process

TensorFlow code implementation

Step1 import the response library and load the dataset

Import tensorflow as tf from tensorflow. Examples. Tutorials. Mnist import input_data # to load in the form of hot coding alone mnist = data set input_data.read_data_sets('./MNIST_data/',one_hot=True)Copy the code

Step2 parameter Settings related to the model set parameters related to the model that may change in advance for the convenience of unified change later

In_unit = 784 h1_unit = 300 learningrate = 0.05 # GDD learningrate dropout_keep_prob = 0.75 # dropout ratio of retained neurons, Neural network is not 0 parameters into the principle of 1/dropout_keep_prob times batch_size = 100 # gradient descent method selected batch size max_iter = 3000 # iteration sava_DIR = '.. /data/' # '; /data/' # '; /log/' # log directoryCopy the code

Step3 set the calculation diagram in TensorFlow 1. When TensorFlow starts the session to run node operations, set the calculation diagram of TensorFlow first, which is conducive to global control of all node operations in the calculation diagram. (The calculation diagram in TensorFlow includes nodes and edges, where nodes are operation, and edges represent data flow, i.e., tensors. However, some edges without data flow represent the dependence between node operation. The node operations in TensorFlow need to be started by the Session Session, using session.run() or the node operation object to call eval() 2.tf.placeholder. As input placeholders for external data (including data sets and other model parameters), the feed_dict parameter needs to be set to populate the placeholder data when running node operations. 3. See the NO3 instructions below 4. Hide the output of the first layer using tF.nn. dropout add another dropout layer 5. AdamOptimizer is used for model training, and the loss function is the cross entropy. Tf.nn.softmax_cross_entropy_with_logits is used to calculate the cross entropy of the real value and the predicted value

tf.reset_default_graph() train_graph = tf.Graph() with train_graph.as_default(): # Step 3.1 Sets the input in the algorithm model, using placeholders, occupying the input data (when using placeholders, Tf.placeholder (dType =tf.float32,shape=[None,in_unit],name = 'train_x') train_y = Tf.placeholder (dType = tF.float32, Shape =[None,10],name = 'train_y' Nn_layer (train_x, INput_DIM =in_unit, output_DIM = H1_unit,layer_name='hider_layer1',act=tf.nn.relu) # Create a dropout layer on the first hidden layer -- Randomly turn off some hidden_layer1 neurons with tf.name_scope('dropout'): dropout_prob = tf.placeholder(dtype=tf.float32, name='dropout_prob') tf.summary.scalar('dropout_keep_probability',dropout_prob) hidden_layer1_dropout = Dropout (hidden_layer1,dropout_prob) # Create an output layer, including 10 categories, The input to the output layer is hidden_layer1_dropout and the output is [1,10] y = Nn_layer (hidden_layer1_dropout, H1_unit,10, Layer_name ='out_layer',act=tf.identity) # step 3.3 create the loss function with tf.name_scope('loss'): cross_entropy_diff = tf.nn.softmax_cross_entropy_with_logits(labels=train_y, logits=y) with tf.name_scope('total'): cross_entropy = tf.reduce_mean(cross_entropy_diff) tf.summary.scalar('loss', Optimizer = tf.train.AdamOptimizer(Learning_rate = Learningrate) train_op = optimizer.minimize(cross_entropy) with tf.name_scope('accuracy'): with tf.name_scope('correct_prediction'): correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(train_y, 1)) with tf.name_scope('accuracy'): accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32)) tf.summary.scalar('accuracy', accuracy)Copy the code

NO3: The operation of building a single-layer neural network is encapsulated in the function nn_layer, which returns a Tensor object. The weight_variable and bias_variable functions use tf.Variable to initialize the corresponding weight w and bias item B in the network layer. Tf-matmul (Input_tensor,weight)+ Bias is the vector representation of wx+ B of all neurons in this layer network. It represents the input of neurons — G (wx+ B), where G is the activation function.

def nn_layer(input_tensor,input_dim,output_dim,layer_name,act=tf.nn.relu): "" Establish the neural network layer (layer 1), Then you have a tensor object for the output of the network layer at that level: Param Input_tensor: characteristic data: Param input_dim: dimension of the input data: Param output_dim: number of neurons at that level: Param "" # set the namespace with tf.name_scope(layer_name): # initialize weights with tf.name_scope('weights'): Weight_variable ([input_dim,output_dim]) variable_summeries(weight)# record weight change with tf.name_scope('bias'): bias = bias_variable([output_dim]) variable_summeries(bias) with tf.name_scope('linear_compute'): preact = tf.matmul(input_tensor,weight)+bias tf.summary.histogram('linear',preact) activeation = act(preact,name = 'activation') tf.summary.histogram('activation',activeation) return activeation def weight_variable(shape): """ inita_w = tf.truncated_normal(shape,stddev=0.1) return tf.Variable(inita_w,dtype=tf.float32) def bias_variable(shape): Inita_b = tf.constant(0.1,shape=shape) return tf.Variable(inita_b)Copy the code

Step4 start the TensorFlow Session to realize the training operations in the calculation diagram and record the corresponding Summary in log file 1. Initialize the Session and set the calculation diagram for the Session. Initialize the TensorFlow variable by performing tf.global_variables_initializer().run() 2. There are two operating modes of nodes in the calculation diagram. Mode 1: Session.run (), which sets up the corresponding node operations in the computation graph, For example, session.run([train_op,cross_entropy,accuracy]), where train_op refers to the operation to run the training model,cross_entropy refers to the operation to obtain the cross entropy of the training model, Accuracy indicates the operation of obtaining the accuracy of model running. The run function returns the corresponding result of node operation running. Oper.eval () can be used to initialize a session in tF.interactivesession, such as train_op.eval(). This mode can only run one node operation at a time.

Session = tf.interactivesession (graph=train_graph) # step 4.1 Merged summary and merged all variables merged = tf.summary.merge_all() train_writer = tf.summary.FileWriter(log_dir+'/train',graph=train_graph) test_writer = Tf.summary.filewriter (log_dir+'/test',graph=train_graph) tf.global_variabLES_initializer ().run() # Step 4.2 Train model and record to TensorBoard for iter in range(max_iter): Trainx_batch_x,train_batch_y = mnist.train.next_batch(batch_size) summmary,acc,loss = The session. The run ([merged, accuracy, cross_entropy], feed_dict = {train_x: trainx_batch_x train_y: train_batch_y, dropout_prob: 1.0}) Print ('loss at step %s: %s'%(iter,loss)) print('Accuracy at step %s: %s'%(iter,acc)) else: if iter % 100 == 0: Run_options = tf.runoptions (trace_level= tf.runoptions.FULL_TRACE) run_metadata = tf.runmetadata () Pass the configuration information and the proTO that records the running information into the running procedure. Summmary,_ = session.run([merged,train_op], feed_dict={train_x:trainx_batch_x,train_y:train_batch_y,dropout_prob:dropout_keep_prob}, options=run_options, Train_writer. add_run_metadata(run_metadata,'step %d' % iter) train_writer.add_summary(summmary,iter) pass else: summmary,_ = session.run([merged,train_op],feed_dict={train_x:trainx_batch_x,train_y:train_batch_y,dropout_prob:dropout_keep_prob}) train_writer.add_summary(summmary,iter) train_writer.close() test_writer.close() session.close()Copy the code

TensorBoard Visualizes the process

TensorBoardThe types of data that can be recorded are:



Scalar: SCALARS

IMAGES: IMAGES

AUDIO: AUDIO

GRAPHS

Data distribution map: Pressing

HISTOGRAMS: HISTOGRAMS

Embed vector: EMBEDDINGS

The above data are recorded in the following ways:

tf.summary.audio(name, tensor) tf.summary.scalar(name,tensor) tf.summary.image(name, tensor) tf.summary.histogram(name, Tensor) # where name means the name of the generated node. Will also serve as a series name in TensorBoard. Tensor is the data you want to record. # embedded vector EMBEDDINGS of visualization, the preservation of the need to rely on model, need to define the tf. Train. Saver object, relocation projector. ProjectorConfig object.Copy the code

Basic visualization steps (excerpted from other blogs) : 1. Create a Graph, that is, you want to record some information from the Graph and visualize it. The code in Step 3 of the code implementation is the process of establishing the calculation graph. Train_graph = tf.graph () 2. Determine which nodes in the Graph to place Summary Operations on to record information. Where tf.name_scope(name) is to set the context for the node, which can be understood as the folder management of the node. One layer contains the relationship of another layer, as shown in the code of establishing neural network in the design calculation diagram of Step 3:

Tf.summary.scalar ('loss', cross_entropy) # Record the output values of the activation function layer tF.summary.histogram ('activation',activeation)Copy the code

3. The Summary Operations designed in the calculation diagram do not actually perform the calculation records. They need to be executed using run in the Session or by other operation nodes. The Summary Operations created in the Step implemented by the TensorFlow code are not dependent on other nodes and need to specifically run the Summary node Operations. Merged = tf.summary.merge_all() merges all summary Operations into one node, and run the node directly. Step 4 merged = tf.summary.merge_all() You can also merge some of the specified summary Operations into a node and then run using code tf.summary.merge(input_list). 4. Use tf.summary.FileWriter to save the output data to the local disk. As in Step 4

The #graph parameter specifies the graph for which the data is to be retrieved, corresponding to the first step. train_writer = tf.summary.FileWriter(log_dir+'/train',graph=train_graph)Copy the code
  1. Run the entire program and run it in terminal inputTensorBoardAnd then open the Web side to view the visual results

    Command:tensorboard --logdir='./ML/log/'



    Open the corresponding urlhttp://0.0.0.0:6006/