Source code: CSSRNN github link, IJCAI2017

Model interpretation

  • Current embedding: [state_size, emb_dim], where state_size is the number of links or grids and emb_dim is the embedding dimension.
  • Destination embedding: [state_size, emb_dim], ibid. The endpoint embedding is a separate set.
  • Neighbor embedding: It is a set of linear transformation coefficients that are not embedding, only embedding to accelerate; W’s SHAPE is [HID_DIM, STATE_SIZE], B’s SHAPE is [STATE_SIZE], where HID_DIM is the output dimension of LSTM. The essence of the last layer of CSSRNN is a Softmax layer with adjacency list constraint. LPIRNN, compared with CSSRNN, adds a dimension with SHAPE [HID_DIM, STATE_SIZE, ADJ_SIZE]. It can be understood that CSSRNN is the coefficient given by node, and LPIRNN is the coefficient given by edge.

Code reading

ID Embedding the edge, shape=[state_size, emb_dim]; The essence of embedding is the coefficient matrix W of the fully connected neural network.

# pretrain emb_ = tf.get_variable("embedding", dtype=tf.float64, Initializer =pretrained_emb_) # No pretrain emb_ = tf.get_variable("embedding", [state_size, emb_dim], dtype=tf.float64)

The encoding of the Input is one-hot, and the construction of a fully connected neural network for the one-hot Input is equivalent to extracting the one-hot line of element 1 from the Embedding according to the ID number. Similar to the tf.gather() method, TensorFlow provides tf.nn.embedding_lookup() to lookup tables in parallel from the embedding, Returns the embedding Tensor (shape=[batch_size, time_steps, state_size]) (shape=[batch_size, time_steps, emb_dim]).

emb_inputs_ = tf.nn.embedding_lookup(emb_, input_label, name="emb_inputs") # [batch, time, emb_dim]

In order to consider the impact of the end point, the destination can be embedded in the same way, and then stitched into the one-hot embedding Tensor through TF. concat.

# Notice that the end is emailed once alone, Not a set of dest_emb_ = tf.get_variable("dest_emb", [state_size, emb_dim], dtype=tf.float64) dest_inputs_ = tf.tile(tf.expand_dims(tf.nn.embedding_lookup(self.dest_emb_, dest_label_), 1), [1, self.max_t_, 1]) # [batch, time, dest_dim] inputs_ = tf.concat(2, [emb_inputs_, dest_inputs_], "input_with_dest") # [batch, time, emb_dim + dest_dim]

RNN layer:

cell = tf.keras.layers.LSTMCell(hidden_dim)
layer = tf.keras.layers.RNN(cell)
rnn_outputs = layer(emb_inputs_, return_sequences=True)  # [batch, time, hid_dim]

Softmax layer: Calculate the loss according to the output of RNN, and consider the constraints of adjacency list when calculating the loss.

outputs_ = tf.reshape(rnn_outputs, ...) # [batch*time, hid_dim] # [batch*time, hid_dim] # wp_ = tf.get_variable("wp", [int(outputs_flat_.get_shape()[1]), config.state_size], dtype=config.float_type) # [hid_dim, state_size] bp_ = tf.get_variable("bp", [config.state_size], dtype=config.float_type) # [state_size] adj_mat = ... # n_edge * n_neighbor, element is the id of edge adj_mask = ... # n_edge * n_neighbor, element is 1 or 0, where 1 means it is an edge in adj_mat and 0 means a padding in adj_mat input_flat_ = tf.reshape(input_, [-1]) # [batch*t] target_flat_ = tf.reshape(target_, [-1, 1]) # [batch*t, 1] sub_adj_mat_ = tf.nn.embedding_lookup(adj_mat_, input_flat_) # [batch*t, max_adj_num] sub_adj_mask_ = tf.nn.embedding_lookup(adj_mask_, input_flat_) # [batch*t, max_adj_num] # first column is target_ target_and_sub_adj_mat_ = tf.concat(1, [target_flat_, sub_adj_mat_]) # [batch*t, max_adj_num+1] outputs_3d_ = tf.expand_dims(outputs_, 1) # [batch*max_seq_len, hid_dim] -> [batch*max_seq_len, 1, hid_dim] sub_w_ = tf.nn.embedding_lookup(w_t_, target_and_sub_adj_mat_) # [batch*max_seq_len, max_adj_num+1, hid_dim] sub_b_ = tf.nn.embedding_lookup(b_, target_and_sub_adj_mat_) # [batch*max_seq_len, max_adj_num+1] sub_w_flat_ = tf.reshape(sub_w_, [-1, int(sub_w_.get_shape()[2])]) # [batch*max_seq_len*max_adj_num+1, hid_dim] sub_b_flat_ = tf.reshape(sub_b_, [-1]) # [batch*max_seq_len*max_adj_num+1] outputs_tiled_ = tf.tile(outputs_3d_, [1, tf.shape(adj_mat_)[1] + 1, 1]) # [batch*max_seq_len, max+adj_num+1, hid_dim] outputs_tiled_ = tf.reshape(outputs_tiled_, [-1, int(outputs_tiled_.get_shape()[2])]) # [batch*max_seq_len*max_adj_num+1, hid_dim] target_logit_and_sub_logits_ = tf.reshape(tf.reduce_sum(tf.multiply(sub_w_flat_, outputs_tiled_), 1) + sub_b_flat_, [-1, tf.shape(adj_mat_)[1] + 1]) # [batch*max_seq_len, max_adj_num+1] # for numerical stability scales_ = tf.reduce_max(target_logit_and_sub_logits_, 1) # [batch*max_seq_len] scaled_target_logit_and_sub_logits_ = tf.transpose(tf.subtract(tf.transpose(target_logit_and_sub_logits_), scales_)) # transpose for broadcasting [batch*max_seq_len, max_adj_num+1] scaled_sub_logits_ = scaled_target_logit_and_sub_logits_[:, 1:] # [batch*max_seq_len, max_adj_num] exp_scaled_sub_logits_ = tf.exp(scaled_sub_logits_) # [batch*max_seq_len, max_adj_num] deno_ = tf.reduce_sum(tf.multiply(exp_scaled_sub_logits_, sub_adj_mask_), 1) # [batch*max_seq_len] #log_deno_ = tf.log(deno_) # [batch*max_seq_len] log_deno_ = Tf.log (tf.clip_by_value(deno_,1e-8,tf.reduce_max(deno_))) # Avoid calculating meaningless log_nume_ = tf.reshape(scaled_target_logit_and_sub_logits_[:, 0:1], [-1]) # [batch*max_seq_len] loss_ = tf.subtract(log_deno_, log_nume_) # [batch*t] since loss is -sum(log(softmax)) max_prediction_ = tf.one_hot(tf.argmax(exp_scaled_sub_logits_ * sub_adj_mask_, 1), int(adj_mat_.get_shape()[1]), dtype=tf.float32) # [batch*max_seq_len, max_adj_num]