Author | Dr. VAIBHAV KUMAR compile | source of vitamin k | Analytics In Diamag

There are many interesting applications of natural language processing (NLP), and text generation is one of them.

When a machine learning model works on sequence models such as recurrent neural networks, LSTM-RNN, GRU, etc., they can generate the next sequence of input text.

PyTorch provides a powerful set of tools and libraries that add momentum to these NLP based tasks. It not only requires less preconditioning, but also speeds up the training process.

In this article, we will train recurrent neural networks (RNN) for several languages in PyTorch. After successful training, the RNN model predicts names that belong to languages that begin with an input letter.

PyTorch implementation

This implementation is done in Google Colab, where the data set is fetched from the Google drive. So, first, we’ll use Colab Notebook to install Google Drive.

from google.colab import drive
drive.mount('/content/gdrive')Copy the code

Now we will import all the necessary libraries.

from __future__ import unicode_literals, print_function, division
from io import open
import glob
import os
import unicodedata
import string
import torch
import torch.nn as nn
import random
import time
import math
import matplotlib.pyplot as plt
import matplotlib.ticker as tickerCopy the code

The following code snippet reads the data set.

all_let = string.ascii_letters + " .,; '-" n_let = len(all_let) + 1 def getFiles(path): return glob.glob(path) # Unicode string to ASCII def unicodeToAscii(s): return ''.join( c for c in unicodedata.normalize('NFD', s) if unicodedata.category(c) ! = 'Mn' and c in all_let) def getLines(filename): lines = open(filename, Encoding =' utF-8 ').read().strip().split('\n') return [unicodeToAscii(line) for line in lines] # create cat_lin dictionary, Storage of each category table cat_lin = {} all_ctg = [] for filename in getFiles (' gdrive/My Drive/Dataset/data/data/names / *. TXT ') : categ = os.path.splitext(os.path.basename(filename))[0] all_ctg.append(category) lines = getLines(filename) cat_lin[categ] = lines n_ctg = len(all_ctg)Copy the code

In the next step, we will define the Module class to generate the name. The module will be a recurrent neural network.

class NameGeneratorModule(nn.Module): def __init__(self, inp_size, hid_size, op_size): super(NameGeneratorModule, self).__init__() self.hid_size = hid_size self.i2h = nn.Linear(n_ctg + inp_size + hid_size, hid_size) self.i2o = nn.Linear(n_ctg + inp_size + hid_size, op_size) self.o2o = nn.Linear(hid_size + op_size, Op_size) self.dropout = nn.dropout (0.1) self.softmax = nn.logsoftmax (dim=1) def forward(self, category, input, hidden): inp_comb = torch.cat((category, input, hidden), 1) hidden = self.i2h(inp_comb) output = self.i2o(inp_comb) op_comb = torch.cat((hidden, output), 1) output = self.o2o(op_comb) output = self.dropout(output) output = self.softmax(output) return output, hidden def initHidden(self): return torch.zeros(1, self.hid_size)Copy the code

The following functions will be used to select random items from the list and random rows from the category

def randChoice(l):
    return l[random.randint(0, len(l) - 1)]

def randTrainPair():
    category = randChoice(all_ctg)
    line = randChoice(cat_lin[category])
    return category, lineCopy the code

The following functions convert the data to a compatible format for the RNN module.

def categ_Tensor(categ):
    li = all_ctg.index(categ)
    tensor = torch.zeros(1, n_ctg)
    tensor[0][li] = 1
    return tensor

def inp_Tensor(line):
    tensor = torch.zeros(len(line), 1, n_let)
    for li in range(len(line)):
        letter = line[li]
        tensor[li][0][all_let.find(letter)] = 1
    return tensor

def tgt_Tensor(line):
    letter_indexes = [all_let.find(line[li]) for li in range(1, len(line))]
    letter_id.append(n_let - 1) # EOS
    return torch.LongTensor(letter_id)Copy the code

The following functions create random training examples, including categories, inputs, and target tensors.

# loss criterion = nn.nllloss () # learning rate = 0.0005 def train(category_tensor, input_line_tensor, target_line_tensor): target_line_tensor.unsqueeze_(-1) hidden = rnn.initHidden() rnn.zero_grad() loss = 0 for i in range(input_line_tensor.size(0)): output, hidden = rnn(category_tensor, input_line_tensor[i], hidden) l = criterion(output, target_line_tensor[i]) loss += l loss.backward() for p in rnn.parameters(): p.data.add_(p.grad.data, alpha=-lr_rate) return output, loss.item() / input_line_tensor.size(0)Copy the code

To display the time during training, define the following function.

def time_taken(since):
    now = time.time()
    s = now - since
    m = math.floor(s / 60)
    s -= m * 60
    return '%dm %ds' % (m, s)Copy the code

In the next step, we will define the RNN model.

model = NameGenratorModule(n_let, 128, n_let)Copy the code

We will see the parameters of the defined RNN model.

print(model)Copy the code

Next, the model will train 10,000 epochs.

Epochs = 100000 print_every = 5000 plot_every = 500 all_losses = [] total_loss = 0 # Reset start = time.time() for each iteration iter in range(1, epochs + 1): output, loss = train(*rand_train_exp()) total_loss += loss if iter % print_every == 0: print('Time: %s, Epoch: (%d - Total Iterations: %d%%), Loss: %.4f' % (time_taken(start), iter, iter / epochs * 100, loss)) if iter % plot_every == 0: all_losses.append(total_loss / plot_every) total_loss = 0Copy the code

We will visualize the loss in training.

Plt.figure (figsize=(7,7)) plt.title("Loss") plt.plot(all_losses) plt.xlabel("Epochs") plt.ylabel("Loss") plt.show() plt.plot(all_losses) plt.xlabel("Epochs") plt.ylabel("Loss") plt.show()Copy the code

Finally, we will test our model to generate language-specific names given a starting alphabet letter.

Def sample_model(category, start_letter='A'): with torch. No_grad (): # no need to track history in sampling category_tensor = categ_Tensor(category) input = inp_Tensor(start_letter) hidden = NameGenratorModule.initHidden() output_name = start_letter for i in range(max_length): output, hidden = NameGenratorModule(category_tensor, input[0], hidden) topv, topi = output.topk(1) topi = topi[0][0] if topi == n_let - 1: break else: Letter = all_let[topi] output_name += letter input = inp_Tensor(letter) return output_name sample_names(category, start_letters='XYZ'): for start_letter in start_letters: print(sample_model(category, start_letter))Copy the code

Now we will examine the sample model to generate the name given the language and initial letter.

print("Italian:-")
sample_names('Italian', 'BPRT')
print("\nKorean:-")
sample_names('Korean', 'CMRS')
print("\nRussian:-")
sample_names('Russian', 'AJLN')
print("\nVietnamese:-")
sample_names('Vietnamese', 'LMT')Copy the code

So, as we saw above, our model has generated names that belong to language categories, starting with the input letters.

References:

  1. Trung Tran, “Text Generation with Pytorch”.
  2. “NLP from Scratch: Generating Names with a Character Level RNN”, PyTorch Tutorial.
  3. Francesca Paulin, “Character-Level LSTM in PyTorch”, Kaggle.

The original link: analyticsindiamag.com/recurrent-n…

Welcome to panchuangai blog: panchuang.net/

Sklearn123.com/

Welcome to docs.panchuang.net/