Construction of intelligent chatbot using Botkit and Rasa NLU

Every day we hear about the launch of new robots capable of travel, social, legal, support, sales and more. The last time I checked, Facebook Messenger alone had more than 11,000 bots, but by the time I write this, it’s estimated to have added several thousand more. The first generation of robots were dumb because they could only analyze a limited number of questions based on keywords in a conversation. But with the commercialization of machine learning services like Wit. Ai, Api. ai, Luis. Ai, Amazon Lex, IBM Watson and NLP Natural Language Processing, Boosting the development of smart robots like Donotpay and chatShopper.

I don’t know if robots are just hype or real reality, but I can say for sure that the process of building a robot can be fun and challenging. In this article, I’ll introduce you to some of the tools you’ll need to build an intelligent chatbot.

The title of the article already makes it clear that we will use Botkit and Rasa (NLU) to build our robot. But before I get into the technology, I want to share the reasons for choosing these two platforms and explain how they should fit into our use case.

Robot Development Framework

Howdy, Botkit, and the Microsoft (MS) Bot Framework are strong contenders in this regard. What these frameworks have in common is:

• They are all open source.
• They have some integration with popular messaging platforms like Slack, Facebook Messenger, Twilio, etc.
• They all have good documentation files.
• They all have active developer communities.

Because of compliance issues, we chose AWS to deploy all our services, and we expect robots to deploy with AWS as well.

NLU Natural Language Understanding

Api. ai (owned by Google) and Wit. Ai (owned by Facebook) are two NLU tools that are popular in the robotics industry and were the first tools we considered for this task. Their common implementation plans are:

• They are all hosted as cloud services.
• They all have Nodejs, the Python SDK, and a REST interface.
• They are well documented.
• They all support interactive understanding of the state or context of a conversation, which makes it much easier to build a conversational platform.

As mentioned earlier, we were unable to use any of the above managed solutions due to compliance issues. Instead, we perfectly replaced api. ai and Wit. Ai by using an open source NLU called Rasa, which we can host and manage on AWS.

Reading this, you may be wondering why I use the term NLU to describe the two tools, api. ai and Wit. Ai, instead of using the term NLP for Natural Language Processing.

In fact, NLP refers to any system that can naturally interact with people. This means that we can communicate with a system by talking to a real person. NLU, by contrast, is a subfield of NLP, a small but complex subfield that transforms unprocessed input signals into data structures that machines can understand and give feedback on. For example, when you say “Date = April 20, 2017, location = San Francisco, Action = hotel reservation,” the system will understand.

Rasa NLU

In this section, I will explain Rasa NLU in detail and provide you with some common terms used in NLP that you should be familiar with.

• Intention: to inform the machine of the user’s demands.
  • For example: making a complaint, asking for a refund and so on.
• Entities: Attributes that extract details of user demands.
  • For example: complaints related to service interruption, refund, etc
• Confidence: A distance indicator that shows how far the NLU analysis results differ from the claims in the intent list.

The following examples should help you understand the above concepts better.

Type: “My Internet has been down this morning.”

• Intent: Service interruption
• Entities: “Service = Internet”, “Duration = whole morning”
• Confidence: 0.84 (may vary according to individual training style)

The NLU’s job (in this case, Rasa) is to take a sentence or statement and output an “intention”, “entity” and “confidence” that can be used by the robot. Rasa basically provides a high-level API on top of various NLP and ML libraries responsible for classifying “intentions” and extracting “entities”. These NLP and ML libraries are called the back ends, and they make Rasa smart. Here are some of the backends commonly used with Rasa:

• MITIE: An all-inclusive library; In other words, it has a built-in NLP library for entity extraction and an ML library for intent classification.
• SpaCy + Sklearn: spaCy is an NLP library that does “entity” extraction only. Sklearn, on the other hand, is used with spaCy to add ML capabilities for intent sorting.
• MITIE + Sklearn: This combination uses two of the best libraries in their respective fields. This combination combines the good “entity” recognition of MITIE with the fast and excellent “intent” classification of SkLearn.

I have used the MITIE backend to train Rasa. In the demo section, we have an “online Support conversation bot” that we train to solve messages such as:

• My cell phone doesn’t work.
• My phone is off.
• My mobile phone is broken and I can’t use it any more.

My training data is as follows:

{
  "rasa_nlu_data": {
      "common_examples": [{"text": "hi"."intent": "greet"."entities": []}, {"text": "my phone isn't turning on."."intent": "device_failure"."entities": [{"start": 3."end": 8,
                "value": "phone"."entity": "device"}]}, {"text": "my phone is not working."."intent": "device_failure"."entities": [{"start": 3."end": 8,
                   "value": "phone"."entity": "device"}]}, {"text": "My phone crashed and isn’t working anymore."."intent": "device_failure"."entities": [{"start": 3."end": 8,
                     "value": "phone"."entity": "device"}]}}Copy the code

Note: We observed that MITIE was more accurate than spaCy + Sklearn in small training sets, but MITIE’s training process became slower and slower as the “intention” sets increased. For a collection of 200 + examples with about 10-15 “intents”, MITIE takes about 35-45 minutes to train it on AWS C4.4 Xlarge instances (16 cores, 30 GB RAM).

This is a good tutorial related to MITIE back-end training Rasa. If you are a beginner, you can install Rasa by referring to this document.

Botkit and Rasa integration

Botkit is an open source robot development framework designed by the creators of Howdy. It basically provides a set of tools to build bots on Facebook Messenger, Slack, Twilio, Kik, and other popular platforms. They also offer an IDE for robot development called Botkit Studio. All in all, Botkit is a tool that lets you write it once and deploy it to multiple messaging platforms.

Botkit also provides support for media software that extends Botkit’s capabilities. These media software provide Botkit with integrated interfaces with databases, CRM, NLU and statistical tools, making the framework more scalable. This design also allows us to easily add Botkit’s ability to integrate with other tools and software by writing media software modules for it.

In this demo, I integrated Slack and BotKit. You can use this template as a template to set up Slack modules for Botkit. Here we have extended botKit-Rasa media software, you can find them here.

Botkit-rasa has two functions: receive and listen, which override Botkit’s original default operations.

1. receiveCalled when Botkit receives a message. It sends the user’s message to Rasa and stores “intents” and “entities” in BotKit’smessageIn the object.
2. hearsOverlays the original Botkit.hears“Method –The controller hears.The defaulthearsMethod uses regular expressions to search for a given pattern in a user’s message, while the “HEAR” method from botkit-Rasa media software does this by retrieving “intent.”

let Botkit = require('botkit');
let rasa = require('./Middleware/rasa')({rasa_uri: 'http://localhost:5000'});
let controller = Botkit.slackbot({
  clientId: process.env.clientId,
  clientSecret: process.env.clientSecret,
  scopes: ['bot'],
  json_file_store: __dirname + '/.db/'}); / / rewrite botkit receiving method of the controller. The middleware. The receive. Use (rasa. The receive); // Rewrite the botkit listener controller.changeears (function (patterns, message) {
  return rasa.hears(patterns, message);
  });
  controller.setupWebserver(3000, function(err, webserver) {/ / configure a can be get from slack webhooks the path of the controller. CreateWebhookEndpoints (webserver); });Copy the code

Let’s take an example: “My phone is not turned on.” Rasa returns the following:

• Intent: The device is down
• Entity: Device = phone

If you notice, the input I’m providing doesn’t exist in my training profile, and Rasa’s built-in intelligence correctly identifies these combined “intentions” and “entities.”

We need to add a hears method that listens for device downtime intent to process the input message. Remember that the “intents” and “entities” returned by Rasa will be stored in message objects by the media software.

controller.hears(['device_failure'].'direct_message,direct_mention,mention'.function (bot, message) {
  let reply = 'Try pressing the power button for 30 seconds. ';
      reply += 'Bring the phone to the service center if it does not start. ';
      reply += 'Do not forget to carry your warranty card.';
      bot.reply(message, reply);
      });Copy the code

You can run the bot through Slack and see the output shown below (Support_bot is the name of my bot).

You should now be familiar with the process of building conversational robots using the Robot development framework and NLU. Hopefully this article will help you start building your own robots faster.

Translator: KX_WEN, from Cloud + community translation service

The original link: https://dzone.com/articles/building-an-intelligent-chatbot-using-botkit-and-r

Original author: Arjun Hariharan