Building Chatbot on FPT.AI platform

Natural Language Processing - NLP

Train your Bot to understand your customers’ questions.

Natural Language Processing - NLP is a core AI feature of FPT.AI Conversation. For your Bot to be able to understand user sentences, bot builders will need to teach in specific knowledge, including:

• Sample: Users’ utterances to inquire about topics of interest.

• Intent: Purposes of users’ utterances.

• Keyword: Important and necessary information in utterances, which help the bot to understand topics to give suitable responses.

• Entity type: Representing the meanings of Keywords.

• Dictionary: Add alternative words and synonyms for the Bot to better identify intents

Example:

Intent: I need to have 2 phones delivered to No. 17 Duy Tan Street
Intent	Keyword	Entity type
Buy	2	Quantity
	phones	Product
	No. 17 Duy Tan Street	Address

Notice:

• Samples are extremely important to chatbots. Therefore, you should add as many samples as possible. Diversity in samples will help your chatbot better understand different utterances from users.

• You should identify the top Intents based on customers’ conversation history. Then, add to the sample amount and diversity of those intents.

Sample

A Sample can be any utterance from users, and each sample will contain a specific intent. To train the bot to understand different utterances from users, you will need to make various samples with the same intent.

There are 2 methods to add samples and intents to the system:

Method 1: Add samples manually

• Step 1: In NLP, select Sample (1) -> Create sample (2)

• Step 2: Enter the sample content (3)

• Step 3: Enter the matching intent

• Step 4: Click Save to finish creating the sample.

Method 2: Insert samples from Excel

• Step 1: In NLP, select Sample (1) -> select Insert samples from Excel

• Step 2: Select Browse to upload your excel file

• Step 3: Select Confirm to finish

Notice:

• The excel file with Samples and Intents has to be formatted as follows:

• After finishing creating Samples and Intents for your bot, you will need to click Train button to train your bot with the new knowledge.

• You can also review your bot understanding by clicking Test NLP on the interface, Enter the Sample, and check the returned result.

Keyword and Entity type

Keywords are important information in samples, which influence the bot’s answering logic. Each keyword represents an actual meaning, called an Entity type.

Example 1: Object Entity

Sample: “What is the price of iphone X”, “How much does a Samsung s9 cost”

• Both of the above samples mean to ask for the price. Therefore, they share the Intent of “what price”.

• However, these 2 products have different prices. Therefore, the bot will need to respond with 2 different answers.

• Conclusion: iphone X and Samsung s9 are Keywords. These are 2 types of products. Therefore, the entity type here is “product”.

Example 2: Information Entity

Sample: Please deliver my goods to 17 Duy Tan Street, phone number 0123456789

• The Intent of the above sample is “delivery”.

• The above sample includes 2 important information that will decide the accuracy of the delivery. Therefore, 17 Duy Tan Street is the keyword that represents entity “address”; while 0123456789 is the keyword that represents entity “phone number”.

Identification of the entities like above examples is the basis for Bot response setup.

In example 1 – Object Entity, Bot will need to provide 2 different answers. (See how to setup responses for Object Entity at Variable Condition Review).

In example 2 – Information Entity, alongside responding to the customers, the Bot will also record customer information onto the Bot management system. In order to this, bot builders will need to use a JSON API card as the transfer medium.

Depends on each problem, FPT.AI Conversation will allow the Bot builder to train their Bot to understand correctly Keywords that correspond to trained entity types; or be able to recognize similar keywords (with machine learning). There are the 2 information matching methods: Exact match and Predictive match

A. Exact match

With exact match, the Bot will seek keywords and entity types in the Keyword list.

This method is often used in scenarios with set keywords – which are identifiable beforehand.

Example 1: “What is the price of iphone X”, “How much does a Samsung s9 cost”

The “product" entity Enter the 2 above sentences are “iphone X” and “Samsung s9”, and the chatbot had used exact match.

Example 2: In an insurance company that provides difference kinds of insurance:

• “Health insurance”, “Accident insurance”, “Vehicle insurance”… are Keywords

• “insurance product” is Entity type.

You will need to tag the Keywords with the above Entity Enter accordance with the exact match method, so that your bot can match from samples.

Step 1: In NLP, select Sample -> Tag Keyword with corresponding Entity type.

Step 2: Go to NLP -> Select Entity type-> Change the entity type matching method into Exact match

Step 3: In Keyword, the system had recorded tagged keywords (highlighted) at the Sample menu. Here, you can add synonyms (words with similar meanings or abbreviations…) for these keywords, so that your Bot can recognize the user Intents correctly.

Step 4: Click Train button to train the Bot with new data.

Step 5: After successful training, you can review the Bot’s ability to recognize entities by clicking on Test NLP.

After training, the Bot will be able to recognize “ip X” as “iphone X”, as it has obtained data about the product. Should you check product words outside the trained synonym list, however, the Bot will not be able to recognize.

Real value is a permanent value in the keyword list. This will be used to design bot responses to each Keyword. (More details can be viewed at Variable Condition Review).

B. Predictive match

For Bots can find entity information using machine learning, you need to tag as many Keywords with their corresponding Entity types as possible. Here, Bot will use Samples tagged with Keywords to predict extra information (but do not seek the permanent keyword list like in Exact match).

This method is often utilized with information unidentifiable beforehand, like names, phone numbers, addresses…

Example: Please deliver my goods 17 Duy Tan Street, phone number 0123456789.*

The information entity “17 Duy Tan Street" and “0123456789” are examples of predictive match.

Step 1: Enter Sample (1) which includes important information entities. Choose Intent (2) and highlight Keywords (3) then select corresponding Entity type (4).

Currently, the platform provides 4 set types of Systematic entity, including $address (address), $datetime (date and time), $email (email) and $number (number), which will simplify your bot building process.

For other information, Enter new Entity type then click Save (6).

For example, there is no phone number in Systematic entity. So if you want to save the phone number Entity 0123456789 of the above example, type “phone_number" in the blank below Entity type.

Step 2:** Go to NLP -> Entity type -> Edit Entity type so change the “phone_number” entity into “Predictive match” or keep the default setting of “Both”.

Step 3: Different to the exact match method – adding synonyms of the keywords, predictive match utilizes machine learning for entity recognition. Therefore, you will need to tag many other similar keywords for the bot to understand and recognize the entity type.

Step 4: Click Training to teach the Bot newly updated entity information.

Step 5: Review the results by clicking on NLP Review on the interface. Then, Enter any random phone numbers different from the one taught to the Bot. The Bot will still be able to recognition that as a “phone_number” entity.

C. Systematic entity

These are entities pre-built by the FPT.AI Conversation platform, which allows users to get information on addresses, emails, dates and times, or any random numbers.

For this to work, you will only need to tag one information in the Sample with any of the above entity, then the Bot will understand and relevant information provided by the customer on default, without any extra entity training.

• $address: Address entity

Bot can recognize Vietnamese addressed from the most detailed to the most vague (home number, lane number…).

Example: “No. 17 Duy tan, Cau giay District, Ha noi". The real value result will be standardized via API (using the JSON API card)

• $datetime: Date and time entity

Bot can recognize time to hour and minute, and date to day month year.

Example:

• $email: Email entity

Bot can recognize any email entity.

Example:

• $number: Entities that are numbers

Bot can recognize any numbers like quantity, price…

Intent

Intents are purposes of users’ utterances, and you can create them beforehand using the conversational flow tree map in the Intent menu by:

• Step 1: In NLP, select Intent -> Enter Intent name

• Step 2: Enter the Content of the Intent. This will be displayed in bot suggestions when it is not confident of the answer

• Step 3: Select Add Intent to finish

Notice: When creating new Intents in NLP with samples from excel files, you will need to change the intent description into meaningful sentences.

Dictionary

Aside from adding to the sample diversity manually, you can add alternative words, synonyms, or existing phrases in Samples to the Dictionary. This will help improve the bot’s ability to recognize Intents.

Example:

We have a list of samples in the “promotion” intent:

When processing a new user utterance with the same Intent, the Bot will correctly recognize the intent, but with low confidence level. To improve Bot understanding, you will need to add more Samples of the same “promotion” intent like: “do you offer promo”, “promo?”… or add “promo” as an alternative word for “promotion”. Then, when customer utterances include the abbreviation “promp”, the Bot will be able to recognize that they are asking about promotions.

After entering the synonyms and alternative words in the Dictionary, click Train to teach the bot.

Perform a NLP review to re-evaluate the bot’s learning results: Bot can correct recognize with higher confidence level.