メニュー

Expand
Rate this page:

Thanks for rating this page!

We are always striving to improve our documentation quality, and your feedback is valuable to us. How could this documentation serve you better?

How to train bots in production

This guide will give you resources and best practices needed to set up tools and processes to continue training your bots after they are deployed to production and handling live customer conversations. It answers the following key questions:

  • Why should I continue training my bot in production?
  • Where can I find the training data?
  • What tools does Autopilot provide for on-going training?

To implement the suggestions in this guide, please enable the Log Queries property on your bot. You will not be able to use the REST API or CLI to retrieve past queries from your users if this property is disabled.

Queries

Queries represent words or phrases used by your users when interacting with the bot. By default, Autopilot logs all queries uttered on each channel. They are your primary source of data for training bots in production. Queries can be accessed via the Queries endpoint and viewed in the Queries page in the console.

Training Data Types

The natural language understanding models used by Autopilot use 3 types of training data — Samples, Field Values and Tagged Fields.

Samples

Samples are assigned to Tasks, with each Task requiring training on a corpus of samples for it to be triggered by the bot. As we’ll see below, Queries need to be turned into Samples so that they can be added to the bot’s natural language understanding model. Samples can be accessed via the Samples endpoint on each Task or by clicking into the Task in the console.

Field Values

Values are also part of the training data used to build the natural language understanding model. They represent sample values associated with a Custom Field Type. For example, Magenta could be a Field Value for a Custom Field Type called Colour. If your bot is likely to encounter new Field Values, the training process you implement will also need to add them to the model.

Tagged Fields

If your use case requires the bot to extract Fields (both Custom and Built-In) from user utterances, you may also need to tag them in Samples. Field tagging involves specifying which locations in a Sample contain Fields. It involves replacing the Field Value with the unique name of the Field in curly braces in the Sample. For example, here’s how you’d tag Fields called `shoe_type` and `colour` :

  • ‘Do you have the new Skechers in red?
  • ‘Do you have the new {shoe_type} in {colour}?

You can also learn more in the API Reference.

Training Tools and Processes

The overarching objective of on-going training is to ensure your bot becomes more accurate over time by incorporating data from actual customer conversations into its natural language understanding model. This involves -

  1. Improving the accuracy of matching intents to Tasks.
  2. Finding new Field Values to incorporate.

We’ve listed three different ways you can accomplish these tasks below.

Queries page

The Queries page in the Autopilot console provides a set of pre-built training tools for incorporating data from real customer conversations into the model.


Improving the accuracy of matching intents to Tasks

Screen Shot 2020-06-30 at 9.53.33 AM.png

  1. Autopilot Queries can have one of three statuses — reviewed for queries that have been used as training data, discarded for queries that don’t need to be used and pending-review for queries that still need to be reviewed.
  2. You can filter queries by their status, the tasks they were matched to and the channel. Start by filtering queries with a pending-review status.
  3. Queries that triggered the fallback task will say `no task` in the Task column.
  4. Next, if you find Queries that were matched to the wrong Task, add them as Samples to the appropriate Task. In the example above, “Do you do buzz cuts?” should have been matched to the `service_inquiry` Task.
  5. When you assign a Task to this Query, it’s status gets changed to `Reviewed` and it is filtered out from your view. It is also automatically created as a Sample.
  6. Once a Query is reviewed, it can not be added again as a Sample.
  7. You can repeat this step periodically as your bot handles more Queries.

Screen Shot 2020-06-30 at 9.56.23 AM.png


Finding new Field Values to incorporate

Let’s say you have a Custom Field called Custom.SERVICE that helps your bot recognize if the user is inquiring about a particular service.

Screen Shot 2020-06-30 at 10.27.00 AM.png

If they use a new word or phrase for one of these services, or ask about a service that isn’t part of the Custom Field, you should add it to your bot’s training data.You should create a synonym if it has the same meaning as an existing value. You can do this using the add value button on the Query page as shown below.

Screen Shot 2020-06-30 at 10.25.44 AM.png

REST APIの使用

If you need capabilities beyond those offered by the Queries console page, you can integrate the REST API with existing dashboards, workflows and other tools, or build a custom training tool.

Improving the accuracy of matching intents to Tasks

1. Retrieve Queries
Start by using the Queries API list resource to retrieve the Queries you need to incorporate into your training data. This endpoint supports filtering by Query status (discarded, pending-review and reviewed) and allows you to retrieve upto 1000 items in one request using the `PageSize` query string parameter, similar to Twilio’s default pagination capabilities. Make sure you use the filter to only pull queries that still need review. Each Query has a property called `query` that includes the user’s word or phrase and `results` that includes the Task matched and Fields detected by the bot, if any.

2. Create Samples
Next, use the Samples API endpoint to add Queries as Samples. You need to make a POST request to this endpoint with the text of the Query included in the `TaggedText` parameter.

3. Update Queries
Finally, update the status of the Query you just used to create the Sample to ‘reviewed’ so that you only pull new Queries into your training workflow.

Finding new Field Values to incorporate

1. Retrieve Queries
Similar to how you improve the accuracy of matching intents to Tasks, you’ll need to first retrieve Queries using the Queries API list resource.

2. Create Field Values
Next, you’ll need to use the Field Value API endpoint to add the Query as a new Field Value. You need to make a POST request with the text of the Query in the `Value` parameter. If the new value is a synonym of an existing one, include it in the `SynonymOf` parameter.

3. Update Queries
Finally, update the status of the Query you just used to create the Sample to ‘reviewed’ so that you only pull new Queries into your training workflow in the future.

Using the CLI

The Autopilot Plugin for the Twilio CLI lets you interact with the API directly via the command line. You can learn how to set up and use it in the docs and in the repo.

Improving the accuracy of matching intents to Tasks

1. Retrieve Queries
You can use the `twilio autopilot:queries:export` command to export Queries into a csv file containing the following columns corresponding to properties of the Query API resource. The columns needed for improving the accuracy of matching Tasks are in bold.

query

Status

Source channel

Date created

Date updated

Task unique name

Field name

Field value

Field type

Language

AssistantSid

2. Create Samples
You can use the `twilio autopilot:samples:upload` command to create Samples. You need to provide a Task SID and a csv file containing the Samples to the command. Each Sample should be on it’s own row in the first column of the file. You can edit the csv provided by `twilio autopilot:queries:export` to simplify the process.

3. Update Queries
Unfortunately, the Autopilot Plugin for the Twilio CLI does not have a command for updating query status currently. You’ll need to do this directly via the REST API.

Finding new Field Values to incorporate

1. Retrieve Queries
Similar to how you improve the accuracy of matching intents to Tasks, you’ll need to first retrieve Queries using the `twilio autopilot:queries:export command.

2. Create Field Values
You can use the `twilio autopilot:fieldvalues:upload` command to create new Field Values. You need to provide the Field Type SID and a csv file containing the Field Values to the command. Each Field Value should be on it’s own row in the first column of the file, with synonyms for each field in subsequent columns. You can edit the csv provided by `twilio autopilot:queries:export` to simplify the process.

3. Update Queries
Unfortunately, the Autopilot Plugin for the Twilio CLI does not have a command for updating query status currently. You’ll need to do this directly via the REST API.

Building the Model

You must build a new model so that your bot uses the training data you just added when processing queries. There are three ways to build a model with Autopilot currently.

Console

Screen Shot 2020-06-30 at 10.41.01 AM.png

You’ll see a notification when Autopilot recognizes that new training data has been added. As a general best practice, build the model after reviewing as many Queries and Fields as possible.

api

A POST request to the ModelBuild endpoint will trigger a new model to be built.

CLI

The twilio autopilot:modelbuilds:create command will trigger a new model to be built.

Rate this page:

ヘルプが必要ですか?

誰しもが一度は考える「コーディングって難しい」。そんな時は、お問い合わせフォームから質問してください。 または、Stack Overflow でTwilioタグのついた情報から欲しいものを探してみましょう。