Labelling
Train your bot using real conversations
Review real user messages in Labelling to associate them with the right intent. By classifying user messages, your bot's AI model will improve and be enriched with real user phrases.
Labelling is also used to estimate the bot's Accuracy.
Labelling only works on AI-based start intents
Rule-based intents can be improved directly from the intent by adding additional keywords to be detected. We do not recommend using AI-based intents in the other User Actions.
Overview
The labeling work needs to be done regularly to progressively improve your chatbot understanding.
For each intent, you will receive a batch of sentences extracted from real user conversations so that you can label them.
The user messages selected come from real conversations. We exclude by default irrelevant ones like test conversations (from the platform), schedulers, and quick replies and we remove duplicates so that you don't need to label the same sentence several times.
Messages to label
Each message can be labelled in 2 ways:
✅ Confirm the message should trigger the selected intent
❌ Reject that the message belongs to the selected intent. Rejected messages will be sent to the Undecided messages for a final review.
Once all the displayed messages are labelled, you can approve the labelling session. A new set of messages will be displayed until all messages from the selected intent are labelled.
Read intent description and hints before labelling
The Intent description and hints help define the scope of the intent more precisely.
Both the description and hints can be edited from the right panel or from the intent page (that you can open in a new tab by clicking on the icon on the right side)
Intent description and hints to better understand the intent's scope
Then, pick another intent to label from the left panel and continue until you review all of them.
Remaining intents to label. They are ranked by the number of messages left to label.
Undecided messages
If a message is labelled negatively, it is directly sent to the Undecided messages section to pick the right intent.
Undecided messages that need to be linked to another intent or ignored by the training
At the top of the intent list, you will have Undecided messages that will gather all rejected sentences from intent batches.
An "Ignore all" button allows you to quickly get rid of all the Undecided messages if you don't want to consider them.
A message that was ignored once won't be "lost" forever
The next time the exact same message is fetched as part of a new labelling batch, it will directly be sent to Undecided again.
Labelling batch
Labelling works with batches of messages: it is a group of 300 messages, randomly selected across all intents of the bot.
Unlock labelling feature
If a bot has not yet handled 300 messages, a minimum of 100 messages is required to create a first labelling batch.
Test messages (sent from the platform), schedulers, referrals, and quick reply payloads are filtered out of the selection as it's not relevant to label them.
Messages that are labelled directly come from the real conversations between users and your chatbot.
To end a labelling batch and get a fresh one you need to label messages for each intent
AND label the undecided messages. When the whole session is approved, you can get a new labelling batch.
Train your NLU model after a batch is done
A few minutes after a batch is done labelling you receive a notification to train the Start User Action again
Accuracy rate calculation
When a batch of messages is done labelling:
- 80% of the labelled messages are automatically sent to the appropriate intents as training phrases and are used to improve the bot's understanding.
- 20% of them go to the validation set: they are used to measure the bot's Accuracy.
For instance :
- You label the first batch of 1000 sentences
- 800 sentences labelled go to the training sentences
- 200 sentences go to the validation set to calculate the accuracy rate
The accuracy rate is computed this way:
- Take the labelled message of the validation set
- Evaluate if the labelled message effectively goes to the intent attached
- Measure the percentage of messages correctly classified
The validation set contains a maximum of 1000 sentences. If you have more, the oldest ones will be destroyed.
Accuracy rate gives the most recent estimate
It is based on the latest validated labelling batch
Updated 6 months ago