Testing

Motivation

To test your bot, you can use built-in test feature Try It Now to uncover bugs and prevent regressions. You can provide user input as text step by step, or you create and execute test case as needed.

Structure Input

When interacting with Try It Now, you can provide user input as text like "Get me two tickets for Star Wars", or a json format represents the same semantics which is used to describe filling for some data structure or provide values for its slots.

Framely dialog engine can be interacted directly in Frame Event in its json format, which allows for easier bug fixes on the interaction logic by separating the dialog understanding issues.

For example, the user utterance Get me two tickets for Star Wars can be encoded as following json structure:

[
  {
    "type": "BuyTicket",
    "packageName": "test001.BuyTicket",
    "slots": [
      {
        "value": "\"starwars\"",
        "attribute": "movie",
        "type": "test001.BuyTicket.Movie"
      },
      {
        "value": "2",
        "attribute": "number",
        "type": "kotlin.Int"
      },
    ]
  }
]

In Try It Now, you can set this structure in the Structure Input field:

Before connect

After connect

Then you can get the conversation in Try It Now like:

Test Cases

The goal of golden test cases is to help you verify that the modifications made to your bot did not modify its behavior in unwanted or undesirable ways. In other words, test case execution verifies chatbot responses have not changed for the same inputs.

To create a test case, you should simulate the path of the conversation you want to save in Try It Now and provide a test case display name. After saving, you can view them in the left sidebar menu Test Cases, which shows the test name, the latest test time and the latest test result.

To run the test case, you can select the one you want to run. The test engine will check the following types of data turn by turn to evaluate the test result:

Test will Pass when the actual output is the expected output.
Test will Fail when the actual output does not match the contents of the golden test case. You then need to figure out whether this is a bug or an intentional change.

How To Use

Try It Now will only test the instances you committed, so you need to commit your chatbot on both structure side and language side. This means that whenever there are new changes, Commit is required, and the new changes will be tested in try it now. Otherwise you will still test the past performance.

Figure 1: click commit on STRUCT side

Figure 2: click commit on EN side

The instructions below show you how to test your bot.

Try It Now

Select the language bot you want to test, click Commit icon in the right sidebar.

Click Try It Now icon in the right sidebar, then the test field will slide out.

Enter Structure Input if you needed and click Connect. In this case, we just need to click Connect.

Chat with the bot to create a conversation that covers the functionality you want to test.

Verify correct values for the triggered type, the response, and the session parameters with State.

Create a Test Case

Click the Save Test Case icon in the topbar to save a conversation as a test case.

Enter a test case display name, for example happy path, click Save to save the test case..

Click the Reset Contexts icon in the topbar to restart a conversation to test.

Run Test Case

Click Test Cases in the left sidebar menu.

Select the test cases you want to run, and click Run.

The test starts running and the result will be updated when it is completed.

# Testing

# Motivation

# Structure Input

# Test Cases

# How To Use

# Try It Now

# Create a Test Case

# Run Test Case