1. Home
  2. Docs
  3. Tutorials
  4. A/B Testing

A/B Testing

A/B testing of conversational applications; Alex skills, Google Home actions or a chatbot, is a method of comparing two versions of the app against each other to determine which one performs better. AB testing is essentially an experiment where two or more variants of answers are returned to users at random, and statistical analysis is used to determine which variation performs better for a given conversion goal.

In this tutorial, we will learn how you can use Optimizely to test different versions of your conversational application, how we can fetch the experiment data from Optimizely and use it to test the different the response options.

Some of the information in this guide is more developer-centric, but don’t be discouraged. If you follow the steps as they’re outlined, you will have a deployed, working “what happened today” app.

Create a new Experiment

  1. Login to Optimizely console. If you don’t have a user create one.
  2. Create a new Node project named “whatHappenedToday”
  3. Select Create new experiment
  4. Use “whatHappenedToday” as the experiment key
  5. Created two variation keys named:
    1. “happened”
    2. “born”
  6. Click “Overall Revenue” and add the default option
  7. Add a new event named “moreEvents”
  8. Click Create Experiment

Locate your project datafile

In the Optimizely console, click setting and find the URL of your project data file. We will need it later in our conversational application code.

Create a conversational app

A Conversation.one application represents your conversational interface through Amazon Alexa, Google Home or Facebook Messenger bot. To create an app:

  1. If you don’t already have a Conversation.one account, sign up. If you have an account, log in.
  2. Click on Create New App in the navigation. If it’s your first application, you will be redirected to the Create New App wizard automatically.
  3. Select your industry and enter your company name. For the purpose of this tutorial, select “Other” for the industry, and enter “What Happened Today” as your company name. Uncheck “Add demo data” to start with an empty application. Click Next to continue.
  4. Select the devices you want to support with your app. For this demo select Messenger, Amazon Alexa, Google Home, and Chatbot. Click Next to continue.
  5. Your new application is ready. Click Customize to start developing your app.

Create an intent

An intent maps what a user says with what your app does. The intent we will create now will be invoked when the user asks for information on what happened today.

To create an intent:

  1. Create a new intent named “whatHappenedToday”
  2. In the User Says section, enter examples of what you might expect a user to ask for. The more examples you provide, the more ways a user can ask a question and the conversational app will understand. Enter these examples:
    • What happened today in history
    • What happened on this day in the past
    • What took place today
    • What occurred today
  3. Click Save to apply your changes
  4. Set the intent response to the following code
    return whatHappenedTodayFunction(context, req, res);

Add the business logic code

In the business logic tab add the following code:

var datafileURL = "https://cdn.optimizely.com/json/10173612600.json";
var optimizely = require('optimizely-server-sdk');

function initOptimizely(context, req, res) {
    if (req.getSession().get("DataFile")) {
        var dataFile = JSON.parse(req.getSession().get("DataFile"));
        var optimizelyClient = optimizely.createInstance({
            datafile: dataFile
        });
        return Promise.resolve(optimizelyClient);
    }
    return fetch(datafileURL)
        .then(function(res) {
            return res.json();
        }).then(function(dataFile) {
            req.getSession().set("DataFile", JSON.stringify(dataFile));
            optimizelyClient = optimizely.createInstance({
                datafile: dataFile
            });
            return Promise.resolve(optimizelyClient);
        });
}

function getUserId(context, req, res){
    if (req.getSession().get("UserId")) {
        return req.getSession().get("UserId");
    }
    var randomstring = require("randomstring");
    var userId = randomstring.generate(7);
    req.getSession().set("UserId", userId);
    return userId;
}

function whatHappenedTodayFunction(context, req, res) {
    var today = new Date();
    var day = today.getDate();
    var month = today.getMonth() + 1;
    return fetch("http://history.muffinlabs.com/date/" + month + "/" + day)
        .then(function(res) {
            return res.json();
        }).then(function(json) {
            var suffix = "";
            var events = json.data.Events;
            return initOptimizely(context, req, res).then(function(optimizelyClient) {
                var variationKey = optimizelyClient.activate('whatHappenedToday', getUserId(context, req, res));
                if (variationKey === 'born') {
                    events = json.data.Births;
                    suffix = ", was born";
                }
                var selectedEvent = events[0];
                res.say('On this day,' + selectedEvent.year + ' - ' + selectedEvent.text.replace(/(\(.*\))/g, '') + suffix);
                if (events.length > 1) {
                    req.getSession().set('onYes', JSON.stringify({
                        callback: 'whatHappenedTodayOnYes',
                        data: {
                            lastIndex: 0,
                            suffix: suffix,
                            events: events
                        }
                    }));
                    req.getSession().set('onNo', JSON.stringify({
                        callback: 'whatHappenedTodayOnNo',
                        data: {}
                    }));
                    res.say(". Would you like to hear more?");
                } else {
                    sdk.conversationHelper.addAnythingElseMessage(res);
                }
            });
        }).catch(function(err) {
            res.say(err);
        });
}

function whatHappenedTodayOnYes(context, req, res, onYesData) {
    return initOptimizely(context, req, res).then(function(optimizelyClient) {
        optimizelyClient.track("moreEvents", getUserId(context, req, res));
        var eventIndex = onYesData.lastIndex + 1;
        var events = onYesData.events;
        var selectedEvent = events[eventIndex];
        if (eventIndex + 1 < events.length) {
            res.say('On this day,' + selectedEvent.year + ' - ' + selectedEvent.text.replace(/(\(.*\))/g, '') + onYesData.suffix);
            req.getSession().set('onYes', JSON.stringify({
                callback: 'whatHappenedTodayOnYes',
                data: {
                    lastIndex: eventIndex,
                    suffix: onYesData.suffix,
                    events: events
                }
            }));
            req.getSession().set('onNo', JSON.stringify({
                callback: 'whatHappenedTodayOnNo',
                data: {}
            }));
            res.say(". Would you like to hear more?");
        } else {
            sdk.conversationHelper.addAnythingElseMessage(res);
        }
    }).catch(function(err) {
        res.say(err);
    });
}

function whatHappenedTodayOnNo(context, req, res, onNoData) {
    sdk.conversationHelper.addAnythingElseMessage(res);
}
  • datafileURL should be changed to your own Optimizely project datafile URL
  • The function initOptimizely, is downloading the experiment file and save it inside the session
  • The function getUserId generates a random user id, this is useful when testing the code
  • In line 45, we run the experiment and choose between an event and birth response
  • In line 77, we mark a success, which in our case is whenever the user listened to more events or births.

Try it out

Now that your app can use an A/B testing experiment, try out what you have so far.

In the chat console on the right, type in a request. The request can be a little different than the examples you provided in the User Says section. This can be something like “what popped up today”. After you type the request, hit “Send”.

The chatbot will answer with some interesting event from the past. For example “On this day,1974 – The Rumble in the Jungle boxing match between Muhammad Ali and George Foreman takes place in Kinshasa, Zaire. Ali wins by KO in the eighth round, regaining the title of World Heavyweight Champion and causing Foreman´s first professional defeat.”

When asked “Would you like to hear more?” answer “Yes”.

Click logout to start a new experiment, and repeat it few times, in some cases say “Yes” and in others say “No”.

Now you should see your experiment results in Optimizely dashboard.

Was this article helpful to you? Yes No

How can we help?