A/B Split Tests

This document will describe how to setup an A/B split test so that you know what content on your web page is most likely to convert visitors into customers more effectively. The whole purpose of doing a split test is to divide customers into groups and provide them with separate versions of the same page so that you can monitor which version is more successful when it comes to conversions.

Based on the reports generated, you would know Split test cases which can also be used to add complexity and experimentation to the content shown in a Content Area and to test the effectiveness of certain templates. It is a very powerful marketing tool.

By the end of this module, you will learn how to:

1. Add and edit Split test cases and add groups.
2. Add personalized content for user groups
3. Set up an experiment on Google Analytics
4. Integrate Google Analytics experiment with the Avetti split test feature so as to develop detailed reports.

Split test process #

As for the purpose of experimentation, we create two or more user groups and separate templates for each group such that a user belonging to group A would see a different content on the same page when compared to a user belonging to group B. A user who has been exposed to a particular content will continue to see the same content when he comes back to the same page. As the experiment progresses through days, it would become evident that users of a particular group might be making more purchases or transactions compared to the other groups. From this you know what content is most appropriate to be put on your page so as to maximize conversions.

The reason we chose to integrate with Google Analytics is because of the rich reports and graphs that it can generate. Remember that the experiment content set up and decision over what variation is to be shown is completely defined within the Avetti system. We are merely sending the experiment information to Google Analytics so that they can plot this information with graphs and custom reports.

At this point, we assume that you already have a Google Analytics account and that you have your pages being tracked.

In case you don’t, please follow these steps:

1. Sign up for an account in Google Analytics (http://www.google.ca/analytics/).
2. The Google tracking code has already been added to the footer template. So when you are asked to add the tracking code to your web page, simply go to the footer template and uncomment the tracking code section.
3. Add your account number at the space provided. Once tracking is enabled you can verify this by clicking on admin | Tracking info. The status would stay ‘tracking code installed’ or ‘receiving data’.
4. Once you have your tracking code setup and the pages are being tracked successfully you can start off an experiment. Go to the Reporting tab, select Behaviour on the left hand menu section and click Experiments.
5. If it’s your first time on the experiments page, it will ask you for the page to test. Enter the URL of the page that you are conducting the test for and click start experimenting.

You need to go through the following four steps to complete the experiment setup process:

1. Provide information:You need to enter details, such as name of the experiment, experiment objective and email notifications. Percentage traffic is determined by Avetti Commerce, changing the value here will not make any difference to the percentage of users exposed to the experiment. Under Objective for this experiment, select the metric that you want to improve, such as pageviews, transactions, or goal completions
2. Setting up the variation pages: Since what we are doing is a content experiment that involves different contents but all in the same URL, this step is not very significant. Simply enter the URL for the page that we are testing as the original URL and enter any random value as the URL for the variation page. However, name the original page as ‘Group A’ and the variation page as ‘Group B’. This will help us later on as the experiment starts running.
3. Set Experiment Codes:You would normally paste the experiment tracking code to the original page, but for now simply ignore this step, since we’ll be modifying the page and showing variations from our end. The experiment code provided by the Google web interface will not be used. Instead, get the Experiment ID which will be used later on to implement the experiment. You can find the experiment ID by clicking on the Manually insert the code button.

Note

1. URLs are not applicable for this implementation. However, they are required to configure the experiment. You can ignore any errors or notifications such as “We could not find the URL you entered”
2. The code validation errors can be ignored as the URLs provided during setup wont be used for the experiment. Changes are made on the same page with the same URL to show variations.

Now that you have setup a new experiment in Google Analytics and obtained the experiment code, it’s time to tell Google what to track. As mentioned earlier, what we are trying to do is a content based experiment. We basically create two separate contents so that one user group is exposed to one specific content while the other group will be shown the other content irrespective of the fact that they are all viewing the same page with same URL.

To set up the split test feature, we will be working with two separate sections on the Admin page.

1. Add a new split tests case and edit it so as to add the user groups. This can be done under Marketing | A/B Split Tests.
2. Specify what content is to be shown to a particular user group and when it is to be shown. This is done by editing personalization details under Templates | Catalog Personalization page.

Add/Edit Split test case #

To add a Split test case,

1. Go to Marketing | A/B Split Tests. You will be taken to the Manage Split test case window.
2. Click the Add button to open the Edit Split test case window and add a new case.
3. Enter the details (Refer table 1).
4. Click the Add group button to open the Edit split test group box.
5. Enter the details (Refer table 2).
6. Click Update and Ok button to save changes.
7. Once you have finished creating your Split test case, you must add it to a Content Area on the Templates | Catalog Personalization page (Refer Add/Edit Personalization Details section)

Field	Description
Case Type	Set the case type to ‘Google Analytics’ as we will be generating the report on Google Analytics.
Case Code	A code to represent the Split test case which is used for administrative purposes.
Available	Check this to ensure that the Split test case is active. If this is unchecked, the templates will not be shown.
Experiment ID	Enter the experiment ID obtained while setting up the experiment in Google Analytics. It links the split test that we are setting up in Avetti with the experiment that has been set up in Google Analytics.

Fields	Description
Group Code	The code that represents a group in the Split test case. The first group (Group A) should have code set to 0. The Second group (Group B) code should be 1 and so on. This is a very crucial step as the Google reports rely majorly on these values.
Group Name	Enter a brief name or description about the group.
Display Method	In order to have a template displayed, it must first have a valid start/end date and be set to available. There are three options for the Display Method, which determines how to cycle through multiple templates. Priority by Date/Time: Always displays the valid template entry with the highest priority. The template entry will have options for start/end time, priority and available. If there are multiple templates available within the start/end dates, the template with the lowest number (same as highest priority) will be displayed. Under this Display Method, only one template will be shown for the given group until it’s not within the start/end dates, it is no longer available or its priorities are altered in such a way that the first template no longer has highest priority. Sequence by Date/Time: Each time a new template is requested, the next template in the Content Area list will be loaded. It will find the current template sequence number (defined later) from a given customer’s cookie and display the template with the next sequence number. Random by Date/Time: Templates are shown at random on each template request. Using this method, the templates can also be weighted so some templates have a higher chance of being shown over others; this is specified on the Define Split Test on Content Area page as Display Weight.
Display Percentage	Display percentage (DP) determines how to distribute customers into split test groups. Using this option, you can specify groups of equal sizes, or you can specify one group to be larger than the other. In most cases you may wish to keep the DP the same for each group, but if you require one test group to have a larger sample size than others, use this feature. The math is as follows: Desired group size = (# of customers) * (DP of desired group) / ([DP of A] + [DP of B] + [DP of C] + …) Example 1:  There are groups A and B. If you wish group A to be the same size as group B, set the DP of each group to be the same (such as 100). In this case, if 20 customers are assigned to a group, each group would have 10 customers. The size of each group would be: 20 * 100 / (100 + 100) = 10 Example 2:  There are groups A, B and C. If you wish group A to be the largest, while B and C are equal but smaller, set the DP of A to 100 and the DP of B & C to 50. Assume customer base is 20. The size of group A would be: 20 * 100 / (100 + 50 + 50) = 10 The size of each group B & C would be: 20 * 50 / (100 + 50 + 50) = 5
Time to Display	Due to the customer’s ability to refresh the page many times in a short time, the templates shouldn’t switch on each page refresh. This feature will keep the current template active, through page refreshes, for as long as the Time to Display feature permits (in seconds, since the first time the page was loaded).

Add/ Edit Personalization Details #

This section is used to search for a template on any available skin.

1. Refer Catalog Personalization for instructions on add/edit personalization details.
2. In add/edit personalization page, Change the Content Type to “Split Test / Sequence of Templates”.
3. Select the desired Split Test Case from the new dropdown below. If there are no Split Test Cases available, you must create one (which is covered in the above section).

You can add new templates once you have created this content group, as it can only be done in the Edit content group page. After the content group has been created, edit it and click on the new “Add Template” link next to the specified Split Test Case name.

On this page, the created groups will appear in their own box, where you can customize several things. Start by editing one of the Split Test Groups, and add a new template.

Enter the file name of an existing template that you wish to display for the specified group. Note: The contents of this template is what will be shown to users in this particular group once the experiment starts. If you have not created one, use the template manual to learn how to create a template. Create separate templates for each group. Example: As an example we have created 2 templates. Template A for group A and Template B for group B. The contents of the templates are to simply display a text highlighting which group you are in. We added the following script to Template A.

1| <h1>

2| You are in group A.

3| </h1>