Implement A/B tests for your Unity game with Firebase Remote Config

1. Introduction

In this codelab, you'll learn how to create a Remote Config experiment using A/B Testing for the sample game, MechaHamster: Level Up with Firebase edition, which you modified in Instrument your game with Firebase Remote Config.

A/B Testing with Remote Config lets you test changes to your app's UI, features, or engagement campaigns on a targeted audience before you roll them out to a wider audience. You can also use experiment results to:

• Determine which parameter value variants improve key metrics like revenue and retention.
• Discover which subgroups of users prefer which variants.
• Gather and store experiment data to perform additional analysis on the effects of setting different parameter values.

This is the essence of A/B Testing: it enables you to take code instrumented for Remote Config and launch experiments that control the values clients receive, based on Remote Config conditions (including Google Analytics user properties), percentage rollouts, Analytics conversion events, and some combination of these.

First, you will implement a condition that gates who will be included in the experiment by setting user properties based on user actions. Then, you will create an A/B Testing experiment that uses Google Analytics user properties to determine which clients are included in the experiment. And, finally, you'll use that data to understand more about your audience.

What you'll learn

• How to set up A/B Testing using instrumented Remote Config values
• How to use Google Analytics user properties as part of conditions for admission into A/B tests

Prerequisites

• Completed Instrument your game with Firebase Remote Config

What you'll need

• Unity 2019.1.0f1 or higher with iOS and/or Android build support

2. Enabling the Debug Menu

There is a Debug Menu hidden in the project, and the button to access this menu exists in the game but is not currently enabled. You must enable the button to access it from the MainMenu prefab.

1. In the Unity editor, select the Project tab, then under Assets, expand Hamster > Prefabs > Menus and click MainMenu.

2. In the prefab hierarchy, find the disabled sub-object named DebugMenuButton and click it to open it in the Inspector tab.

3. Within the Inspector tab, check the box at the upper left corner next to the text field containing DebugMenuButton to enable it.

4. Save the prefab.

If you run the game in either the editor or on your device, the menu should now be accessible.

3. Enable the Remote Config submenu

1. From the Project tab in Unity Editor, expand Assets > Hamster > Prefabs > Menus and double-click the DebugMenu object to open it in the editor Hierarchy tab.
   
   
2. In the Hierarchy tab, expand the hierarchy and click the sub-object under DebugMenu > Panel, which is labeled Remote Config Actions.

3. In the Unity Inspector tab, enable Remote Config Actions by checking the box to the left of the text field containing the object's name.

This has two GameObject children named Set Bored Of Subtitle and Set Enjoys Subtitle, which are both configured to call existing but unimplemented methods in DebugMenu.cs.

4. Reset the subtitle override to in-app default

In the previous codelab, you overrode a parameter's default value as JSON and used conditions to serve different variants. As part of this codelab, you will delete the condition you created and reintroduce the in-app default, and you will only override it with the result of A/B Testing.

To re-enable the in-app default:

1. Open the Remote Config page in the Firebase console and click the pencil icon next to the subtitle_override parameter to open the Edit parameter side panel.
2. Click the X icon next to the condition to delete it.
3. Next to the remaining default value, enable the Use in-app default toggle.

4. Click Save to save your changes, then click Publish changes to publish your changes.

5. Set User Properties in Debug functions

You will now write function bodies for some pre-configured but unimplemented Google Analytics functions in DebugMenu.cs (which can be found in Assets > Hamster > Scripts > States).

These functions set user properties, which are ways to describe segments of your user base and are used to record how the user feels about the game's subtitle.

Implement SetUserBoredOfSubtitle and SetUserEnjoysSubtitle by finding ​​the existing versions of them in DebugMenu.cs and overwriting them as follows:

void SetUserBoredOfSubtitle()
{
  Firebase.Analytics.FirebaseAnalytics.SetUserProperty("subtitle_sentiment", "bored");
}

void SetUserEnjoysSubtitle()
{
  Firebase.Analytics.FirebaseAnalytics.SetUserProperty("subtitle_sentiment", "enjoys");
}

If your app is correctly configured with Google Analytics, you can call one of these functions to make the property available to Remote Config's conditions. To call SetUserBoredOfSubtitle from a mobile device, start the game and press the Debug Menu button in the main menu, then press Set Bored of Subtitle.

6. Create a Custom Dimension

Next, you will set up various subtitle variants for the subtitle_override to see which subtitle works best. But, in your A/B test, you will only serve these variants to users whose feelings about the current subtitle (as recorded in subtitle_sentiment) include the word "bored."

You'll use a custom dimension to create and track custom parameters on Analytics events. See Custom dimensions and metrics for more information.

To create a new custom dimension:

1. Open the Firebase console, expand the Analytics menu, and select Custom definitions.
2. From the Custom definitions page, click Create custom dimensions.
3. In the New custom dimension window, set the Dimension name to "Subtitle Sentiment" and, from the Scope drop-down, select User.
4. For the User property field, select subtitle_sentiment.

7. Set up the A/B Testing experiment

Next, create an A/B Testing experiment to set different values for subtitle_override to be tested against each other in order to optimize two to three day user retention.

1. First, select Edit on the subtitle_override parameter on the Firebase console Remote Config page:

2. From within the Edit parameter dialog that appears, click on Add new.

3. From the list that appears, select Experiment.

4. Enter a name and description for your experiment.

5. Next, choose targeting conditions. First, select your App from the drop-down.

6. Next, click And to add a new condition, then select User Property and choose subtitle_sentiment. If it does not appear, manually enter it.
7. Since you only want to set a subtitle for those whose current subtitle sentiment includes "bored," choose contains and type in bored.
8. Optionally, choose which percentage of the audience that matches the above criteria to expose to the test. Select 100% to avoid randomness you can't easily control.

9. Next, select a goal that the test will try to maximize for. Pick Retention (2-3 days).

10. Next, set up the parameters of the experiment and create different subtitle variants . These variants are the different values A/B Testing will serve to users whose subtitle_sentiment contains ‘bored,' and A/B Testing will determine which variant is best for maximizing retention.
11. Enter the following parameter value for Variant A:

    {"text":"A: A game about a mechanical hamster","fontSize":8,"textColor":{"r":0.0,"g":255.0,"b":0.0,"a":255.0}}
    

12. Enter the following parameter value for Variant B:

    {"text":"B: A game about a mechanical hamster","fontSize":14,"textColor":{"r":255.0,"g":0.0,"b":0.0,"a":255.0}}
    

13. Set the variant ratios with whole numbers as follows:
    • Baseline: 1
    • Variant A: 100
    • Variant B: 100
    This will result in total weights of 0.5% for the Baseline, 49.8% for Variant A, and 49.8% for Variant B.These variants specify that one out of 201 times, A/B Testing serves the default to those who are bored of the subtitles, but 200/201 times, it will serve one of two new values and override the title screen.
14. Save and submit by clicking Start experiment, then click Start on the confirmation popup.

8. Run the Set User Property method and refresh

You now have set various properties about the user, which then can be used to direct the logic or presentation of your game.

If you have not previously run the SetUserProperty method or have set it to enjoys, you should still see the default subtitle when opening the game.

If you have set it to bored (before re-fetching), you should see one of the new values in a roughly 50/50 ratio.

Once a device has entered into an A/B Testing experiment, the values they receive from that experiment will not change and are persistent per installation. As a consequence, to receive one of the other experimental values, you must create a new install either by reinstalling the game to the same device/simulator or installing the game to a new device/simulator.

In a real A/B test launched across your user base, you should give the baseline a similar weight to the other variants. But in this case, you assign very skewed probabilities to validate that the experiment is working. If (in the 1/201 case), you still receive the default value, try reinstalling the game to your device/simulator.

Another effect of this is that switching the user property back to enjoys will not change the value back to the baseline but again, you can do this by switching to enjoys and reinstalling.

9. Congratulations!

You've used Remote Config A/B Testing to experiment with different Remote Config values and determine how each affects Analytics metrics.

What we've covered

• How to set up A/B Testing using instrumented Remote Config values
• How to use Google Analytics User Properties as part of conditions for admission into A/B Testing experiments

Next Steps

Once the experiment is over, you can select one from the list of experiments in your project to decide what you want to do with it. Do you want to choose one as the "winner" or conduct more experiments?