Create Messaging Experiments with A/B Testing

When you are reaching out to your users or starting a new marketing campaign, you want to make sure that you get it right. A/B testing can help you find the optimal wording and presentation by testing message variants on selected portions of your user base. Whether your goal is better retention or conversion on an offer, A/B testing can perform statistical analysis to determine if a message variant is outperforming the control group for your selected objective.

To A/B test feature variants with a control group, do the following:

1. Create your experiment.
2. Validate your experiment on a test device.
3. Manage your experiment.

Create an experiment

An experiment that uses the Notifications composer lets you evaluate multiple variants on a single notification message.

1. Sign in to Firebase, update Data sharing settings and make sure that data sharing is enabled. Without data sharing, your experiment won't have access to analytics data.
2. On the Firebase console navigation bar, expand Grow, then click A/B Testing.
3. Click Create experiment, and then select Notifications when prompted for the service you want to experiment with.
4. Enter a Name and optional Description for your experiment, and click Next.

5. Fill out the Targeting fields, first choosing the app that uses your experiment. You can also target a subset of your users to participate in your experiment by choosing one or more of the following options:

   • Version: One or more versions of your app
   • User audience: Analytics audiences used to target users who might be included in the experiment
   • User property: One or more Analytics user properties for selecting users who might be included in the experiment
   • Prediction: Groups of users predicted by machine learning to engage in particular behavior
   • Country/Region: One or more countries or regions for selecting users who might be included in the experiment
   • Device language: One or more languages and locales used to select users who might be included in the experiment

6. Set the Percentage of target users: Enter the percentage of your app's user base matching the criteria set under Target users that you want to evenly divide between the control group and one or more variants in your experiment. This can be any percentage between 0.01% and 100%. Percentages are randomly reassigned to users for each experiment, including duplicated experiments.

7. In the Variants section, type a message to send to the control group in the Enter message text field. To send no message to the control group, leave this field blank.

8. (optional) To add more than one variant to your experiment, click Add Variant. By default, experiments have one control group and one variant.

9. (optional) Enter a name for each variant in your experiment to replace the names Variant A, Variant B, etc.

10. Define a goal metric for your experiment to use when evaluating experiment variants along with any desired additional metrics from the dropdown list. These metrics include built-in objectives (engagement, purchases, revenue, retention, etc.), Analytics conversion events, and other Analytics events.

11. Choose options for your message:

    • Delivery date: Either choose Send Now to launch your experiment immediately on saving, or Scheduled to specify a time to launch your experiment in the future.
    • Advanced options: To choose advanced options for all notifications included in your experiment, expand Advanced options, and then change any of the listed message options.

12. Click Review to save your experiment.

Validate your experiment on a test device

Each Firebase app installation has an instance ID token (or registration token) associated with it. You can use this token to test specific experiment variants on a test device with your app installed. To validate your experiment on a test device, do the following:

1. Get the instance ID token as follows:

   Swift

   InstanceID.instanceID().instanceID { (result, error) in
     if let error = error {
       print("Error fetching remote instance ID: \(error)")
     } else if let result = result {
       print("Remote instance ID token: \(result.token)")
       self.instanceIDTokenMessage.text  = "Remote InstanceID token: \(result.token)"
     }
   }
   

   Objective-C

   [[FIRInstanceID instanceID] instanceIDWithHandler:^(FIRInstanceIDResult * _Nullable result,
                                                       NSError * _Nullable error) {
     if (error != nil) {
       NSLog(@"Error fetching remote instance ID: %@", error);
     } else {
       NSLog(@"Remote instance ID token: %@", result.token);
       NSString* message =
         [NSString stringWithFormat:@"Remote InstanceID token: %@", result.token];
       self.instanceIDTokenMessage.text = message;
     }
   }];
   

   Java

   FirebaseInstanceId.getInstance().getInstanceId()
           .addOnCompleteListener(new OnCompleteListener<InstanceIdResult>() {
               @Override
               public void onComplete(@NonNull Task<InstanceIdResult> task) {
                   Log.d("IID_TOKEN", task.getResult().getToken());
               }
           });
   MainActivity.java

   Kotlin

   FirebaseInstanceId.getInstance().instanceId
           .addOnSuccessListener { result ->
               Log.d("IID_TOKEN", result.token)
           }
   MainActivity.kt

   C++

   firebase::InitResult init_result;
   auto* instance_id_object = firebase::instance_id::InstanceId::GetInstanceId(
       firebase::App::GetInstance(), &init_result);
   instance_id_object->GetToken().OnCompletion(
       [](const firebase::Future<std::string>& future) {
         if (future.status() == kFutureStatusComplete &&
             future.error() == firebase::instance_id::kErrorNone) {
           printf("Instance ID Token %s\n", future.result()->c_str());
         }
       });
   

   Unity

   Firebase.InstanceId.FirebaseInstanceId.DefaultInstance.GetTokenAsync().ContinueWith(
     task => {
       if (!(task.IsCanceled || task.IsFaulted) && task.IsCompleted) {
         UnityEngine.Debug.Log(System.String.Format("Instance ID Token {0}", task.Result));
       }
     });
   

2. On the Firebase console navigation bar, click Remote Config or Notifications, and then click A/B testing.
3. Click Draft, and then click the title of your experiment.
4. Click Manage test devices, and then enter the instance ID token for a test device and choose the experiment variant to send to that test device.
5. Run the app and confirm that the selected variant is being received on the test device.

To learn more about the instance ID token, see FIRInstanceID (iOS) or FirebaseInstanceId (Android).

Manage your experiment

Whether you create an experiment with Remote Config or the Notifications composer, you can then validate and start your experiment, monitor your experiment while it is running, and increase the number of users included in your running experiment.

When your experiment is done, you can take note of the settings used by the winning variant, and then roll out those settings to all users. Or, you can run another experiment.

Start an experiment

1. On the Firebase console navigation bar, expand Grow, then click A/B Testing.
2. Click Draft, and then click the title of your experiment.
3. To validate that your app has users who would be included in your experiment, check for a number greater than 0% in the Targeting and distribution section (for example, 1% of users matching the criteria).
4. To change your experiment, click Edit.
5. To start your experiment, click Start Experiment. You can run up to 24 experiments per project at a time.

Monitor an experiment

Once an experiment has been running for a while, you can check in on its progress and see what your results look like for the users who have participated in your experiment so far.

1. On the Firebase console navigation bar, click Grow, then click A/B Testing.

2. Click Running, and then click the title of your experiment. On this page, you can view various statistics about your running experiment, including your goal metric and other metrics. For each metric, the following information is available:

   • Improvement: A measure of the improvement of a metric for a given variant as compared to the baseline (or control group). Calculated by comparing the value range for the variant to the value range for the baseline.
   • Probability to beat baseline: The estimated probability that a given variant beats the baseline for the selected metric.
   • Probability to be the best variant: The estimated probability that a given variant beats other variants for the selected metric.
   • Value per user: Based on experiment results, this is the predicted range that the metric value will fall into over time.
   • Total value: The observed cumulative value for the control group or variant. The value is used to measure how well each experiment variant performs, and is used to calculate Improvement, Value range, Probability to beat baseline, and Probability to be the best variant. Depending on the metric being measured, this column may be labeled "Duration per user," "Retention rate," or "Conversion rate."

3. To increase the number of users included in your experiment, click Increase Distribution, and then select an increased percentage to add more eligible users to your experiment.

4. After your experiment has run for a while (at least 24 hours), data on this page indicates which variant, if any, is the "leader." Some measurements are accompanied by a bar chart that presents the data in a visual format.

Roll out an experiment to all users

After an experiment has run long enough that you have a "leader," or winning variant, for your goal metric, you can roll out the experiment to 100% of users. This allows you to select a value to publish in Remote Config for all users moving forward. Even if your experiment has not created a clear winner, you can still choose to roll out a variant to all of your users.

1. On the Firebase console navigation bar, click Grow, then click A/B Testing.
2. Click Completed or Running, click an experiment that you want to roll out to all users, click the context menu (more_vert), and then click Roll out variant.

3. Roll out your experiment to all users by doing one of the following:

   • For an experiment that uses the Notifications composer, use the Roll out message dialog to send the message to the remaining targeted users who were not part of the experiment.
   • For a Remote Config experiment, use the dialog to determine which Remote Config parameter values to change for all users.

Expand an experiment

If you find that an experiment isn't bringing in enough users for A/B Testing to declare a leader, you can increase distribution of your experiment to reach a larger percentage of the app's user base.

1. On the Firebase console navigation bar, click Grow, then click A/B Testing.
2. Click Running, hover over your experiment, click the context menu (more_vert), and then click Increase Distribution.
3. The console displays a dialog with an option to increase the percentage of users who are in the currently running experiment. Input a number greater than the current percentage and click Send. The experiment will be pushed out to the percentage of users you have specified.

Duplicate or stop an experiment

1. On the Firebase console navigation bar, click Grow, then click A/B Testing.
2. Click Completed or Running, hover over your experiment, click the context menu (more_vert), and then click Duplicate or Stop.

User targeting

You can target the users to include in your experiment using the following user-targeting criteria.

A/B Testing metrics

When you create your experiment, you choose a metric that is used to compare experiment variants, and you can also choose other metrics to track to help you to better understand each experiment variant and detect any significant side-effects (such as app crashes). The following tables provide details on how goal metrics and other metrics are calculated.

Goal metrics

Other metrics

BigQuery data export

You can access all analytics data related to your A/B tests in BigQuery. BigQuery allows you to analyze the data using BigQuery SQL, export it to another cloud provider, or use the data for your custom ML models. See Link BigQuery to Firebase for more information.

To get started, make sure that your Firebase project is linked to BigQuery. Select Settings > Project Settings from the left navigation bar, then select Integrations > BigQuery > Link. This page displays options to perform BiqQuery analytics data export for all apps in the project.

To query analytics data in the context of an experiment, you can open the experiment results page, and select View in BigQuery. This opens the BigQuery console's query composer with an example query of experiment data preloaded for your review. Note that, because Firebase data in BigQuery is updated only once daily, the data available in the experiment page may be more up to date than the data available in the BigQuery console.