Using Firebase Test Lab for Android from the gcloud Command Line

Firebase Test Lab for Android provides cloud-based infrastructure for testing Android apps, including full integration with the gcloud command-line interface (CLI). This document covers the installation and configuration required to get started using Test Lab from the gcloud command line interface.

For a complete list of gcloud firebase test android commands that you can use with Test Lab, see gcloud firebase test android.

Create a Firebase project

If you don't have a Firebase project for your app, go to the Firebase console and click Create New Project to create one now. You will need ownership or edit permissions in your project.

You can use Test Lab for a limited number of daily test executions on the Spark and Flame plans. To use Test Lab without daily quota limits, you must upgrade to the Firebase Blaze plan.

To learn more, see Test Lab quota and billing.

Configure your local Google Cloud SDK environment

1. If it's not present, install the Google Cloud SDK on your local system.

2. Make sure your Cloud SDK installation is up-to-date and includes the gcloud firebase command:

   gcloud components update
   

3. Set your current gcloud project to your project ID, as follows:

   gcloud config set project <PROJECT-ID>
   

4. Make sure your authentication credentials are current, as follows:

   gcloud auth login
   

Choosing test configurations

In this example, you will run some tests on a simple note-taking Android app named Notepad.

1. Download the binary APK file for the Notepad app (app-debug-unaligned.apk) and its corresponding instrumentation tests (app-debug-test-unaligned.apk) provided in the NotePad/app/build/outputs/apk/ directory of notepad.zip.

2. Get the current list of Android devices available to test against, as follows:

   
   $ gcloud firebase test android models list
   

   The first column of the command output, MODEL_ID, contains the identifier that you can use later to run tests on a specific model. The OS_VERSION_ID column lists the operating system versions supported by that device. If you don't specify which MODEL_ID(s) to test against, the default noted under the TAGS column is used.

3. Learn more about a specific Android MODEL_ID with the firebase test android models describe command, as follows:

   
   $ gcloud firebase test android models describe Nexus5
   

   The example command shown above provides detailed information about the Nexus5 model, including the brand, manufacturer, and supported API levels, and whether the model is physical or virtual.

4. Get the current list of Android OS versions available to test against:

   
   $ gcloud firebase test android versions list
   

   You can use an identifier from either of the first two columns of command output (OS_VERSION_ID and VERSION), to later run tests against an Android OS version. If you don't specify the Android OS versions to test against, the default noted under the TAGS column is used.

5. Get the current list of locales available to test against:

   
   $ gcloud firebase test android locales list
   

   The first column of the command output, LOCALE, contains the identifier that you can use later to run tests against a locale. If you don't specify the locales to test against, English is used as the default locale. Command output is not shown here because hundreds of locales are available.

Running tests

Now that you know the range of device models, OS versions, and locales available to use when testing your app, you can use that information to specify test devices using the gcloud firebase test android run command and the --device flag. This command and flag are used whether you are using the Robo test to automatically test your app, or whether you are running instrumentation tests written specifically to test your app.

Running the Robo test

Even if you don't have any instrumentation tests, you can still look for bugs in your app. Use the Robo test to perform automated review of your app's user interface. Robo test exercises the app by performing a static analysis of the various paths through the app's user interface, and then crawling through the app to find crashes and other potential issues.

Let's start by running an example command:

gcloud firebase test android run \
  --type robo \
  --app app-debug-unaligned.apk \
  --device model=Nexus6,version=21,locale=en,orientation=portrait  \
  --device model=Nexus7,version=19,locale=fr,orientation=landscape \
  --timeout 90s

The --type robo parameter is implicit if no --type value is specified. You can see the complete set of command line options for running tests by typing: gcloud help firebase test android run. As an alternative to specifying all these arguments on the command line, you can optionally specify your arguments in a YAML-formatted argument file. Run gcloud topic arg-files to learn how to use this feature.

See the Analyze your test results section to learn how to investigate the test results from the Robo test.

Running your instrumentation tests

Now use the gcloud command line tool to run the Notepad app's Espresso tests on your specified Android device configurations, using the instrumentation test type to run the tests in app-debug-test-unaligned.apk as follows:

    gcloud firebase test android run \
      --type instrumentation \
      --app app-debug-unaligned.apk \
      --test app-debug-test-unaligned.apk \
      --device model=Nexus6,version=21,locale=en,orientation=portrait  \
      --device model=Nexus7,version=19,locale=fr,orientation=landscape

The --type instrumentation parameter is implicit if a test APK has been specified with --test. As an alternative to specifying all these arguments on the command line, you can optionally specify your arguments in a YAML-formatted argument file. Run gcloud topic arg-files to learn how to use this feature.

Note: You can also control how Test Lab runs your instrumentation tests using additional flags that are not shown above. For example, you can use the --test-targets flag to test just one class used by your test APK, or to test just one method from a class used by your test APK. To learn more, see gcloud firebase test android run.

Analyze your test results

After a few minutes, a basic summary of your test results is printed by the gcloud tool:

The output of your command line test run also includes a link to view test results. To learn more about how to interpret these results, see Analyzing Firebase Test Lab for Android Results.

Custom login and text input with Robo test

Robo test automatically completes sign-in screens that use a Google account for authentication, unless you use the --no-auto-google-login parameter. It can also complete custom login screens using test account credentials that you provide. You can also use this parameter to provide custom input text for other text fields used by your app.

To complete text fields in your app, use the --robo-directives parameter and provide a comma-separated list of key-value pairs, where the key is the Android resource name of the target UI element, and the value is the text string. EditText fields are supported but not text fields in WebView UI elements.

For example, you could use the following parameter for custom login:

--robo-directives username_resource=username,password_resource=password

Beta commands and flags

The Test Lab gcloud CLI currently has several commands and flags available in public beta, including the following:

• Game loop test support, which lets you test game apps. To learn more, see Firebase Test Lab for Android Game Loop Test.

• Network traffic profiles, which lets you set a variety of parameters to emulate various network conditions on physical devices. This capability can be especially useful when developing apps that must function well under unreliable or unpredictable network conditions. To learn more, see gcloud beta firebase test network-profiles.

Scripting gcloud commands with Test Lab

You can use shell scripts or batch files to automate mobile app testing commands that you would otherwise run using the gcloud command line. The following example bash script runs an instrumentation test with a two-minute timeout, and reports if the test run completed successfully:

if gcloud firebase test android run --app my-app.apk --test my-test.apk --timeout 2m
then
    echo "Test matrix successfully finished"
else
    echo "Test matrix exited abnormally with non-zero exit code: " $?
fi

Script exit codes

Test Lab provides several exit codes that you can use to better understand the results of tests that you run using scripts or batch files.

Scripting exit codes for Test Lab