Check out what’s new from Firebase@ Google I/O 2021, and join our alpha program for early access to the new Remote Config personalization feature. Learn more

Get started with instrumentation tests

This guide describes how to prepare and run an instrumentation test using Firebase Test Lab. To use this guide, you'll need an instrumentation test (written by you or your team) that uses the Espresso or UI Automator 2.0 Android test frameworks. Instrumentation tests can run up to 45 minutes on physical devices and up to 60 minutes on virtual devices.

In the steps later, you'll upload your app's APK and your test's APK to Firebase. See Test your app to learn more about creating a test APK. As an option, you can also download the NotePad sample app.

Step 1. (Optional) Add the screenshot library to your app

Test Lab provides a screenshot library that lets you take screenshots when running instrumentation tests on your app. When your test finishes, you can view the screenshots in Android Studio or in the Firebase console.

Note that the ability to take screenshots is already incorporated into the test APK app-debug-test-unaligned.apk and the NotePad sample app. Screenshots are also automatically captured when you run a Robo test.

  1. In Android Studio, open the Project view and right-click your project name. Then, click New > Directory.

  2. In the New Directory dialog, type aars. This creates an aars directory at the root of your test project (as a peer directory to the app folder).

  3. Copy cloudtestingscreenshotter_lib.aar and paste it into the aars folder.

  4. In your app's root-level (project-level) build.gradle file, add a reference to the aars folder in every repositories block:

    repositories {
        jcenter()
        flatDir {
            dirs '../aars'
        }
    }
    ...
  5. In your module's top-level directory (for the NotePad example app, this is the app directory), open the build.gradle file and add a dependency to cloudtestingscreenshotter_lib.aar to the top-level dependencies block:

    dependencies {
        // Cloud testing
        androidTestCompile (name:'cloudtestingscreenshotter_lib', ext:'aar')
        // Other dependencies go here
        }
    
  6. In your AndroidManifest.xml file, specify system permissions for your app by adding the following lines within the <manifest> tag. If you're testing on Android 10 (API level 29) or higher, omit the WRITE_EXTERNAL_STORAGE permission (your app does not require this permission in order to read and write screenshots to the device).

    <manifest ... >
       <!-- WRITE_EXTERNAL_STORAGE is not needed on Android 10 (API level 29) or higher. -->
       <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"/>
       <uses-permission android:name="android.permission.INTERNET"/>
       ...
    </manifest>
    

At any point in your test where you want to take a screenshot, call the ScreenShotter.takeScreenshot method from the cloudtestingscreenshotter_lib library, where the first argument is a label that you use to identify the screenshot (main_screen_2 is used in the following example):

Java

ScreenShotter.takeScreenshot("main_screen_2", this /* activity */);

Kotlin+KTX

ScreenShotter.takeScreenshot("main_screen_2", this /* activity */)

View your screenshots

If you're using Android Studio to run your test, when your test finishes running, you can compare any screenshots taken during the test by selecting an element in the test results tree and then clicking the View Screenshots View Screenshots option.

Screenshot comparison screen

You can select and compare screenshots from different configurations as follows:

Task Action
Switch between test executions. Use the drop-down menu in the top-left corner.

test case menu
Switch between screenshots within a test execution. Use the arrows in the top-right corner.

screenshot switcher
Add additional screenshot comparison panels to the current view. Click Compare.

Compare
Choose a different test dimension (device type, orientation, locale, etc.). Select a new dimension member from the list at the bottom of the screenshot.

Step 2. Enable optional test features

You can enable the following features in your test before running it with Test Lab:

Enable Orchestrator

Android Test Orchestrator is a tool that runs each of your app's instrumentation tests independently. Test Lab always uses the latest version of Orchestrator.

To enable Orchestrator for Test Lab, in instrumentation test setup, click Additional options > Run with Orchestrator.

Benefits and drawbacks

  • Benefit: No shared state. Each test runs in its own instrumentation instance so a shared state doesn't accumulate across tests.
  • Benefit: Isolated crashes. If a test crashes, only that instrumentation is terminated and other tests in your suite can still run.
  • Drawback: Longer runtimes. Each test runs its own instrumentation instance, meaning that the testing process takes slightly longer overall. If not checked, the increased run times could potentially impact your quota usage or billed time and may cause you to hit your devices' time-out limits.

Enable sharding

Test sharding divides up a set of tests into sub-groups (shards) that run separately in isolation. Test Lab automatically runs each shard in parallel using multiple devices and completes the entire set of tests in less time.

How test sharding works

Say you create N shards. For each device you select, Test Lab spins up N identical devices and runs a subset of the tests on each device. This means that sharded test cases can result in multiple test executions per device, unlike non-sharded test cases, which always result in one test execution per device (for a quick overview of key concepts in Test Lab, see Key concepts).

You can enable test sharding in the Firebase console:

  1. In instrumentation test setup, click Additional options.

  2. In the Sharding section, enter the number of shards you want to run.

Billing for test shards

Test Lab implements your shards by leveraging AndroidJUnitRunner's built-in sharding mechanism. To avoid being charged for spinning up empty shards (shards without assigned test cases), the number of shards you create should be less than the total number of test cases. Depending on how long each test case takes to run, it's typically a good idea to assign 2-10 test cases per shard.

For more information on billing, read Usage, quotas, and billing.