Get started with instrumentation tests

An instrumentation test is a test written by you or your team to specifically to test your app, using the Espresso and UI Automator 2.0 Android test frameworks. Test Lab provides results for all test cases that complete execution during that time.

When you write an instrumentation test, you create a second APK module that you later upload to Test Lab along with the APK module for your app. To learn about creating test APKs, see Test your app.

Instrumentation test timeout

Instrumentation tests are allowed to run up to 45 minutes on physical devices and up to 60 minutes on virtual devices.

Run tests independently with Orchestrator

Android Test Orchestrator allows you to run each of your app's instrumentation tests independently. Test Lab always uses the latest version of Orchestrator. Using Orchestrator comes with a few benefits and one drawback:

Pros:

No shared state: Each test runs in its own instrumentation instance so a shared state doesn't accumulate across tests.
Isolated crashes: If a test crashes, only that instrumentation is terminated and other tests in your suite can still run.

Con:

Longer runtime: Each test runs its own instrumentation instance, meaning that the testing process takes slightly longer overall. The increased run time could impact your quota usage or billed time and might cause you to hit your devices' time-out limits.

Enable Orchestrator

To enable Orchestrator for Test Lab, in instrumentation test setup, click Additional options > Run with Orchestrator.

Speed up tests with sharding

Test sharding divides up a set of tests into sub-groups (shards) that run separately in isolation. Test Lab automatically runs each shard in parallel using multiple devices and completes the entire set of tests in less time.

Say you create N shards. For each device you select, Test Lab spins up N identical devices and runs a subset of the tests on each device. This means that sharded test cases can result in multiple test executions per device, unlike non-sharded test cases, which always result in one test execution per device (for a quick overview of key concepts in Test Lab, see Key concepts).

Billing for test shards

Test Lab implements your shards by leveraging AndroidJUnitRunner's built-in sharding mechanism. To avoid being charged for spinning up empty shards (shards without assigned test cases), the number of shards you create should be less than the total number of test cases. Depending on how long each test case takes to run, it's typically a good idea to assign 2-10 test cases per shard.

For more information on billing, read Usage, quotas, and billing.

Enable test sharding

You can enable test sharding by using the Firebase Console. To enable test sharding:

In instrumentation test setup, click Additional options.
In the Sharding section, enter the number of shards you want to run.

Taking screenshots

The following instructions describe how to add Test Lab's screenshot library to your app and call that library from your test. Test Lab provides a library you can use to take screenshots when running instrumentation tests, such as tests written using the Espresso test framework. After your test has run, you can review the screenshots in Android Studio or in the Firebase console.

To enable your test for screenshotting, add the following library: cloudtestingscreenshotter_lib.aar. Note that the ability to take screenshots is already incorporated into the test APK app-debug-test-unaligned.apk and the NotePad sample app. Screenshots are also automatically captured when you run a Robo test.

1. Add the screenshot library to your project

In Android Studio, open the Project view and right-click your project name. Then, click New > Directory.
In the New Directory dialog, type aars. This creates an aars directory at the root of your test project (as a peer directory to the app folder).
Copy cloudtestingscreenshotter_lib.aar and paste it into the aars folder.
In your app's root-level (project-level) build.gradle file, add a reference to the aars folder in every repositories block:
```
repositories {
    jcenter()
    flatDir {
        dirs '../aars'
    }
}
...
```
In your module's top-level directory (for the NotePad example app, this is the app directory), open the build.gradle file and add a dependency to cloudtestingscreenshotter_lib.aar to the top-level dependencies block:
```
dependencies {
    // Cloud testing
    androidTestCompile (name:'cloudtestingscreenshotter_lib', ext:'aar')
    // Other dependencies go here
    }
```
In your AndroidManifest.xml file, specify system permissions for your app by adding the following lines within the <manifest> tag. If you're testing on Android 10 (API level 29) or higher, omit the WRITE_EXTERNAL_STORAGE permission (your app does not require this permission in order to read and write screenshots to the device).
```
<manifest ... >
   
   <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"/>
   <uses-permission android:name="android.permission.INTERNET"/>
   ...
</manifest>
```

2. Take a screenshot from your test

At any point in your test where you want to take a screenshot, call the ScreenShotter.takeScreenshot method from the cloudtestingscreenshotter_lib library, where the first argument is a label that you use to identify the screenshot (main_screen_2 is used in the following example):

Java

ScreenShotter.takeScreenshot("main_screen_2", this /* activity */);MainActivity.java

Kotlin+KTX

ScreenShotter.takeScreenshot("main_screen_2", this /* activity */)MainActivity.kt

3. View your screenshots

After your test has completed, you can compare any screenshots taken during testing by selecting an element in the test results tree and then clicking the View Screenshots option.

Screenshot comparison screen

You can select and compare screenshots from different configurations as follows:

Task	Action
Switch between test executions.	Use the drop-down menu in the top-left corner.
Switch between screenshots within a test execution.	Use the arrows in the top-right corner.
Add additional screenshot comparison panels to the current view.	Click Compare.
Choose a different test dimension (device type, orientation, locale, etc.).	Select a new dimension member from the list at the bottom of the screenshot.