An instrumentation test is a test written by you or your team to specifically to test your app, using the Espresso and UI Automator 2.0 Android test frameworks. Test Lab provides results for all test cases that complete execution during that time.
When you write an instrumentation test, you create a second APK module that you later upload to Test Lab along with the APK module for your app. To learn about creating test APKs, see Test your app.
Instrumentation test timeout
Instrumentation tests are allowed to run up to 45 minutes on physical devices and up to 60 minutes on virtual devices.
Run tests independently with Orchestrator
Android Test Orchestrator allows you to run each of your app's instrumentation tests independently. Test Lab always uses the latest version of Orchestrator. Using Orchestrator comes with a few benefits and one drawback:
Pros:
- No shared state: Each test runs in its own instrumentation instance so a shared state doesn't accumulate across tests.
- Isolated crashes: If a test crashes, only that instrumentation is terminated and other tests in your suite can still run.
Con:
- Longer runtime: Each test runs its own instrumentation instance, meaning that the testing process takes slightly longer overall. The increased run time could impact your quota usage or billed time and might cause you to hit your devices' time-out limits.
Enable Orchestrator
To enable Orchestrator for Test Lab, in instrumentation test setup, click Additional options > Run with Orchestrator.
Speed up tests with sharding
Test sharding divides up a set of tests into sub-groups (shards) that run separately in isolation. Test Lab automatically runs each shard in parallel using multiple devices and completes the entire set of tests in less time.
Say you create N shards. For each device you select, Test Lab spins up N identical devices and runs a subset of the tests on each device. This means that sharded test cases can result in multiple test executions per device, unlike non-sharded test cases, which always result in one test execution per device (for a quick overview of key concepts in Test Lab, see Key concepts).
Billing for test shards
Test Lab implements your shards by leveraging AndroidJUnitRunner's built-in sharding mechanism. To avoid being charged for spinning up empty shards (shards without assigned test cases), the number of shards you create should be less than the total number of test cases. Depending on how long each test case takes to run, it's typically a good idea to assign 2-10 test cases per shard.
For more information on billing, read Billing for the Blaze Plan.
Enable test sharding
You can enable test sharding by using the Firebase Console. To enable test sharding:
In instrumentation test setup, click Additional options.
In the Sharding section, enter the number of shards you want to run.