Firebase Test Lab for Android runs tests against a test matrix that you create to validate your app across a range of device types and device configurations. To understand how this works, we first need to explain test dimensions and test executions.
- Test Dimension. A test dimension is a set of related device attributes. The dimensions used by Test Lab are device type, Android API level, locale, and screen orientation. Before testing your app, you select attributes from each dimension, and Test Lab creates a list of all valid combinations of these attributes. Test Lab supports testing your app on up to 200 unique combinations at a time.
Test Execution. Each valid combination of test dimensions that you select for your test matrix is a test execution. For example, if you select five device types (Asus Google Nexus 7, LG G3, Samsung Galaxy Note 2, etc.), two Android API levels (21, 22), two locales (en_US, es_ES), and two screen orientations (landscape, portrait), the test contains 5*2*2*2 = 40 test executions. Each test execution can either pass or fail. If one test execution fails, the whole test matrix is marked as failed.
Test Matrix. The test matrix is the set of test executions defined by the combinations of the test dimension attributes that you select. When you run your app test on a test matrix, you get test results for each valid test execution created by combining test dimension attributes.
You can choose how to initiate testing of your app:
- Android Studio integration lets you test your app from your development environment.
- The gcloud command line interface enables you run tests from the command line interactively, and is also well suited for scripting as part of your automated build and testing process.
- The Test Lab UI in the Firebase console lets you upload an app binary and initiate testing from anywhere.
You can also test your app with Test Lab when you upload and publish your app's APK files to the Play Store using either the alpha or beta channel. For more information, see Use pre-launch reports to identify issues and Robo test.
Test Lab supports testing on several makes and models of Android devices installed and running in a Google data center. Tests on devices help you to detect issues that might not occur when testing your app using the Android emulator.
Currently, the following types of physical devices are available:
Table 1: Devices used by Test Lab
|HTC||Nexus 9, HTC One (M8)|
|LG||LG G4, G3, Nexus 5, Nexus 4|
|Motorola||Moto E, Moto G (1st, 2nd and 3rd Generation), Moto G4, Moto G4 Plus, Moto X, Nexus 6|
|Samsung||Galaxy J5, Galaxy Note 2, Galaxy Note 3 Duos, Galaxy Note 4, Galaxy S3, Galaxy S4 (3G), Galaxy S4 Mini, Galaxy S6, Galaxy S6 Edge, Galaxy S7, Galaxy Tab 3, Nexus 10|
|Sony||Xperia Z2, Xperia Z3, Xperia Z5 Compact|
You can use Test Lab to validate the stability of your app's user experience and the compatibility of your app across a range of devices.
Two testing methods are available to help you to test your app: You can run Instrumentation tests and the Test Lab Robo test.
Instrumentation tests are tests that you or your team have written specifically to test your app, using the Espresso, Robotium and UI Automator 2.0 Android test frameworks. Instrumentation tests run for up to 30 minutes on physical devices, and for up to 60 minutes on virtual devices. Test results are provided for all test cases that complete execution during that time.
When you write instrumentation tests, you create a second .APK module that you will later upload to Test Lab along with the .APK module for your app. To learn about creating test APKs, see Test APK Module.
Even if you have not written any instrumentation tests for your app, you can use Robo test to exercise your app by simulating user actions. To learn more, see Robo test.
Review your test results
Regardless of how you initiate your tests, all your test results are managed by Test Lab and can be viewed online. Test results include test logs and the details of any app failure. For Robo test, videos and screenshots are also available. See Analyzing Test Results for details.
When you initiate a test from Android Studio, you can also review test results from inside your development environment.
Quota and billing
Firebase is available to use with all three pricing plans: Spark (free), Flame, and Blaze.
Quota for Spark and Flame plans
On the Spark and Flame plans, you can use Test Lab at no charge. On both the Spark and Flame plans, you can test with a limited daily quota for test executions (10 virtual devices and 5 physical devices), and with up to four devices in each test matrix.
Billing for the Blaze plan
To use Test Lab without limits on daily usage or test matrix size, upgrade to the Firebase Blaze plan. To upgrade your billing plan, click UPGRADE in the Firebase console, and then click SELECT PLAN in the Blaze plan column.
With the Blaze plan, you can use Test Lab to test your app at the following hourly rates:
- $5 per hour for each physical device
- $1 per hour for each virtual device
Charges are calculated on a per-minute basis, and you are only charged for the time required to run your tests (not for time required to install your app or collect test results).
Test Lab and mobile advertising
For app developers that use or work with digital advertising providers (e.g., ad networks, demand-side platforms), and for digital advertising providers:
Test Lab provides developers with a scalable app testing infrastructure that automates app testing. Unfortunately, this capability could be misused by malicious apps designed to generate fraudulent ad revenue.
To mitigate this issue:
- App developers should notify any digital advertising providers they work with to filter out revenues and all corresponding traffic generated from devices that belong to testing providers, including Test Lab.
- Digital advertising providers can filter ad revenues and all corresponding traffic generated from Test Lab by filtering traffic originating from the following IP address blocks: 126.96.36.199/24 (for physical devices) and 188.8.131.52/16 (for virtual devices).
Device clean up
Google takes the security of your app data very seriously. For the physical devices used by Test Lab, we follow industry-standard best practices to remove app data from those devices after every test run, to ensure that they are ready to run new tests. For devices that we can flash with a custom recovery image, we go one step further by flashing these devices between test runs.
For the virtual devices used by Test Lab, device instances are deleted after they are used, so that each test run uses a new virtual device instance.