Firebase Test Lab for Android runs tests against a test matrix that you create to validate your app across a range of device types and device configurations. To understand how this works, we first need to explain test dimensions and test executions.
Test Dimension. A test dimension is a set of related device attributes. The dimensions used by Test Lab are model, Android API level, locale, and screen orientation.
Test Execution. Each valid combination of test dimensions that you select for your test matrix is a test execution. Test Lab supports testing your app on up to 200 unique combinations at a time. You can select the specific model and Android API level combinations to use when testing your app to focus your testing; for example, when you are validating a bug fix. Each test execution can either pass or fail. If one test execution fails, the whole test matrix is marked as failed.
Test Matrix. The test matrix is the set of test executions that you select. When you run your app test on a test matrix, you get test results for each valid test execution created by combining test dimension attributes.
You can choose how to initiate testing of your app, as follows:
- Android Studio integration lets you test your app from your development environment.
- The gcloud command line interface enables you to run tests from the command line interactively, and is also well suited for scripting as part of your automated build and testing process.
- The Test Lab UI in the Firebase console lets you upload an app binary and initiate testing from anywhere.
You can also test your app with Test Lab when you upload and publish your app's APK files to the Play Store using either the alpha or beta channel. For more information, see Use pre-launch reports to identify issues and Robo test.
You can use Test Lab to validate the stability of your app's user experience and the compatibility of your app across a range of devices.
Two testing methods are available to help you test your app: You can run Instrumentation tests and the Test Lab Robo test.
Instrumentation tests are tests that you or your team have written specifically to test your app, using the Espresso and UI Automator 2.0 Android test frameworks. Instrumentation tests run for up to 30 minutes on physical devices, and for up to 60 minutes on virtual devices. Test results are provided for all test cases that complete execution during that time.
When you write instrumentation tests, you create a second APK module that you later upload to Test Lab along with the APK module for your app. To learn about creating test APKs, see Test your app.
Even if you have not written any instrumentation tests for your app, you can use Robo test to exercise your app by simulating user actions. To learn more, see Robo test.
Game Loop test
To support game app testing, Test Lab now includes beta support for using a "demo mode" where the game app runs while simulating the actions of a player. This mode can include multiple loops (or scenarios), which can be logically organized using labels so that you can run related loops together. To learn more, see Game Loop test.
Test Lab supports testing on several makes and models of Android devices installed and running in a Google data center. Tests on devices help you to detect issues that might not occur when testing your app using the Android emulator. To learn more, see Available devices.
Review your test results
Regardless of how you initiate your tests, all your test results are managed by Test Lab and can be viewed online. Test results include test logs and the details of any app failure. For Robo test, videos and screenshots are also available. See Analyze Firebase Test Lab for Android Results for details.
When you initiate a test from Android Studio, you can also review test results from inside your development environment.
Quota and billing
Firebase is available to use with all three pricing plans: Spark (free), Flame, and Blaze.
Quota for Spark and Flame plans
On the Spark and Flame plans, you can use Test Lab at no charge. On both the Spark and Flame plans, you can test with a limited daily quota for test executions (10 virtual devices and 5 physical devices), and with up to 4 devices in each test matrix.
Billing for the Blaze plan
To use Test Lab without limits on daily usage or test matrix size, upgrade to the Firebase Blaze plan after selecting your project. To upgrade your billing plan, click UPGRADE in the Firebase console, and then click SELECT PLAN in the Blaze plan column.
With the Blaze plan, you can use Test Lab to test your app at the following hourly rates:
- $5 per hour for each physical device
- $1 per hour for each virtual device
Charges are calculated on a per-minute basis, and you are only charged for the time required to run your tests (not for time required to install your app or collect test results). After you run a test matrix, you can see the time required to run each test on each device on the test execution results and test matrix results pages. The billed time for each test execution is rounded up to the nearest minute.
Test Lab and Google Play services
Test Lab devices usually run the most recent version of Google Play services. Because of the number of devices and configurations, however, it sometimes takes a few days for Test Lab to update the devices after a new Google Play services release. To ensure your tests run properly, check that your app uses a version of the Google Play services SDK at or below the one listed below.
Current supported Google Play services SDK: 11.8.0
Test Lab and mobile advertising
For app developers that use or work with digital advertising providers (for example, ad networks, demand-side platforms) and for digital advertising providers:
Test Lab provides developers with a scalable app testing infrastructure that automates app testing. Unfortunately, this capability could be misused by malicious apps designed to generate fraudulent ad revenue.
To mitigate this issue:
App developers should notify any digital advertising providers they work with to filter out revenues and all corresponding traffic generated from devices that belong to testing providers, including Test Lab.
Digital advertising providers can filter ad revenues and all corresponding traffic generated from Test Lab by filtering traffic originating from the following IP address blocks:
Google takes the security of your app data very seriously. For the physical devices used by Test Lab, we follow industry-standard best practices to remove app data from those devices after every test run to ensure that they are ready to run new tests. For devices that we can flash with a custom recovery image, we go one step further by flashing these devices between test runs.
For the virtual devices used by Test Lab, device instances are deleted after they are used so that each test run uses a new virtual device instance.