Firebase Test Lab lets you test your app on a range of devices and configurations, to give you a better idea of how it'll perform in your users' hands. To facilitate such broad testing, Test Lab runs your tests in batches called test matrices. In each matrix, your app is tested against multiple combinations of device configurations.
Test Dimensions × Test Executions = Test Matrix
Test Dimensions: device attributes to test your app against, including device model, OS version, locale, and screen orientation.
Test Executions: individual test runs, each with a unique combination of test dimensions that you've selected to test.
Test Matrix: the results of your test executions on the selected test dimensions. If any test execution in the matrix fails, Test Lab marks the whole matrix as failed.
You can choose how to initiate testing of your app, as follows:
- Android Studio integration lets you test your app from your development environment.
- The gcloud command line interface enables you to run tests from the command line interactively, and is also well suited for scripting as part of your automated build and testing process.
- The Test Lab UI in the Firebase console lets you upload an app binary and initiate testing from anywhere.
You can also test your app with Test Lab when you upload and publish your app's APK files to the Play Store using either the alpha or beta channel. For more information, see Use pre-launch reports to identify issues and Robo test.
You can use Test Lab to validate the stability of your app's user experience and the compatibility of your app across a range of devices.
Two testing methods are available to help you test your app: You can run Instrumentation tests and the Test Lab Robo test.
Instrumentation tests are tests that you or your team have written specifically to test your app, using the Espresso and UI Automator 2.0 Android test frameworks. Instrumentation tests run for up to 30 minutes on physical devices, and for up to 60 minutes on virtual devices. Test results are provided for all test cases that complete execution during that time.
When you write instrumentation tests, you create a second APK module that you later upload to Test Lab along with the APK module for your app. To learn about creating test APKs, see Test your app.
Even if you have not written any instrumentation tests for your app, you can use Robo test to exercise your app by simulating user actions. To learn more, see Robo test.
Game Loop test
To support game app testing, Test Lab now includes support for using a "demo mode" where the game app runs while simulating the actions of a player. This mode can include multiple loops (or scenarios), which can be logically organized using labels so that you can run related loops together. To learn more, see Game Loop test.
Test Lab supports testing on several makes and models of Android devices installed and running in a Google data center. Tests on devices help you to detect issues that might not occur when testing your app using the Android emulator. To learn more, see Available devices.
Review your test results
Regardless of how you initiate your tests, all your test results are managed by Test Lab and can be viewed online. Test results include test logs and the details of any app failure. For Robo test, videos and screenshots are also available. See Analyze Firebase Test Lab Results for details.
When you initiate a test from Android Studio, you can also review test results from inside your development environment.
Quota and billing
Test Lab is available to use with all three pricing plans: Spark (free), Flame, and Blaze.
Quota for Spark and Flame plans
On both Spark and Flame plans, you can test with a limited daily quota for test executions: 10 virtual devices and 5 physical devices (for a maximum of 15 test runs per day). This limit is shared among the test types (Instrumentation, Robo, Game Loop) and matrices.
Billing for the Blaze plan
To use Test Lab without limits on daily usage or test matrix size, upgrade to the Firebase Blaze plan after selecting your project. To upgrade your billing plan, click UPGRADE in the Firebase console, and then click SELECT PLAN in the Blaze plan column.
Free usage equivalent to the Spark plan is included in the Blaze plan:
- 15 hours of tests on physical devices
- 30 hours of tests on virtual devices
All usage above those limits has the following hourly rates:
- $5 per hour for each physical device
- $1 per hour for each virtual device
Charges are calculated on a per-minute basis, and you are only charged for the time required to run your tests (not for time required to install your app or collect test results). After you run a test matrix, you can see the time required to run each test on each device on the test execution results and test matrix results pages. The billed time for each test execution is rounded up to the nearest minute.
Test Lab and Google Play services
Test Lab devices usually run the most recent version of Google Play services. Because of the number of devices and configurations, however, it sometimes takes a few days for Test Lab to update the devices after a new Google Play services release. To ensure your tests run properly, check that your app uses a version of the Google Play services SDK at or below the one listed below.
Current supported Google Play services SDK: 11.8.0
Test Lab and mobile advertising
For app developers that use or work with digital advertising providers (for example, ad networks, demand-side platforms) and for digital advertising providers:
Test Lab provides developers with a scalable app testing infrastructure that automates app testing. Unfortunately, this capability could be misused by malicious apps designed to generate fraudulent ad revenue.
To mitigate this issue:
App developers should notify any digital advertising providers they work with to filter out revenues and all corresponding traffic generated from devices that belong to testing providers, including Test Lab.
Digital advertising providers can filter ad revenues and all corresponding traffic generated from Test Lab by filtering traffic originating from the following IP address blocks:
- 220.127.116.11/27 (added 7-2019)
Google takes the security of your app data very seriously. For the physical devices used by Test Lab, we follow industry-standard best practices to remove app data from those devices after every test run to ensure that they are ready to run new tests. For devices that we can flash with a custom recovery image, we go one step further by flashing these devices between test runs.
For the virtual devices used by Test Lab, device instances are deleted after they are used so that each test run uses a new virtual device instance.