Check out what’s new from Firebase@ Google I/O 2021, and join our alpha program for early access to the new Remote Config personalization feature. Learn more

Get started testing for Android with Firebase Test Lab

Firebase Test Lab lets you test your app on a range of devices and configurations. This Get Started guide provides an implementation path for you to follow, as well as an introduction to Test Lab's Android offerings.

For information about Test Lab quotas and pricing plans, see Usage, Quotas, and Pricing.

Key concepts

When you run a test or a set of test cases against devices and configurations you've selected, Test Lab runs the test against your app in a batch, then displays the results as a test matrix.

Devices × Test Executions = Test Matrix

Device
A physical or virtual device (Android only) you run a test on, such as a phone, tablet, or wearable device. Devices in a test matrix are identified by device model, OS version, screen orientation, and locale (also known as geography and language settings).
Test, test execution
A test (or a set of test cases) to be run on a device. You can run one test per device, or optionally shard the test and run its test cases on different devices.
Test matrix
Contains the statuses and test results for your test executions. If any test execution in a matrix fails, the whole matrix fails.

Step 1: Prepare your test for uploading to Test Lab

Available test types

You can run the following tests with Test Lab. Note that all test types are limited to running 45 minutes on physical devices and 60 minutes on virtual devices. Any uncaught exception will cause a test failure.

  • Instrumentation test or instrumented unit test: A test you've written using the Espresso or UI Automator 2.0 frameworks. With this test, you can make explicit assertions about the state of your app to verify correct functionality using AndroidJUnitRunnerAPIs.

  • Robo test: An automated test that analyzes your app's UI and then explores it methodically by simulating user activities, without requiring you to write any code. Visit About Robo tests for more information.

  • Game Loop test: A test that uses a "demo mode" to simulate player actions in gaming apps. This is a fast and scalable way to verify that your game performs well for users. When you choose to run a Game Loop test, you can:

    • Write tests native to your game engine

    • Avoid writing the same code for different UIs or testing frameworks

    • Optionally create multiple loops to run in a single test execution (visit About Game Loop tests to learn more). You can also organize loops by using labels so you can keep track of them and re-run specific loops.

    See Run a Game Loop test for instructions on running this test with Test Lab.

Tools to run your test

You can choose the following tools to run your test with:

You can also test your app for free with Test Lab when you upload and publish your app's APK files to the Play Store using either the alpha or beta channel. For more information, see Use pre-launch reports to identify issues and Robo tests.

Step 2: Choose your testing device

Test Lab supports testing on several makes and models of Android devices installed and running in a Google data center. Testing on devices in Test Lab help you detect issues that might not occur when testing your app using emulators in Android Studio. To learn more, see Available devices.

Step 3: Review test results

Regardless of how you initiate your tests, all your test results are managed by Test Lab and can be viewed online.

The test result summary is automatically stored and can be viewed in the Firebase console. It contains the most relevant data for your test, including test case-specific videos, screenshots, the number of tests that passed, failed, or got flaky results, and more.

The raw test results contain test logs and app failure details, and is automatically stored in a Google Cloud bucket. If you specify a bucket, you are responsible for the cost of the storage. If you don't specify a bucket, Test Lab creates one for you for free.

For more details, see Analyze Firebase Test Lab Results.

When you initiate a test from Android Studio, you can also review test results from inside your development environment.

Device cleanup

Google takes the security of your app data very seriously. We follow industry-standard best practices to remove app data and reset system settings for physical devices after every test run to ensure that they are ready to run new tests. For devices that we can flash with a custom recovery image, we go one step further by flashing these devices between test runs.

For the virtual devices used by Test Lab, device instances are deleted after they are used so that each test run uses a new virtual device instance.


Additional information

Test Lab and Google Play services

Test Lab devices usually run on the latest version of the Google Play services SDK, but some may require a few days to update after a new version of the SDK is released. Note that you may encounter compatibility issues with some devices.

Test Lab and mobile advertising

For app developers that use or work with digital advertising providers (for example, ad networks, demand-side platforms) and for digital advertising providers:

Test Lab provides developers with a scalable app testing infrastructure that automates app testing. Unfortunately, this capability could be misused by malicious apps designed to generate fraudulent ad revenue.

To mitigate this issue:

  • App developers should notify any digital advertising providers they work with to filter out revenues and all corresponding traffic generated from devices that belong to testing providers, including Test Lab.

  • Digital advertising providers can filter ad revenues and all corresponding traffic generated from Test Lab by filtering traffic originating from the following IP address blocks (note that you can also access this list by using the gcloud beta firebase test ip-blocks list command in the gcloud CLI):

Platform and device type CIDR IP address block
Android and iOS physical devices

108.177.6.0/23

70.32.147.112/28 (added 03-2020)

74.125.122.32/29 (added 04-2020)

108.177.29.80/28 (added 04-2020)

216.239.44.24/29 (added 04-2020)

Android virtual devices

34.68.194.64/29 (added 11-2019)

34.69.234.64/29 (added 11-2019)

34.73.34.72/29 (added 11-2019)

34.73.178.72/29 (added 11-2019)

35.192.160.56/29

35.196.166.80/29

35.196.169.240/29

35.203.128.0/28

35.234.176.160/28

35.243.2.0/27 (added 7-2019)

199.192.115.0/30

199.192.115.8/30

199.192.115.16/29