There are multiple ways to use Firebase Test Lab for Android to run tests on your Android app, including the command line interface, Android Studio, and the Test Lab UI in the Firebase console. However you choose to start your tests, the results are stored in the Firebase project that you specify. This page describes how to review and analyze these test results.
To see the results from all your previous test runs, select Test Lab in the left navigation panel of your project in the Firebase console. This page displays all the test runs from the apps that you have tested with your project using Test Lab.
To review test results, you first need to understand three concepts:
Test Dimension. A test dimension is a set of related device attributes. The dimensions used by Test Lab are model, Android API level, locale, and screen orientation.
Test Execution. Each valid combination of test dimensions that you select for your test matrix is a test execution. Test Lab supports testing your app on up to 200 unique combinations at a time. You can select the specific model and Android API level combinations to use when testing your app to focus your testing; for example, when you are validating a bug fix. Each test execution can either pass or fail. If one test execution fails, the whole test matrix is marked as failed.
Test Matrix. The test matrix is the set of test executions that you select. When you run your app test on a test matrix, you get test results for each valid test execution created by combining test dimension attributes.
The following sections explain how to navigate test results.
Interpret test history results
When you navigate to your test results by selecting Test Lab, you see the results of tests you have run so far.
Testing history is grouped by app. Only the most recent five test matrices are shown for each app; if more are available, you can click the All Matrices link at the bottom of the app test list to see the complete list for that app.
Interpret test matrix results
When starting a test through the Test Lab UI, you are redirected to a page where you can see your test matrix and click a specific test execution to view test results. Android Studio and the gcloud command provide a URL for the test matrix results page as well.
In a typical test matrix, you might run a test across a dozen or so different devices. Each test execution can have a different outcome. The possible outcomes for any test execution in a test matrix include the following:
- Passed : No failures were encountered.
- Failed : At least one failure was encountered.
- Inconclusive : Test results were inconclusive, possibly due to a Test Lab error.
- Skipped : The selected dimension values for some test executions in the matrix were incompatible. This occurs when devices that you selected are incompatible with one or more of the Android API levels that you selected.
To review aggregated test results for all test matrices for a given app in your Firebase project, click the name of the app, as shown in the following example:
Example test matrix results page with only four test executions
This takes you to the test matrix list for your app, where you can click the name of any test matrix to see the test matrix results, and where you can click the name of the app (shown in the red box below) to view the test matrix list for other apps associated with your Firebase project.
Example test matrix list page
A test matrix can pass, fail, or be inconclusive. A test matrix is shown as failed or inconclusive if any test executions in that matrix fail or are inconclusive.
Interpret Robo test results
If you ran your tests with Robo, your results include videos and screenshots of Robo crawling your UI, in addition to the usual test metrics. Those video and screenshots include visual indications of the actions Robo took during the crawl, similar to the 'Show touches' feature in Android. You can use the indications to help you follow along with Robo's progress, and reproduce any bugs it might uncover.
Example Robo test results video
Interpreting results from a single test execution
From the test matrix results page, click on one of the test executions to see the result of that specific test execution.
Example test execution results page
On this page, you can see the time required for each test execution. You can also see the results for specific test cases that correspond to methods in your test APK (for instrumentation tests) and detailed test results, including test logs, screenshots, and videos. For Robo test, detailed test results also include an activity map that graphically shows the UI paths that were visited by Robo test.
For test executions run on physical devices with API level 21 and higher, you can also view the following performance metrics:
- Available for all apps: CPU usage, memory usage, and network activity (send/receive)
- Available for game apps that use the SurfaceView class: frames per second (fps)
Example performance metrics for a test execution
Detailed test results
Detailed test results are available for 90 days after you run a test and are stored in a Google Cloud Storage (GCS) bucket (but are also visible in the Firebase console). You can view detailed test results in the GCS bucket when you click View Source Files on the test execution results page. When detailed test results are no longer available, you can still see which tests passed or failed.
To retain detailed test results for longer than 90 days, you need to send these test results to a GCS bucket that you own using the --results-bucket gcloud command-line option. You can then set the Age setting to determine how long results are stored in your GCS bucket. See Lifecycle conditions for information about how to change the Age setting.