There are multiple ways to use Firebase Test Lab for Android to run tests on your Android app, including the command line interface, Android Studio, and the Test Lab UI in the Firebase console. However you choose to start your tests, the results are stored in the Firebase project that you specify. This page describes how to review and analyze these test results.
To see the results from all your previous test runs, select Test Lab in the left navigation panel of your project in the Firebase console. This page displays all the test runs from the apps that you have tested with your project using Test Lab.
To review test results, you first need to understand three concepts:
- Test Dimension. A test dimension is a set of related device attributes. The dimensions used by Test Lab are device type, Android API level, locale, and screen orientation. Before testing your app, you select attributes from each dimension, and Test Lab creates a list of all valid combinations of these attributes. Test Lab supports testing your app on up to 200 unique combinations at a time.
Test Execution. Each valid combination of test dimensions that you select for your test matrix is a test execution. For example, if you select five device types (Asus Google Nexus 7, LG G3, Samsung Galaxy Note 2, etc.), two Android API levels (21, 22), two locales (en_US, es_ES), and two screen orientations (landscape, portrait), the test contains 5*2*2*2 = 40 test executions. Each test execution can either pass or fail. If one test execution fails, the whole test matrix is marked as failed.
Test Matrix. The test matrix is the set of test executions defined by the combinations of the test dimension attributes that you select. When you run your app test on a test matrix, you get test results for each valid test execution created by combining test dimension attributes.
The following sections explain how to navigate test results.
Interpret test history results
When you navigate to your test results by selecting Test Lab, you see the results of tests you have run so far.
Testing history is grouped by app. Only the most recent five test matrices are shown for each app; if more are available, you can click the All Matrices link at the bottom of the app test list to see the complete list for that app.
Interpret test matrix results
When starting a test through Test Lab UI, you are redirected to a page where you can see your test matrix and click a specific test execution to view test results. Android Studio and the gcloud command provide a URL for the test matrix results page as well.
In a typical test matrix, you might run a test across a dozen or so different devices. Each test execution can have a different outcome. The possible outcomes for any test execution in a test matrix include:
- Passed : No failures were encountered.
- Failed : At least one failure was encountered.
- Inconclusive : Test results were inconclusive, possibly due to a Test Lab error.
- Skipped : The selected dimension values for some test executions in the matrix were incompatible. This occurs when devices that you selected are incompatible with one or more of the Android API levels that you selected.
To review aggregated test results for all test matrices for a given app in your Firebase project, click the name of the app, as shown in the example test matrix results page shown below.
Example of a test matrix results page with only four test executions
This takes you to the test matrix list for your app, where you can click the name of any test matrix to see the test matrix results, and where you can click the name of the app (shown in the red box below) to view the test matrix list for other apps associated with your Firebase project.
Example test matrix list page
A test matrix can either pass, fail, or be inconclusive. A test matrix is shown as failed or inconclusive if any test executions in that matrix fail or are inconclusive.
Interpreting results from a single test execution
From the test matrix results page, click on one of the test executions to see the result of that specific test execution.
Example test execution results page
On this page, you can see results for specific test cases that correspond to methods in your test APK (for instrumentation tests). You can also see detailed test results; including test logs, screenshots, and videos. For Robo test, detailed test results also include an Activity Map that graphically shows the UI paths that were visited by Robo test.
Detailed test results
Detailed test results are available for 90 days after you run a test, and are stored in a Google Cloud Storage (GCS) bucket (but are also visible in the Firebase console). You can view detailed test results in the GCS bucket when you click View Source Files on the test execution results page. When detailed test results are no longer available, you can still see which tests passed and failed.
To retain detailed test results for longer than 90 days, you need to send these test results to a GCS bucket that you own using the --results-bucket gcloud command-line option. You can then set the Age setting to determine how long results are stored in your GCS bucket. See Lifecycle conditions for information about how to change the Age setting.