Analyzing Test Results

Whether you run your tests through the Firebase console or the command line interface, you can find your detailed test results in the Firebase console. Read on to learn how to analyze your test results.

Key concepts

To see test results head to the Test Lab section of the Firebase console.

You'll find a list of all of your previous test runs there. To understand the the results, it helps to know a bit about test matrices: Test Dimensions × Test Executions = Test Matrix

  • Test Dimensions: device attributes to test your app against, including device model, OS version, locale, and screen orientation.

  • Test Executions: individual test runs, each with a unique combination of test dimensions that you've selected to test.

  • Test Matrix: the results of your test executions on the selected test dimensions. If any test execution in the matrix fails, Test Lab marks the whole matrix as failed.

Interpret test matrix results

If you start your tests in Firebase console, Test Lab takes you right to your in-progress test matrix, where you can watch your tests' progress as it happens. If you used the command line tool instead, you can follow the URL it provides to get to the in-progress test matrix page.

The matrix has a line for each test execution Test Lab runs. The icon before the execution name displays that execution's status:

  • play_circle_outline In progress: The test execution is still running. Executions can take up to an hour to complete, depending on the number and complexity of tests in your app.
  • check_circle Passed: None of the execution's tests failed.
  • warning Failed: At least one of the execution's tests failed.
  • error Inconclusive: The test results were inconclusive, possibly due to a Test Lab error.
  • block Skipped: Test Lab skipped your test, because the device/OS version combination you selected is unavailable.

Interpreting results from a single test execution

From the test matrix results page, click one of the test executions to see the result of that specific test execution.

The page shows you stats for that test execution, including issues encountered in testing, a list of test cases, logs from the execution, and a video of the test running.

Detailed test results

Detailed test results are available in a Google Cloud Storage (GCS) bucket for 90 days. Click View Source Files in a test execution result page to see the bucket.

To retain detailed test results for longer than 90 days, send the test results to a more-permanent GCS bucket that you own using the --results-bucket gcloud command-line option. You can then set the Age setting to determine how long GCS stores the results. For more information on how to change the Age setting, see Lifecycle conditions.

Send feedback about...

Need help? Visit our support page.