Automatically test Android apps

This guide describes how to automatically distribute builds to testers using the App Distribution Automated Tester feature in the Firebase console. The Automated Tester feature provides an always-ready, easy-to-use smoke testing tool for your app.

Before you begin

If you haven't already, add Firebase to your Android project.

If you aren't using any other Firebase products, you only have to create a project and register your app. However, if you decide to use additional products in the future, be sure to complete all of the steps in Add Firebase using the Firebase console.

When you're ready to distribute a pre-release version of your app to testers, build your APK using your normal process. You must sign the APK with your debug key or app signing key.

Distribute your app to Automated Tester using the Firebase console

To distribute your app to Automated Tester, upload your APK file using the Firebase console:

  1. Open the App Distribution page of the Firebase console. Select your Firebase project when prompted.
  2. On the Releases page, select the app you want to distribute from the drop-down menu.
  3. Drag your app's APK file to the console to upload it.
  4. When the upload completes, select Automated Tester in the Add testers or groups search bar.
  5. To customize your automated test, click the Edit icon and select the customizations in the Customize your automated test dialog.
  6. (Optional) In the device configuration dialog, select the API level, device orientation, and locale that meets your testing specifications.
  7. (Optional) Click the Login credentials drop-down menu to create a custom username and password that can be used during the test if your app requires login credentials.
  8. Click Distribute. You can also add an optional release note for your app.

Distribute your app to Automated Tester using the Firebase CLI

Run the appdistribution:distribute command to upload your app and distribute it to testers. Use the following parameters to configure your distribution to the Automated Tester feature:

appdistribution:distribute options
--test-devices or --test-devices-file

The test devices you want to distribute builds to the Automated Tester feature.

You can specify the test devices as a semicolon-separated list of test devices:

--test-devices: "model=shiba, version=34, locale=en, orientation=portrait;model=b0q, version=33, locale=en, orientation=portrait"

Or, you can specify the path to a plain text file containing a semicolon-separated list of test devices:

--test-devices-file: "/path/to/test-devices.txt"
--test-username

The username for automatic login to be used during automated tests.

--test-password or --test-password-file

The password for automatic login to be used during automated tests.

Or, you can specify the path to a plain text file containing a password:

--test-password-file: "/path/to/test-password.txt"
--test-username-resource

Resource name for the username field for automatic login to be used during automated tests.

--test-password-resource

Resource name for the password field for automatic login to be used during automated tests.

--test-non-blocking

Run automated tests asynchronously. Visit the Firebase console for the automatic test results.

For more information on getting started with the Firebase CLI and different ways to configure your distribution, see Distribute Android apps to testers using the Firebase CLI.

Distribute your app to Automated Tester using Gradle

You can configure App Distribution by adding at least one firebaseAppDistribution section and use the following parameters to configure the distribution to the Automated Tester feature:

App Distribution Build Parameters
testDevices or testDevicesFile

The test devices you want to distribute builds to using automated tests.

You can specify the test devices as a semicolon-separated list of device specifications

testDevices="model=shiba, version=34, locale=en, orientation=portrait;model=b0p, version=33, locale=en, orientation=portrait"

Or you can specify the path to a file containing a semicolon-separated list of device specifications:

testDevicesFile="/path/to/testDevices.txt"
testUsername

The username for automatic login to be used during automated tests.

testUsernameResource

Resource name for the username field for automatic login to be used during automated tests.

testPassword or testPasswordFile

The password for automatic login to be used during automated tests.

Or, you can specify the path to a plain text file containing a password:

testPasswordFile="/path/to/testPassword.txt"
testPasswordResource

Resource name for the password field for automatic login to be used during automated tests.

testNonBlocking

Run automated tests asynchronously. Visit the FIrebase console for the automatic test results.

For more information on getting started with Gradle and different ways to configure your distribution, see Distribute Android apps to testers using Gradle.

Distribute your app to Automated Tester using fastlane

In a ./fastlane/Fastfile lane, add a firebase_app_distribution block. Use the following parameters to configure your distribution to the Automated Tester feature:

firebase_app_distribution parameters
test_devices or test_devices_file

The test devices you want to distribute builds to the Automated Tester feature. You can specify the test devices as a semicolon-separated list of test devices:

test_devices: "model=shiba, version=34, locale=en, orientation=portrait;model=b0q, version=33, locale=en, orientation=portrait"

Or, you can specify the path to a plain text file containing a semicolon-separated list of test devices:

test_devices_file: "/path/to/test-devices.txt"
test_username

The username for automatic login to be used during automated tests.

test_password or test_password_file

The password for automatic login to be used during automated tests.

Or, you can specify the path to a plain text file containing a password:

test_password_file: "/path/to/test-password.txt"
test_username_resource

Resource name for the username field for automatic login to be used during automated tests.

test_password_resource

Resource name for the password field for automatic login to be used during automated tests.

test_non_blocking

Run automated tests asynchronously. Visit the FIrebase console for the automatic test results.

For more information on getting started with fastlane and different ways to configure your distribution, see Distribute Android apps to testers using fastlane.

Automatic test crawl statuses

To help you interpret your test results, the Automated test results tab shows any test issues, screenshots of the app, and a video of the test crawl for each device. The status of the test can be found in your test results page:

Crawl status Description
Pending The automatic test is still running and your results will be available soon.
Passed The automatic test successfully crawled your app and did not find any crashes.
Failed The automatic test failed because your app crashed during the test.
Inconclusive The automatic test failed because of infrastructure errors.