<br />

> [!WARNING]
> **Preview:** The App Testing agent feature is a
> preview release. A preview release is not subject to any SLA or deprecation policy and
> may receive limited or no support.
>
> In order to use the App Testing agent,
> you will need to opt-in to the feature from the [Firebase console](https://console.firebase.google.com/project/_/appdistribution).
> For information, see
> [Accessing the App Testing agent preview](https://firebase.google.com/docs/app-distribution/troubleshooting.md.txt?platform=android#app-testing-agent-preview).

The App Testing agent is a test case generation, management, and execution agent
powered by Gemini in Firebase. You define test goals in natural language,
and the agent uses AI to understand and navigate your app, simulate user
interactions, and provide detailed test results.

### AI prompt for AI assistance


***For AI assistance, copy the prompt below into your
AI coding agent
(like Gemini in Android Studio, Antigravity, Claude Code, Codex, and Cursor).**
You can customize the prompt and iterate with your agent. Visit
[Firebase agent skills](https://firebase.google.com/docs/ai-assistance/agent-skills.md.txt)
for detailed installation info.*

```
Help me use Firebase in my app.
1. Make sure that **Firebase agent skills** are installed and updated. For Claude Code, Codex, or Cursor agents, help me install the Firebase plugin. For other agents, run `npx skills add firebase/agent-skills`. Always try to use **Firebase agent skills** for Firebase-related tasks.
2. Help me create YAML test cases and run App Testing agent tests for my app using https://firebase.google.com/docs/app-distribution/android/app-testing-agent.md.txt.
```

## How App Testing agent uses your data

App Testing agent is provided by
[Gemini in Firebase](https://firebase.google.com/docs/gemini-in-firebase) and is governed under the
same terms. See [How Gemini in Firebase uses your
data](https://firebase.google.com/docs/gemini-in-firebase#how-gemini-in-firebase-uses-your-data) for more
information about how Gemini in Firebase uses your data.

## Before you begin

If you haven't already, [register your app with
Firebase](https://firebase.google.com/docs/android/setup#console).

If you aren't using any other Firebase products, you only have to create a
project and register your app. You don't need to add any SDKs to your app.
However, if you decide to use additional products in the future, be sure to
complete all of the steps in [Add Firebase using the
Firebase console](https://firebase.google.com/docs/android/setup#console).

> [!CAUTION]
> **Caution:** When you register your app with Firebase, make sure to enter the same package name as the app you're distributing. The package name value is case-sensitive and cannot be changed for your app in Firebase after it's registered with your Firebase project.

## Create a test case

To run AI-guided tests, the App Testing agent uses your natural language test
cases to execute tests against your app.

Each test case is broken down into steps, which will be executed in sequence.
Steps allow you to break up your test case into phases, each with their own
assertion. The agent may take many actions during any one step.

There are two ways to create a test case: using a YAML file or using the
Firebase console. YAML files let you manage the test cases
yourself, typically in a versioned source code repository. Alternatively, the
Firebase console can store your test cases remotely along with your App
Distribution data.

### Using YAML files

The following example shows a YAML file that defines two test cases:

```yaml
tests:
- displayName: Login as guest
  id: login-as-guest
  steps:
  - goal: Log in as a guest
    finalScreenAssertion: The home screen is visible
- displayName: View biography card birth date
  prerequisiteTestCaseId: login-as-guest
  steps:
  - goal: Open the article on "Bob Dylan"
    hint: Use the search function to find it
    finalScreenAssertion: >-
      The article is opened and the title "Bob Dylan" is visible.
  - goal: Find Bob Dylan's birthday in the article
    hint: >-
      Look for the "Born" section in the infobox on the right side of the page.
    finalScreenAssertion: >-
      The text "May 24, 1941" is visible on the screen.
```

You can organize groups of tests in a single file and separate these test suites
in multiple files. You can add a prerequisite test case by adding an ID to the
test and then referencing that ID using `prerequisiteTestCaseId`.

### Using the App Distribution console

Alternatively, you can create and manage your test cases in the Firebase
Console. To create a test case, open the App Distribution page of the
[Firebase console](https://console.firebase.google.com/project/_/appdistribution) and take
the following steps:

1. On the **Test Cases** tab, click **New test case** . If you don't want to create your own test case, you can modify or use the provided [example test
   case](https://firebase.google.com/docs/app-distribution/android/app-testing-agent#test-cases).
2. In the **Add test case** dialog, give the test case a name. This is used to identify the test, but is ignored by the agent.
3. (Optional) Select a **Prerequisite test case** that contains set-up steps to run before the main test. If the prerequisite test fails, the entire test will be marked as a failure. Steps and outcomes from the prerequisite and main tests will be shown together in the test results.
4. Consider breaking your test into multiple steps, by clicking the **Add
   another step** button.
5. Give each step a **Goal** that describes what the App Testing agent should do during that step.
6. (Optional) Add a **Hint** to provide additional information to help the App Testing agent understand and navigate your app during that step.
7. Add **Final screen assertion** to help the App Testing agent determine when the step has been successfully completed. This assertion should only refer to what's visible on the screen.
8. Click **Save** once you're done customizing your test.

#### Example test case

The following is an example of how to create a test case using the App Testing
agent:

|---|---|
| ##### Test title | Home page loads |
| ##### Goal | Load the home page |
| ##### Hint | Navigate past any onboarding screens. Dismiss any popups. Don't sign in. |
| ##### Final screen assertion | The main app home page is visible on screen, all images have loaded, and no errors are displayed. |

## Run a test

How you run your tests depends on how you create and manage your test cases. If
you define test cases using YAML files, you will run those tests using the
Firebase CLI. If you create your test cases in the App Distribution console, you
will run them from the console or through one of the App Distribution CLI tools.

### Using YAML files

Test cases defined in YAML files can be executed using the Firebase CLI.

1. Install or update to the latest version of the [Firebase
   CLI](https://firebase.google.com/docs/cli#install_the_firebase_cli). We recommend downloading the standalone binary for the CLI specific to your OS.
2. [Sign in](https://firebase.google.com/docs/cli.md.txt#sign-in-test-cli) and test that you can access your projects. Note that if you're using the Firebase CLI in a CI environment, you can also authenticate with a [service
   account](https://firebase.google.com/docs/app-distribution/authenticate-service-account?platform=android) or by using [login:ci](https://firebase.google.com/docs/cli.md.txt#cli-ci-systems).
3. Run the `apptesting:execute` command. For example:

   ```bash
   firebase apptesting:execute \
     --app=1:1234567890:android:0a1b2c3d4e5f67890 \
     --test-dir=./mytests \
     ./app/build/outputs/apk/debug/app-debug.apk
   ```

| apptesting:execute \[options\] \[/path/to/app/binary\] ||
|---|---|
| `--app` | **Required:** Your app's Firebase App ID. You can find the App ID in the Firebase console, on the [General Settings](https://console.firebase.google.com/project/_/settings/general/) page. `--app 1:1234567890:android:0a1b2c3d4e5f67890` |
| `--test-dir` | Path to a directory that contains test case YAML files. The command will search recursively under this directory, so the files can optionally be organized into subdirectories. If unset, "./tests" is used by default. |
| `--test-devices` or `--test-devices-file` | The devices to execute the tests on. You can specify the test devices as a semicolon-separated list of device specifications: `--test-devices "model=tokay,version=36,locale=en,orientation=portrait;model=b0q,version=33,locale=en,orientation=portrait"` Or, you can specify the path to a plain text file containing a semicolon-separated list of device specifications: `--test-devices-file "/path/to/test-devices.txt"` You can look up the available device models using the [gcloud CLI](https://firebase.google.com/docs/test-lab/android/available-testing-devices.md.txt). |
| `--test-username` | The username for automatic login to be used during the tests. |
| `--test-password` or `--test-password-file` | The password for automatic login to be used during the tests. Or, you can specify the path to a plain text file containing a password: `--test-password-file: "/path/to/test-password.txt"` |
| `--test-non-blocking` | If set, the command will start the tests and then return immediately instead of waiting for them to complete. To see the test results, visit the Firebase console. If this flag is not set, the command will block until the tests are complete, and exit with a failure code if any of the tests fail. |
| `--results-bucket` | A custom Google Cloud Storage (GCS) bucket where test results are stored. If left blank, the default bucket is used. The bucket must be owned by a billing-enabled project, and specifying a bucket results in billing charges for the storage used. |
| `--test-file-pattern` | A [regular expression](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_expressions) pattern. Only tests contained in files that match this pattern will be executed. |
| `--test-name-pattern` | A [regular expression](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_expressions) pattern. Only tests with display names that match this pattern will be executed. |
| `/path/to/app/binary` | **Optional:** The path to your app's binary. If omitted, the agent will use the last release uploaded to App Distribution for the specified app. |

### Using the App Distribution console

To run test cases stored in App Distribution, you can use the
Firebase console, the Firebase CLI, or App Distribution's Gradle or fastlane
plugins.

<br />

#### Run tests the Firebase console

<br />

The App Testing agent lets you run **AI-guided tests** in the console by
clicking the **Run tests** button from either the Releases or Test Cases
page. This opens the App Testing agent customization screen, where you can
choose one or more of your existing test cases for the agent to execute. You can
also choose the devices you want to test against, provide any login credentials,
and optionally specify a custom Google Cloud Storage bucket to store your test
results. Note that running from the console only supports tests that were
created using the console.

When a test run is successful, the App Testing agent will cache the actions
taken for each device. These actions can be replayed during the future test runs
to improve test repeatability. When **replayed actions** are used, the agent
will still use AI to confirm the actions have succeeded, using any final screen
assertions you have defined. If a test fails when using replayed actions, the
App Testing agent will fall back to using AI for all actions.

You can also choose to run a **Random crawl test** by changing the test
type. Random crawl tests use the [Automated
Tester](https://firebase.google.com/docs/app-distribution/android-automated-tester) feature.

> [!NOTE]
> **Note:** You can add a system variable to your app
> behavior to modify the way that it behaves when being run by the App Testing
> agent. For more information, see [Run
> tests with Android Studio](https://firebase.google.com/docs/test-lab/android/android-studio#modify_instrumented_test_behavior_for).

<br />

<br />

<br />

#### Run tests stored in App Distribution from the Firebase CLI

<br />

You must specify at least one test device and one test case ID to use the App
Testing agent feature. Test case IDs can be found and downloaded on the Test
Cases page of the Firebase console. For more information on getting started
with the Firebase CLI and different ways to configure your distribution, see
[Distribute Android apps to testers using the Firebase
CLI](https://firebase.google.com/docs/app-distribution/android/distribute-cli?apptype=apk#step_2_distribute_your_app_to_testers).

Run the `appdistribution:distribute` command to upload your app,
and use the following parameters to configure your distribution to the App
Testing agent feature:

| appdistribution:distribute options ||
|---|---|
| `--test-devices` or `--test-devices-file` | The devices to execute the tests on. You can specify the test devices as a semicolon-separated list of device specifications: `--test-devices "model=tokay,version=36,locale=en,orientation=portrait;model=b0q,version=33,locale=en,orientation=portrait"` Or, you can specify the path to a plain text file containing a semicolon-separated list of device specifications: `--test-devices-file "/path/to/test-devices.txt"` You can look up the available device models using the [gcloud CLI](https://firebase.google.com/docs/test-lab/android/available-testing-devices.md.txt). |
| `--test-username` | The username for automatic login to be used during tests. |
| `--test-password` or `--test-password-file` | The password for automatic login to be used during tests. Or, you can specify the path to a plain text file containing a password: `--test-password-file "/path/to/test-password.txt"` |
| `--test-non-blocking` | Run tests asynchronously. Visit the Firebase console for the automatic test results. |
| `--results-bucket` | A custom Google Cloud Storage (GCS) bucket where test results are stored. If left blank, the default bucket is used. The bucket must be owned by a billing-enabled project, and specifying a bucket results in billing charges for the storage used. |
| `--test-case-ids` or `--test-case-ids-file` | A comma-separated list of the test case IDs to run: `--test-case-ids "load-app,play-the-first-level"` Or, you can specify the path to a plain text file containing a comma or newline separated list of test case IDs: `--test-case-ids-file "/path/to/test-case-ids.txt"` |

<br />

<br />

<br />

#### Run tests stored in App Distribution with Gradle

<br />

You must specify at least one test device and one test case ID to use the App
Testing agent feature. Test case IDs can be found and downloaded on the Test
Cases page of the Firebase console. For more information on getting started
with Gradle and different ways to configure your distribution, see [Distribute
Android apps to testers using
Gradle](https://firebase.google.com/docs/app-distribution/android/distribute-gradle?apptype=apk#step_3_configure_your_distribution_properties).

You can configure App Distribution by adding at least one `firebaseAppDistribution`
section and use the following parameters to configure the distribution to the
App Testing agent feature:

| App Distribution Build Parameters ||
|---|---|
| `testDevices` or `testDevicesFile` | The devices to execute the tests on. You can specify the test devices as a semicolon-separated list of device specifications: `testDevices="model=tokay, version=36, locale=en, orientation=portrait;model=b0q, version=33, locale=en, orientation=portrait"` Or you can specify the path to a file containing a semicolon-separated list of device specifications: `testDevicesFile="/path/to/testDevices.txt"` You can look up the available device models using the [gcloud CLI](https://firebase.google.com/docs/test-lab/android/available-testing-devices.md.txt). |
| `testUsername` | The username for automatic login to be used during the tests. |
| `testPassword` or `testPasswordFile` | The password for automatic login to be used during the tests. Or, you can specify the path to a plain text file containing a password: ``` testPasswordFile="/path/to/testPassword.txt" ``` |
| `testNonBlocking` | Run tests asynchronously. Visit the FIrebase console for the automatic test results. |
| `resultsBucket` | A custom Google Cloud Storage (GCS) bucket where test results are stored. If left blank, the default bucket is used. The bucket must be owned by a billing-enabled project, and specifying a bucket results in billing charges for the storage used. |
| `testCases` or `testCasesFile` | A comma-separated list of the test case IDs to run: `testCases: "load-app,play-the-first-level"` Or, you can specify the path to a plain text file containing a comma or newline separated list of test case IDs: `testCasesFile: "/path/to/test-case-ids.txt"` |

<br />

<br />

<br />

#### Run tests stored in App Distribution with fastlane

<br />

You must specify at least one test device and one test case ID to use the App
Testing agent feature. Test case IDs can be found and downloaded on the Test
Cases page of the Firebase console. For more information on getting started
with fastlane and different ways to configure your distribution, see [Distribute
Android apps to testers using
fastlane](https://firebase.google.com/docs/app-distribution/android/firebase.google.com/docs/app-distribution/android/distribute-fastlane?apptype=apk#distribute).

In a `./fastlane/Fastfile` lane, add a
`firebase_app_distribution` block. Use the following parameters to
configure your distribution to the App Testing agent feature:

| firebase_app_distribution parameters ||
|---|---|
| `test_devices` or `test_devices_file` | The devices to execute the tests on. You can specify the test devices as a semicolon-separated list of device specifications: `test_devices: "model=tokay, version=36, locale=en, orientation=portrait;model=b0q, version=33, locale=en, orientation=portrait"` Or, you can specify the path to a plain text file containing a semicolon-separated list of device specifications: `test_devices_file: "/path/to/test-devices.txt"` You can look up the available device models using the [gcloud CLI](https://firebase.google.com/docs/test-lab/android/available-testing-devices.md.txt). |
| `test_username` | The username for automatic login to be used during tests. |
| `test_password` or `test_password_file` | The password for automatic login to be used during tests. Or, you can specify the path to a plain text file containing a password: `test_password_file: "/path/to/test-password.txt"` |
| `test_non_blocking` | Run tests asynchronously. Visit the Firebase console for the automatic test results. |
| `test_case_ids` or `test_case_ids_file` | A comma-separated list of the test case IDs to run: `test_case_ids: "load-app,play-the-first-level"` Or, you can specify the path to a plain text file containing a comma or newline separated list of test case IDs: `test_case_ids_file: "/path/to/test-case-ids.txt"` |

<br />

<br />

## Import \& export test cases with YAML files

Importing test cases from YAML files is useful when you want to manage test
cases outside of the Firebase console. It can also be useful to export test
cases to move them between projects. You can use an LLM to refine existing test
cases or create new test cases. You can import and export test cases from the
Test Cases page in the Firebase console or programmatically using the
Firebase CLI. For an example of a YAML test case, see [Create a YAML test
case](https://firebase.google.com/docs/app-distribution/android/app-testing-agent#create-using-yaml).

## View your test results

You can view the results of your tests from the **Releases** page in the
**App Testing agent** tab of a release. The **View details** button will open
the Test Results dialog and show you any issues, screenshots of the app, and the
actions that Gemini took during the test.

If a test includes replayed actions, you can view the original AI-guided test
run or clear the cache.

To learn more about your test results, use the following table:

| Icon | Name | Description |
|---|---|---|
| spark | AI action | Indicates that the App Testing agent used Gemini to decide to take an action or end the step. |
| replay | Replayed action | Indicates that the App Testing agent replayed an action from a previous successful run of the test. |
| spark | AI assertion | Indicates that the App Testing agent used Gemini to validate a final screen assertion, after replaying actions from a previous successful run of the same test. |

### Debug your test results

If your test results are different from what you expected, you can debug your
test using the **Show agent view** toggle under **View details** on the Test
Results page. The agent view shows you the on screen elements that the App
Testing agent was able to detect when using the app's accessibility information.
If you would like to take a closer look at what the agent saw, you can download
that information from the action overflow menu.

You can also use the **View artifacts** button on the Test Results page to look
at all of the videos, logs, and other Cloud artifacts for your test results.

## Known issues and limitations

The App Testing agent preview has some known limitations:

- AI-guided tests have a timeout of 5 minutes. After a test starts it must succeed within that time or it will end early and be counted as a test failure.
- Because the App Testing agent uses generative AI to test your app, it will sometimes take different actions while still following the same instructions.
- The App Testing agent only supports the following actions: tap, enter text, swipe up/down/left/right, long press, drag-and-drop, back, and wait.
- The App Testing agent has trouble executing tests containing only a single step that takes many actions to accomplish. It performs better when complex tasks are broken up into multiple shorter steps.
- The App Testing agent sometimes won't scroll to expose other elements off screen. This happens more often when there is no visual indication of scrollability. As a workaround, the "hints" field can be used to suggest scrolling.
- The App Testing agent sometimes has trouble counting, for example performing an action a specific number of times.
- The App Testing agent cannot navigate your app if [`FLAG_SECURE`](https://developer.android.com/security/fraud-prevention/activities#flag_secure) is enabled. Instead of screenshots of your app, it will only see a blank screen.

## Testing quotas

During the preview, the AI-guided tests will be offered at no cost within a
quota limit. The default quota limit is 200 tests per month, per Firebase
project.

Note that if you choose to run multiple test cases, or run the same test case,
on multiple devices, this counts as multiple tests. For example, if you run 2
test cases on 2 devices, this counts as a total of 4 tests.

To increase your quota above the default limit, contact [Firebase
Support](https://firebase.google.com/support) with your use case.