REST Resource: projects.testMatrices

Resource: TestMatrix

TestMatrix captures all details about a test. It contains the environment configuration, test specification, test executions and overall state and outcome.

JSON representation 
{
  "testMatrixId": string,
  "projectId": string,
  "clientInfo": {
    object (ClientInfo)
  },
  "testSpecification": {
    object (TestSpecification)
  },
  "environmentMatrix": {
    object (EnvironmentMatrix)
  },
  "testExecutions": [
    {
      object (TestExecution)
    }
  ],
  "resultStorage": {
    object (ResultStorage)
  },
  "state": enum (TestState),
  "timestamp": string,
  "invalidMatrixDetails": enum (InvalidMatrixDetails),
  "extendedInvalidMatrixDetails": [
    {
      object (MatrixErrorDetail)
    }
  ],
  "flakyTestAttempts": integer,
  "outcomeSummary": enum (OutcomeSummary),
  "failFast": boolean
}
Fields
testMatrixId

string

Output only. Unique id set by the service.
projectId

string

The cloud project that owns the test matrix.
clientInfo

object (ClientInfo)

Information about the client which invoked the test.
testSpecification

object (TestSpecification)

Required. How to run the test.
environmentMatrix

object (EnvironmentMatrix)

Required. The devices the tests are being executed on.
testExecutions[]

object (TestExecution)

Output only. The list of test executions that the service creates for this matrix.
resultStorage

object (ResultStorage)

Required. Where the results for the matrix are written.
state

enum (TestState)

Output only. Indicates the current progress of the test matrix.
timestamp

string (Timestamp format)

Output only. The time this test matrix was initially created.

A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: "2014-10-02T15:01:23Z" and "2014-10-02T15:01:23.045123456Z".
invalidMatrixDetails

enum (InvalidMatrixDetails)

Output only. Describes why the matrix is considered invalid. Only useful for matrices in the INVALID state.
extendedInvalidMatrixDetails[]

object (MatrixErrorDetail)

Output only. Details about why a matrix was deemed invalid. If multiple checks can be safely performed, they will be reported but no assumptions should be made about the length of this list.
flakyTestAttempts

integer

The number of times a TestExecution should be re-attempted if one or more of its test cases fail for any reason. The maximum number of reruns allowed is 10.

Default is 0, which implies no reruns.
outcomeSummary

enum (OutcomeSummary)

Output Only. The overall outcome of the test. Only set when the test matrix state is FINISHED.
failFast

boolean

If true, only a single attempt at most will be made to run each execution/shard in the matrix. Flaky test attempts are not affected.

Normally, 2 or more attempts are made if a potential infrastructure issue is detected.

This feature is for latency sensitive workloads. The incidence of execution failures may be significantly greater for fail-fast matrices and support is more limited because of that expectation.

ClientInfo

Information about the client which invoked the test.

JSON representation 
{
  "name": string,
  "clientInfoDetails": [
    {
      object (ClientInfoDetail)
    }
  ]
}
Fields
name

string

Required. Client name, such as gcloud.
clientInfoDetails[]

object (ClientInfoDetail)

The list of detailed information about client.

ClientInfoDetail

Key-value pair of detailed information about the client which invoked the test. Examples: {'Version', '1.0'}, {'Release Track', 'BETA'}.

JSON representation 
{
  "key": string,
  "value": string
}
Fields
key

string

Required. The key of detailed client information.
value

string

Required. The value of detailed client information.

TestSpecification

A description of how to run the test.

JSON representation 
{
  "testTimeout": string,
  "disableVideoRecording": boolean,
  "disablePerformanceMetrics": boolean,

  // Union field setup can be only one of the following:
  "testSetup": {
    object (TestSetup)
  },
  "iosTestSetup": {
    object (IosTestSetup)
  }
  // End of list of possible types for union field setup.

  // Union field test can be only one of the following:
  "androidInstrumentationTest": {
    object (AndroidInstrumentationTest)
  },
  "androidRoboTest": {
    object (AndroidRoboTest)
  },
  "androidTestLoop": {
    object (AndroidTestLoop)
  },
  "iosXcTest": {
    object (IosXcTest)
  },
  "iosTestLoop": {
    object (IosTestLoop)
  },
  "iosRoboTest": {
    object (IosRoboTest)
  }
  // End of list of possible types for union field test.
}
Fields
testTimeout

string (Duration format)

Max time a test execution is allowed to run before it is automatically cancelled. The default value is 5 min.

A duration in seconds with up to nine fractional digits, ending with 's'. Example: "3.5s".
disableVideoRecording

boolean

Disables video recording. May reduce test latency.
disablePerformanceMetrics

boolean

Disables performance metrics recording. May reduce test latency.
Union field setup. Test setup requirements. setup can be only one of the following:
testSetup

object (TestSetup)

Test setup requirements for Android e.g. files to install, bootstrap scripts.
iosTestSetup

object (IosTestSetup)

Test setup requirements for iOS.
Union field test. Required. The type of test to run. test can be only one of the following:
androidInstrumentationTest

object (AndroidInstrumentationTest)

An Android instrumentation test.
androidRoboTest

object (AndroidRoboTest)

An Android robo test.
androidTestLoop

object (AndroidTestLoop)

An Android Application with a Test Loop.
iosXcTest

object (IosXcTest)

An iOS XCTest, via an .xctestrun file.
iosTestLoop

object (IosTestLoop)

An iOS application with a test loop.
iosRoboTest

object (IosRoboTest)

An iOS Robo test.

TestSetup

A description of how to set up the Android device prior to running the test.

JSON representation 
{
  "filesToPush": [
    {
      object (DeviceFile)
    }
  ],
  "directoriesToPull": [
    string
  ],
  "initialSetupApks": [
    {
      object (Apk)
    }
  ],
  "additionalApks": [
    {
      object (Apk)
    }
  ],
  "account": {
    object (Account)
  },
  "networkProfile": string,
  "environmentVariables": [
    {
      object (EnvironmentVariable)
    }
  ],
  "systrace": {
    object (SystraceSetup)
  },
  "dontAutograntPermissions": boolean
}
Fields
filesToPush[]

object (DeviceFile)

List of files to push to the device before starting the test.
directoriesToPull[]

string

List of directories on the device to upload to GCS at the end of the test; they must be absolute paths under /sdcard, /storage or /data/local/tmp. Path names are restricted to characters a-z A-Z 0-9 _ - . + and /

Note: The paths /sdcard and /data will be made available and treated as implicit path substitutions. E.g. if /sdcard on a particular device does not map to external storage, the system will replace it with the external storage path prefix for that device.
initialSetupApks[]

object (Apk)

Optional. Initial setup APKs to install before the app under test is installed. Currently capped at 100.
additionalApks[]

object (Apk)

APKs to install in addition to those being directly tested. These will be installed after the app under test. Currently capped at 100.
account

object (Account)

The device will be logged in on this account for the duration of the test.
networkProfile

string

The network traffic profile used for running the test. Available network profiles can be queried by using the NETWORK_CONFIGURATION environment type when calling TestEnvironmentDiscoveryService.GetTestEnvironmentCatalog.
environmentVariables[]

object (EnvironmentVariable)

Environment variables to set for the test (only applicable for instrumentation tests).
systrace
(deprecated)

object (SystraceSetup)

Systrace configuration for the run. Deprecated: Systrace used Python 2 which was sunsetted on 2020-01-01. Systrace is no longer supported in the Cloud Testing API, and no Systrace file will be provided in the results.
dontAutograntPermissions

boolean

Whether to prevent all runtime permissions to be granted at app install

DeviceFile

A single device file description.

JSON representation 
{

  // Union field device_file can be only one of the following:
  "obbFile": {
    object (ObbFile)
  },
  "regularFile": {
    object (RegularFile)
  }
  // End of list of possible types for union field device_file.
}
Fields
Union field device_file. Required. device_file can be only one of the following:
obbFile

object (ObbFile)

A reference to an opaque binary blob file.
regularFile

object (RegularFile)

A reference to a regular file.

ObbFile

An opaque binary blob file to install on the device before the test starts.

JSON representation 
{
  "obbFileName": string,
  "obb": {
    object (FileReference)
  }
}
Fields
obbFileName

string

Required. OBB file name which must conform to the format as specified by Android e.g. [main|patch].0300110.com.example.android.obb which will be installed into <shared-storage>/Android/obb/<package-name>/ on the device.
obb

object (FileReference)

Required. Opaque Binary Blob (OBB) file(s) to install on the device.

FileReference

A reference to a file, used for user inputs.

JSON representation 
{

  // Union field file can be only one of the following:
  "gcsPath": string
  // End of list of possible types for union field file.
}
Fields
Union field file. Required. The file reference. file can be only one of the following:
gcsPath

string

A path to a file in Google Cloud Storage. Example: gs://build-app-1414623860166/app%40debug-unaligned.apk These paths are expected to be url encoded (percent encoding)

RegularFile

A file or directory to install on the device before the test starts.

JSON representation 
{
  "content": {
    object (FileReference)
  },
  "devicePath": string
}
Fields
content

object (FileReference)

Required. The source file.
devicePath

string

Required. Where to put the content on the device. Must be an absolute, allowlisted path. If the file exists, it will be replaced. The following device-side directories and any of their subdirectories are allowlisted:

${EXTERNAL_STORAGE}, /sdcard, or /storage

${ANDROID_DATA}/local/tmp, or /data/local/tmp

Specifying a path outside of these directory trees is invalid.

The paths /sdcard and /data will be made available and treated as implicit path substitutions. E.g. if /sdcard on a particular device does not map to external storage, the system will replace it with the external storage path prefix for that device and copy the file there.

It is strongly advised to use the Environment API in app and test code to access files on the device in a portable way.

Apk

An Android package file to install.

JSON representation 
{
  "location": {
    object (FileReference)
  },
  "packageName": string
}
Fields
location

object (FileReference)

The path to an APK to be installed on the device before the test begins.
packageName

string

The java package for the APK to be installed. Value is determined by examining the application's manifest.

Account

Identifies an account and how to log into it.

JSON representation 
{

  // Union field account_type can be only one of the following:
  "googleAuto": {
    object (GoogleAuto)
  }
  // End of list of possible types for union field account_type.
}
Fields
Union field account_type. Required. The type of account, based what it's for (e.g. Google) and what its login mechanism is (e.g. username and password). account_type can be only one of the following:
googleAuto

object (GoogleAuto)

An automatic google login account.

GoogleAuto

This type has no fields.

Enables automatic Google account login. If set, the service automatically generates a Google test account and adds it to the device, before executing the test. Note that test accounts might be reused. Many applications show their full set of functionalities when an account is present on the device. Logging into the device with these generated accounts allows testing more functionalities.

EnvironmentVariable

A key-value pair passed as an environment variable to the test.

JSON representation 
{
  "key": string,
  "value": string
}
Fields
key

string

Key for the environment variable.
value

string

Value for the environment variable.

SystraceSetup

JSON representation 
{
  "durationSeconds": integer
}
Fields
durationSeconds
(deprecated)

integer

Systrace duration in seconds. Should be between 1 and 30 seconds. 0 disables systrace.

IosTestSetup

A description of how to set up an iOS device prior to running the test.

JSON representation 
{
  "networkProfile": string,
  "additionalIpas": [
    {
      object (FileReference)
    }
  ],
  "pushFiles": [
    {
      object (IosDeviceFile)
    }
  ],
  "pullDirectories": [
    {
      object (IosDeviceFile)
    }
  ]
}
Fields
networkProfile

string

The network traffic profile used for running the test. Available network profiles can be queried by using the NETWORK_CONFIGURATION environment type when calling TestEnvironmentDiscoveryService.GetTestEnvironmentCatalog.
additionalIpas[]

object (FileReference)

iOS apps to install in addition to those being directly tested.
pushFiles[]

object (IosDeviceFile)

List of files to push to the device before starting the test.
pullDirectories[]

object (IosDeviceFile)

List of directories on the device to upload to Cloud Storage at the end of the test.

Directories should either be in a shared directory (such as /private/var/mobile/Media) or within an accessible directory inside the app's filesystem (such as /Documents) by specifying the bundle ID.

IosDeviceFile

A file or directory to install on the device before the test starts.

JSON representation 
{
  "content": {
    object (FileReference)
  },
  "bundleId": string,
  "devicePath": string
}
Fields
content

object (FileReference)

The source file
bundleId

string

The bundle id of the app where this file lives.

iOS apps sandbox their own filesystem, so app files must specify which app installed on the device.
devicePath

string

Location of the file on the device, inside the app's sandboxed filesystem

AndroidInstrumentationTest

A test of an Android application that can control an Android component independently of its normal lifecycle. Android instrumentation tests run an application APK and test APK inside the same process on a virtual or physical AndroidDevice. They also specify a test runner class, such as com.google.GoogleTestRunner, which can vary on the specific instrumentation framework chosen.

See https://developer.android.com/training/testing/fundamentals for more information on types of Android tests.

JSON representation 
{
  "testApk": {
    object (FileReference)
  },
  "appPackageId": string,
  "testPackageId": string,
  "testRunnerClass": string,
  "testTargets": [
    string
  ],
  "orchestratorOption": enum (OrchestratorOption),
  "shardingOption": {
    object (ShardingOption)
  },

  // Union field app_under_test can be only one of the following:
  "appApk": {
    object (FileReference)
  },
  "appBundle": {
    object (AppBundle)
  }
  // End of list of possible types for union field app_under_test.
}
Fields
testApk

object (FileReference)

Required. The APK containing the test code to be executed.
appPackageId

string

The java package for the application under test. The default value is determined by examining the application's manifest.
testPackageId

string

The java package for the test to be executed. The default value is determined by examining the application's manifest.
testRunnerClass

string

The InstrumentationTestRunner class. The default value is determined by examining the application's manifest.
testTargets[]

string

Each target must be fully qualified with the package name or class name, in one of these formats:

  • "package packageName"
  • "class packageName.class_name"
  • "class packageName.class_name#methodName"

If empty, all targets in the module will be run.
orchestratorOption

enum (OrchestratorOption)

The option of whether running each test within its own invocation of instrumentation with Android Test Orchestrator or not. ** Orchestrator is only compatible with AndroidJUnitRunner version 1.1 or higher! ** Orchestrator offers the following benefits:

  • No shared state
  • Crashes are isolated
  • Logs are scoped per test

See https://developer.android.com/training/testing/junit-runner.html#using-android-test-orchestrator for more information about Android Test Orchestrator.

If not set, the test will be run without the orchestrator.
shardingOption

object (ShardingOption)

The option to run tests in multiple shards in parallel.
Union field app_under_test. Required. app_under_test can be only one of the following:
appApk

object (FileReference)

The APK for the application under test.
appBundle

object (AppBundle)

A multi-apk app bundle for the application under test.

AppBundle

An Android App Bundle file format, containing a BundleConfig.pb file, a base module directory, zero or more dynamic feature module directories.

See https://developer.android.com/guide/app-bundle/build for guidance on building App Bundles.

JSON representation 
{

  // Union field bundle can be only one of the following:
  "bundleLocation": {
    object (FileReference)
  }
  // End of list of possible types for union field bundle.
}
Fields
Union field bundle. Required. Bundle location information. bundle can be only one of the following:
bundleLocation

object (FileReference)

.aab file representing the app bundle under test.

OrchestratorOption

Specifies how to execute the test.

Enums
ORCHESTRATOR_OPTION_UNSPECIFIED Default value: the server will choose the mode. Currently implies that the test will run without the orchestrator. In the future, all instrumentation tests will be run with the orchestrator. Using the orchestrator is highly encouraged because of all the benefits it offers.
USE_ORCHESTRATOR Run test using orchestrator. ** Only compatible with AndroidJUnitRunner version 1.1 or higher! ** Recommended.
DO_NOT_USE_ORCHESTRATOR Run test without using orchestrator.

ShardingOption

Options for enabling sharding.

JSON representation 
{

  // Union field option can be only one of the following:
  "uniformSharding": {
    object (UniformSharding)
  },
  "manualSharding": {
    object (ManualSharding)
  },
  "smartSharding": {
    object (SmartSharding)
  }
  // End of list of possible types for union field option.
}
Fields

Union field option.

option can be only one of the following:
uniformSharding

object (UniformSharding)

Uniformly shards test cases given a total number of shards.
manualSharding

object (ManualSharding)

Shards test cases into the specified groups of packages, classes, and/or methods.
smartSharding

object (SmartSharding)

Shards test based on previous test case timing records.

UniformSharding

Uniformly shards test cases given a total number of shards.

For instrumentation tests, it will be translated to "-e numShard" and "-e shardIndex" AndroidJUnitRunner arguments. With uniform sharding enabled, specifying either of these sharding arguments via environmentVariables is invalid.

Based on the sharding mechanism AndroidJUnitRunner uses, there is no guarantee that test cases will be distributed uniformly across all shards.

JSON representation 
{
  "numShards": integer
}
Fields
numShards

integer

Required. The total number of shards to create. This must always be a positive number that is no greater than the total number of test cases. When you select one or more physical devices, the number of shards must be <= 50. When you select one or more ARM virtual devices, it must be <= 200. When you select only x86 virtual devices, it must be <= 500.

ManualSharding

Shards test cases into the specified groups of packages, classes, and/or methods.

With manual sharding enabled, specifying test targets via environmentVariables or in InstrumentationTest is invalid.

JSON representation 
{
  "testTargetsForShard": [
    {
      object (TestTargetsForShard)
    }
  ]
}
Fields
testTargetsForShard[]

object (TestTargetsForShard)

Required. Group of packages, classes, and/or test methods to be run for each manually-created shard. You must specify at least one shard if this field is present. When you select one or more physical devices, the number of repeated testTargetsForShard must be <= 50. When you select one or more ARM virtual devices, it must be <= 200. When you select only x86 virtual devices, it must be <= 500.

TestTargetsForShard

Test targets for a shard.

JSON representation 
{
  "testTargets": [
    string
  ]
}
Fields
testTargets[]

string

Group of packages, classes, and/or test methods to be run for each shard. The targets need to be specified in AndroidJUnitRunner argument format. For example, "package com.my.packages" "class com.my.package.MyClass".

The number of testTargets must be greater than 0.

SmartSharding

Shards test based on previous test case timing records.

JSON representation 
{
  "targetedShardDuration": string
}
Fields
targetedShardDuration

string (Duration format)

The amount of time tests within a shard should take.

Default: 300 seconds (5 minutes). The minimum allowed: 120 seconds (2 minutes).

The shard count is dynamically set based on time, up to the maximum shard limit (described below). To guarantee at least one test case for each shard, the number of shards will not exceed the number of test cases. Shard duration will be exceeded if:

  • The maximum shard limit is reached and there is more calculated test time remaining to allocate into shards.
  • Any individual test is estimated to be longer than the targeted shard duration.

Shard duration is not guaranteed because smart sharding uses test case history and default durations which may not be accurate. The rules for finding the test case timing records are:

  • If the service has processed a test case in the last 30 days, the record of the latest successful test case will be used.
  • For new test cases, the average duration of other known test cases will be used.
  • If there are no previous test case timing records available, the default test case duration is 15 seconds.

Because the actual shard duration can exceed the targeted shard duration, we recommend that you set the targeted value at least 5 minutes less than the maximum allowed test timeout (45 minutes for physical devices and 60 minutes for virtual), or that you use the custom test timeout value that you set. This approach avoids cancelling the shard before all tests can finish.

Note that there is a limit for maximum number of shards. When you select one or more physical devices, the number of shards must be <= 50. When you select one or more ARM virtual devices, it must be <= 200. When you select only x86 virtual devices, it must be <= 500. To guarantee at least one test case for per shard, the number of shards will not exceed the number of test cases. Each shard created counts toward daily test quota.

A duration in seconds with up to nine fractional digits, ending with 's'. Example: "3.5s".

AndroidRoboTest

A test of an android application that explores the application on a virtual or physical Android Device, finding culprits and crashes as it goes.

JSON representation 
{
  "appPackageId": string,
  "appInitialActivity": string,
  "maxDepth": integer,
  "maxSteps": integer,
  "roboDirectives": [
    {
      object (RoboDirective)
    }
  ],
  "roboMode": enum (RoboMode),
  "roboScript": {
    object (FileReference)
  },
  "startingIntents": [
    {
      object (RoboStartingIntent)
    }
  ],

  // Union field app_under_test can be only one of the following:
  "appApk": {
    object (FileReference)
  },
  "appBundle": {
    object (AppBundle)
  }
  // End of list of possible types for union field app_under_test.
}
Fields
appPackageId

string

The java package for the application under test. The default value is determined by examining the application's manifest.
appInitialActivity

string

The initial activity that should be used to start the app.
maxDepth
(deprecated)

integer

The max depth of the traversal stack Robo can explore. Needs to be at least 2 to make Robo explore the app beyond the first activity. Default is 50.
maxSteps
(deprecated)

integer

The max number of steps Robo can execute. Default is no limit.
roboDirectives[]

object (RoboDirective)

A set of directives Robo should apply during the crawl. This allows users to customize the crawl. For example, the username and password for a test account can be provided.
roboMode

enum (RoboMode)

The mode in which Robo should run. Most clients should allow the server to populate this field automatically.
roboScript

object (FileReference)

A JSON file with a sequence of actions Robo should perform as a prologue for the crawl.
startingIntents[]

object (RoboStartingIntent)

The intents used to launch the app for the crawl. If none are provided, then the main launcher activity is launched. If some are provided, then only those provided are launched (the main launcher activity must be provided explicitly).
Union field app_under_test. Required. app_under_test can be only one of the following:
appApk

object (FileReference)

The APK for the application under test.
appBundle

object (AppBundle)

A multi-apk app bundle for the application under test.

RoboDirective

Directs Robo to interact with a specific UI element if it is encountered during the crawl. Currently, Robo can perform text entry or element click.

JSON representation 
{
  "resourceName": string,
  "inputText": string,
  "actionType": enum (RoboActionType)
}
Fields
resourceName

string

Required. The android resource name of the target UI element. For example, in Java: R.string.foo in xml: @string/foo Only the "foo" part is needed. Reference doc: https://developer.android.com/guide/topics/resources/accessing-resources.html
inputText

string

The text that Robo is directed to set. If left empty, the directive will be treated as a CLICK on the element matching the resourceName.
actionType

enum (RoboActionType)

Required. The type of action that Robo should perform on the specified element.

RoboActionType

Actions which Robo can perform on UI elements.

Enums
ACTION_TYPE_UNSPECIFIED DO NOT USE. For proto versioning only.
SINGLE_CLICK Direct Robo to click on the specified element. No-op if specified element is not clickable.
ENTER_TEXT Direct Robo to enter text on the specified element. No-op if specified element is not enabled or does not allow text entry.
IGNORE Direct Robo to ignore interactions with a specific element.

RoboMode

The mode in which Robo should run.

Enums
ROBO_MODE_UNSPECIFIED This means that the server should choose the mode. Recommended.
ROBO_VERSION_1 Runs Robo in UIAutomator-only mode without app resigning
ROBO_VERSION_2 Runs Robo in standard Espresso with UIAutomator fallback

RoboStartingIntent

Message for specifying the start activities to crawl.

JSON representation 
{
  "timeout": string,

  // Union field starting_intent can be only one of the following:
  "launcherActivity": {
    object (LauncherActivityIntent)
  },
  "startActivity": {
    object (StartActivityIntent)
  },
  "noActivity": {
    object (NoActivityIntent)
  }
  // End of list of possible types for union field starting_intent.
}
Fields
timeout

string (Duration format)

Timeout in seconds for each intent.

A duration in seconds with up to nine fractional digits, ending with 's'. Example: "3.5s".
Union field starting_intent. Required. Intent details to start an activity. starting_intent can be only one of the following:
launcherActivity

object (LauncherActivityIntent)

An intent that starts the main launcher activity.
startActivity

object (StartActivityIntent)

An intent that starts an activity with specific details.
noActivity

object (NoActivityIntent)

Skips the starting activity

LauncherActivityIntent

This type has no fields.

Specifies an intent that starts the main launcher activity.

StartActivityIntent

A starting intent specified by an action, uri, and categories.

JSON representation 
{
  "action": string,
  "uri": string,
  "categories": [
    string
  ]
}
Fields
action

string

Action name. Required for START_ACTIVITY.
uri

string

URI for the action.
categories[]

string

Intent categories to set on the intent.

NoActivityIntent

This type has no fields.

Skips the starting activity

AndroidTestLoop

A test of an Android Application with a Test Loop. The intent <intent-name> will be implicitly added, since Games is the only user of this api, for the time being.

JSON representation 
{
  "appPackageId": string,
  "scenarios": [
    integer
  ],
  "scenarioLabels": [
    string
  ],

  // Union field app_under_test can be only one of the following:
  "appApk": {
    object (FileReference)
  },
  "appBundle": {
    object (AppBundle)
  }
  // End of list of possible types for union field app_under_test.
}
Fields
appPackageId

string

The java package for the application under test. The default is determined by examining the application's manifest.
scenarios[]

integer

The list of scenarios that should be run during the test. The default is all test loops, derived from the application's manifest.
scenarioLabels[]

string

The list of scenario labels that should be run during the test. The scenario labels should map to labels defined in the application's manifest. For example, player_experience and com.google.test.loops.player_experience add all of the loops labeled in the manifest with the com.google.test.loops.player_experience name to the execution. Scenarios can also be specified in the scenarios field.
Union field app_under_test. Required. The Android package to test. app_under_test can be only one of the following:
appApk

object (FileReference)

The APK for the application under test.
appBundle

object (AppBundle)

A multi-apk app bundle for the application under test.

IosXcTest

A test of an iOS application that uses the XCTest framework. Xcode supports the option to "build for testing", which generates an .xctestrun file that contains a test specification (arguments, test methods, etc). This test type accepts a zip file containing the .xctestrun file and the corresponding contents of the Build/Products directory that contains all the binaries needed to run the tests.

JSON representation 
{
  "testsZip": {
    object (FileReference)
  },
  "xctestrun": {
    object (FileReference)
  },
  "xcodeVersion": string,
  "appBundleId": string,
  "testSpecialEntitlements": boolean
}
Fields
testsZip

object (FileReference)

Required. The .zip containing the .xctestrun file and the contents of the DerivedData/Build/Products directory. The .xctestrun file in this zip is ignored if the xctestrun field is specified.
xctestrun

object (FileReference)

An .xctestrun file that will override the .xctestrun file in the tests zip. Because the .xctestrun file contains environment variables along with test methods to run and/or ignore, this can be useful for sharding tests. Default is taken from the tests zip.
xcodeVersion

string

The Xcode version that should be used for the test. Use the TestEnvironmentDiscoveryService to get supported options. Defaults to the latest Xcode version Firebase Test Lab supports.
appBundleId

string

Output only. The bundle id for the application under test.
testSpecialEntitlements

boolean

The option to test special app entitlements. Setting this would re-sign the app having special entitlements with an explicit application-identifier. Currently supports testing aps-environment entitlement.

IosTestLoop

A test of an iOS application that implements one or more game loop scenarios. This test type accepts an archived application (.ipa file) and a list of integer scenarios that will be executed on the app sequentially.

JSON representation 
{
  "appIpa": {
    object (FileReference)
  },
  "scenarios": [
    integer
  ],
  "appBundleId": string
}
Fields
appIpa

object (FileReference)

Required. The .ipa of the application to test.
scenarios[]

integer

The list of scenarios that should be run during the test. Defaults to the single scenario 0 if unspecified.
appBundleId

string

Output only. The bundle id for the application under test.

IosRoboTest

A test that explores an iOS application on an iOS device.

JSON representation 
{
  "appIpa": {
    object (FileReference)
  },
  "appBundleId": string,
  "roboScript": {
    object (FileReference)
  }
}
Fields
appIpa

object (FileReference)

Required. The ipa stored at this file should be used to run the test.
appBundleId

string

The bundle ID for the app-under-test. This is determined by examining the application's "Info.plist" file.
roboScript

object (FileReference)

An optional Roboscript to customize the crawl. See https://firebase.google.com/docs/test-lab/android/robo-scripts-reference for more information about Roboscripts.

EnvironmentMatrix

The matrix of environments in which the test is to be executed.

JSON representation 
{

  // Union field environment_matrix can be only one of the following:
  "androidMatrix": {
    object (AndroidMatrix)
  },
  "androidDeviceList": {
    object (AndroidDeviceList)
  },
  "iosDeviceList": {
    object (IosDeviceList)
  }
  // End of list of possible types for union field environment_matrix.
}
Fields
Union field environment_matrix. Required. The environment matrix. environment_matrix can be only one of the following:
androidMatrix

object (AndroidMatrix)

A matrix of Android devices.
androidDeviceList

object (AndroidDeviceList)

A list of Android devices; the test will be run only on the specified devices.
iosDeviceList

object (IosDeviceList)

A list of iOS devices.

AndroidMatrix

A set of Android device configuration permutations is defined by the the cross-product of the given axes. Internally, the given AndroidMatrix will be expanded into a set of AndroidDevices.

Only supported permutations will be instantiated. Invalid permutations (e.g., incompatible models/versions) are ignored.

JSON representation 
{
  "androidModelIds": [
    string
  ],
  "androidVersionIds": [
    string
  ],
  "locales": [
    string
  ],
  "orientations": [
    string
  ]
}
Fields
androidModelIds[]

string

Required. The ids of the set of Android device to be used. Use the TestEnvironmentDiscoveryService to get supported options.
androidVersionIds[]

string

Required. The ids of the set of Android OS version to be used. Use the TestEnvironmentDiscoveryService to get supported options.
locales[]

string

Required. The set of locales the test device will enable for testing. Use the TestEnvironmentDiscoveryService to get supported options.
orientations[]

string

Required. The set of orientations to test with. Use the TestEnvironmentDiscoveryService to get supported options.

AndroidDeviceList

A list of Android device configurations in which the test is to be executed.

JSON representation 
{
  "androidDevices": [
    {
      object (AndroidDevice)
    }
  ]
}
Fields
androidDevices[]

object (AndroidDevice)

Required. A list of Android devices.

IosDeviceList

A list of iOS device configurations in which the test is to be executed.

JSON representation 
{
  "iosDevices": [
    {
      object (IosDevice)
    }
  ]
}
Fields
iosDevices[]

object (IosDevice)

Required. A list of iOS devices.

IosDevice

A single iOS device.

JSON representation 
{
  "iosModelId": string,
  "iosVersionId": string,
  "locale": string,
  "orientation": string
}
Fields
iosModelId

string

Required. The id of the iOS device to be used. Use the TestEnvironmentDiscoveryService to get supported options.
iosVersionId

string

Required. The id of the iOS major software version to be used. Use the TestEnvironmentDiscoveryService to get supported options.
locale

string

Required. The locale the test device used for testing. Use the TestEnvironmentDiscoveryService to get supported options.
orientation

string

Required. How the device is oriented during the test. Use the TestEnvironmentDiscoveryService to get supported options.

TestExecution

A single test executed in a single environment.

JSON representation 
{
  "id": string,
  "matrixId": string,
  "projectId": string,
  "testSpecification": {
    object (TestSpecification)
  },
  "shard": {
    object (Shard)
  },
  "environment": {
    object (Environment)
  },
  "state": enum (TestState),
  "toolResultsStep": {
    object (ToolResultsStep)
  },
  "timestamp": string,
  "testDetails": {
    object (TestDetails)
  }
}
Fields
id

string

Output only. Unique id set by the service.
matrixId

string

Output only. Id of the containing TestMatrix.
projectId

string

Output only. The cloud project that owns the test execution.
testSpecification

object (TestSpecification)

Output only. How to run the test.
shard

object (Shard)

Output only. Details about the shard.
environment

object (Environment)

Output only. How the host machine(s) are configured.
state

enum (TestState)

Output only. Indicates the current progress of the test execution (e.g., FINISHED).
toolResultsStep

object (ToolResultsStep)

Output only. Where the results for this execution are written.
timestamp

string (Timestamp format)

Output only. The time this test execution was initially created.

A timestamp in RFC3339 UTC "Zulu" format, with nanosecond resolution and up to nine fractional digits. Examples: "2014-10-02T15:01:23Z" and "2014-10-02T15:01:23.045123456Z".
testDetails

object (TestDetails)

Output only. Additional details about the running test.

Shard

Output only. Details about the shard.

JSON representation 
{
  "shardIndex": integer,
  "numShards": integer,
  "testTargetsForShard": {
    object (TestTargetsForShard)
  },
  "estimatedShardDuration": string
}
Fields
shardIndex

integer

Output only. The index of the shard among all the shards.
numShards

integer

Output only. The total number of shards.
testTargetsForShard

object (TestTargetsForShard)

Output only. Test targets for each shard. Only set for manual sharding.
estimatedShardDuration

string (Duration format)

Output only. The estimated shard duration based on previous test case timing records, if available.

A duration in seconds with up to nine fractional digits, ending with 's'. Example: "3.5s".

Environment

The environment in which the test is run.

JSON representation 
{

  // Union field environment can be only one of the following:
  "androidDevice": {
    object (AndroidDevice)
  },
  "iosDevice": {
    object (IosDevice)
  }
  // End of list of possible types for union field environment.
}
Fields
Union field environment. Required. The environment. environment can be only one of the following:
androidDevice

object (AndroidDevice)

An Android device which must be used with an Android test.
iosDevice

object (IosDevice)

An iOS device which must be used with an iOS test.

TestState

The state (i.e., progress) of a test execution or matrix.

Enums
TEST_STATE_UNSPECIFIED Do not use. For proto versioning only.
VALIDATING The execution or matrix is being validated.
PENDING The execution or matrix is waiting for resources to become available.
RUNNING

The execution is currently being processed.

Can only be set on an execution.
FINISHED

The execution or matrix has terminated normally.

On a matrix this means that the matrix level processing completed normally, but individual executions may be in an ERROR state.
ERROR The execution or matrix has stopped because it encountered an infrastructure failure.
UNSUPPORTED_ENVIRONMENT

The execution was not run because it corresponds to a unsupported environment.

Can only be set on an execution.
INCOMPATIBLE_ENVIRONMENT

The execution was not run because the provided inputs are incompatible with the requested environment.

Example: requested AndroidVersion is lower than APK's minSdkVersion

Can only be set on an execution.
INCOMPATIBLE_ARCHITECTURE

The execution was not run because the provided inputs are incompatible with the requested architecture.

Example: requested device does not support running the native code in the supplied APK

Can only be set on an execution.
CANCELLED

The user cancelled the execution.

Can only be set on an execution.
INVALID

The execution or matrix was not run because the provided inputs are not valid.

Examples: input file is not of the expected type, is malformed/corrupt, or was flagged as malware

ToolResultsStep

Represents a tool results step resource.

This has the results of a TestExecution.

JSON representation 
{
  "projectId": string,
  "historyId": string,
  "executionId": string,
  "stepId": string
}
Fields
projectId

string

Output only. The cloud project that owns the tool results step.
historyId

string

Output only. A tool results history ID.
executionId

string

Output only. A tool results execution ID.
stepId

string

Output only. A tool results step ID.

TestDetails

Additional details about the progress of the running test.

JSON representation 
{
  "progressMessages": [
    string
  ],
  "errorMessage": string
}
Fields
progressMessages[]

string

Output only. Human-readable, detailed descriptions of the test's progress. For example: "Provisioning a device", "Starting Test".

During the course of execution new data may be appended to the end of progressMessages.
errorMessage

string

Output only. If the TestState is ERROR, then this string will contain human-readable details about the error.

ResultStorage

Locations where the results of running the test are stored.

JSON representation 
{
  "googleCloudStorage": {
    object (GoogleCloudStorage)
  },
  "toolResultsHistory": {
    object (ToolResultsHistory)
  },
  "toolResultsExecution": {
    object (ToolResultsExecution)
  },
  "resultsUrl": string
}
Fields
googleCloudStorage

object (GoogleCloudStorage)

Required.
toolResultsHistory

object (ToolResultsHistory)

The tool results history that contains the tool results execution that results are written to.

If not provided, the service will choose an appropriate value.
toolResultsExecution

object (ToolResultsExecution)

Output only. The tool results execution that results are written to.
resultsUrl

string

Output only. URL to the results in the Firebase Web Console.

GoogleCloudStorage

A storage location within Google cloud storage (GCS).

JSON representation 
{
  "gcsPath": string
}
Fields
gcsPath

string

Required. The path to a directory in GCS that will eventually contain the results for this test. The requesting user must have write access on the bucket in the supplied path.

ToolResultsHistory

Represents a tool results history resource.

JSON representation 
{
  "projectId": string,
  "historyId": string
}
Fields
projectId

string

Required. The cloud project that owns the tool results history.
historyId

string

Required. A tool results history ID.

ToolResultsExecution

Represents a tool results execution resource.

This has the results of a TestMatrix.

JSON representation 
{
  "projectId": string,
  "historyId": string,
  "executionId": string
}
Fields
projectId

string

Output only. The cloud project that owns the tool results execution.
historyId

string

Output only. A tool results history ID.
executionId

string

Output only. A tool results execution ID.

InvalidMatrixDetails

The detailed reason that a Matrix was deemed INVALID.

Enums
INVALID_MATRIX_DETAILS_UNSPECIFIED Do not use. For proto versioning only.
DETAILS_UNAVAILABLE The matrix is INVALID, but there are no further details available.
MALFORMED_APK The input app APK could not be parsed.
MALFORMED_TEST_APK The input test APK could not be parsed.
NO_MANIFEST The AndroidManifest.xml could not be found.
NO_PACKAGE_NAME The APK manifest does not declare a package name.
INVALID_PACKAGE_NAME The APK application ID (aka package name) is invalid. See also https://developer.android.com/studio/build/application-id
TEST_SAME_AS_APP The test package and app package are the same.
NO_INSTRUMENTATION The test apk does not declare an instrumentation.
NO_SIGNATURE The input app apk does not have a signature.
INSTRUMENTATION_ORCHESTRATOR_INCOMPATIBLE The test runner class specified by user or in the test APK's manifest file is not compatible with Android Test Orchestrator. Orchestrator is only compatible with AndroidJUnitRunner version 1.1 or higher. Orchestrator can be disabled by using DO_NOT_USE_ORCHESTRATOR OrchestratorOption.
NO_TEST_RUNNER_CLASS

The test APK does not contain the test runner class specified by the user or in the manifest file. This can be caused by one of the following reasons:

  • the user provided a runner class name that's incorrect, or
  • the test runner isn't built into the test APK (might be in the app APK instead).
NO_LAUNCHER_ACTIVITY A main launcher activity could not be found.
FORBIDDEN_PERMISSIONS The app declares one or more permissions that are not allowed.
INVALID_ROBO_DIRECTIVES There is a conflict in the provided roboDirectives.
INVALID_RESOURCE_NAME There is at least one invalid resource name in the provided robo directives
INVALID_DIRECTIVE_ACTION Invalid definition of action in the robo directives (e.g. a click or ignore action includes an input text field)
TEST_LOOP_INTENT_FILTER_NOT_FOUND There is no test loop intent filter, or the one that is given is not formatted correctly.
SCENARIO_LABEL_NOT_DECLARED The request contains a scenario label that was not declared in the manifest.
SCENARIO_LABEL_MALFORMED There was an error when parsing a label's value.
SCENARIO_NOT_DECLARED The request contains a scenario number that was not declared in the manifest.
DEVICE_ADMIN_RECEIVER Device administrator applications are not allowed.
MALFORMED_XC_TEST_ZIP The zipped XCTest was malformed. The zip did not contain a single .xctestrun file and the contents of the DerivedData/Build/Products directory.
BUILT_FOR_IOS_SIMULATOR The zipped XCTest was built for the iOS simulator rather than for a physical device.
NO_TESTS_IN_XC_TEST_ZIP The .xctestrun file did not specify any test targets.
USE_DESTINATION_ARTIFACTS One or more of the test targets defined in the .xctestrun file specifies "UseDestinationArtifacts", which is disallowed.
TEST_NOT_APP_HOSTED XC tests which run on physical devices must have "IsAppHostedTestBundle" == "true" in the xctestrun file.
PLIST_CANNOT_BE_PARSED An Info.plist file in the XCTest zip could not be parsed.
TEST_ONLY_APK

The APK is marked as "testOnly". Deprecated and not currently used.
MALFORMED_IPA The input IPA could not be parsed.
MISSING_URL_SCHEME The application doesn't register the game loop URL scheme.
MALFORMED_APP_BUNDLE The iOS application bundle (.app) couldn't be processed.
NO_CODE_APK APK contains no code. See also https://developer.android.com/guide/topics/manifest/application-element.html#code
INVALID_INPUT_APK Either the provided input APK path was malformed, the APK file does not exist, or the user does not have permission to access the APK file.
INVALID_APK_PREVIEW_SDK APK is built for a preview SDK which is unsupported
MATRIX_TOO_LARGE The matrix expanded to contain too many executions.
TEST_QUOTA_EXCEEDED Not enough test quota to run the executions in this matrix.
SERVICE_NOT_ACTIVATED A required cloud service api is not activated. See: https://firebase.google.com/docs/test-lab/android/continuous#requirements
UNKNOWN_PERMISSION_ERROR There was an unknown permission issue running this test.

MatrixErrorDetail

Describes a single error or issue with a matrix.

JSON representation 
{
  "reason": string,
  "message": string
}
Fields
reason

string

Output only. The reason for the error. This is a constant value in UPPER_SNAKE_CASE that identifies the cause of the error.
message

string

Output only. A human-readable message about how the error in the TestMatrix. Expands on the reason field with additional details and possible options to fix the issue.

OutcomeSummary

Outcome summary for a finished test matrix.

Enums
OUTCOME_SUMMARY_UNSPECIFIED Do not use. For proto versioning only.
SUCCESS

The test matrix run was successful, for instance:

  • All the test cases passed.
  • Robo did not detect a crash of the application under test.
FAILURE

A run failed, for instance:

  • One or more test cases failed.
  • A test timed out.
  • The application under test crashed.
INCONCLUSIVE Something unexpected happened. The run should still be considered unsuccessful but this is likely a transient problem and re-running the test might be successful.
SKIPPED

All tests were skipped, for instance:

  • All device configurations were incompatible.

Methods

cancel

 Cancels unfinished test executions in a test matrix.

create

 Creates and runs a matrix of tests according to the given specifications.

get

 Checks the status of a test matrix and the executions once they are created.