Firebase Test Lab lets you test your iOS app on multiple devices through the gcloud command line interface. Keep reading to find out how to get started.
Create a Firebase project
If you don't have a Firebase project for your app, go to the Firebase console and click Create New Project to create one now. You will need ownership or edit permissions in your project.
You can use Test Lab for a limited number of daily test executions on the Spark plan. To use Test Lab without daily quota limits, you must upgrade to the Firebase Blaze plan.
Configure your local Google Cloud SDK environment
- Download the Google Cloud SDK.
- Make sure your installation is up-to-date and includes
gcloud components update
- Log in to the gcloud CLI using your Google account:
gcloud auth login
- Set your Firebase project in gcloud, where PROJECT_ID is
the ID of your Firebase project:
gcloud config set project PROJECT_ID
This includes the gcloud CLI tool.
Build XCTests for your app
To build your test, use the following command in the terminal:
xcodebuild -project PATH/TO/YOUR_WORKSPACE.xcodeproj \ -scheme YOUR_SCHEME \ -derivedDataPath FOLDER_WITH_TEST_OUTPUT \ -sdk iphoneos build-for-testing
xcodebuild -workspace PATH/TO/YOUR_WORKSPACE.xcworkspace \ -scheme YOUR_SCHEME \ -derivedDataPath FOLDER_WITH_TEST_OUTPUT \ -sdk iphoneos build-for-testing
After your test successfully builds, zip it for upload to Test Lab:
cd FOLDER_WITH_TEST_OUTPUT/Build/Products ; \ zip -r MyTests.zip Debug-iphoneos YOUR_SCHEME_iphoneosDEPLOYMENT_TARGET-arm64.xctestrun
Run your tests
Choose your test dimensions
Test Lab lets you run tests on a variety of iOS versions, devices, screen orientations, and
locales. These configurations are known as the test's test dimensions. To see the
options for each dimension, substitute
dimension in the following command:
gcloud firebase test ios dimension list
Screen orientation is somewhat simpler, as its only options are
Look through the list of test dimensions, and select a few combinations that you'd like to run your test on. If you're on the Spark plan, you can run up to 5 combinations per day. If you're on the Blaze plan, you can run tests on an unlimited number combinations per day, but you can only include up to 200 combinations in each run. For more information on test limits and pricing, see Pricing Plans.
Run the tests
Once you've chosen a set of test dimensions, you can have Test Lab run your tests using the
firebase test ios run command. For each combination of test dimensions you'd like to
test on, include a separate
gcloud firebase test ios run --test PATH/TO/MyTests.zip \ --device model=MODEL_ID_1,version=VERSION_ID_1,locale=LOCALE_1,orientation=ORIENTATION_1 \ --device model=MODEL_ID_2,version=VERSION_ID_2,locale=LOCALE_2,orientation=ORIENTATION_2 \ etc...
It is possible that your test will fail due to an incompatibility between the Xcode version with
which the test was built and the default Xcode version used by Test Lab. To choose the version
of Xcode for the tests, use the
gcloud firebase test ios run --test PATH/TO/MyTests.zip \ --device model=MODEL_ID_1,version=VERSION_ID_1,locale=LOCALE_1,orientation=ORIENTATION_1 \ --xcode-version=11.3
Analyze your test results
When the test is done, the gcloud tool prints a basic summary of your test results. As part of the summary, the tool includes a link to view more detailed results in the Firebase console. To learn more about how to interpret these results, see Analyzing Firebase Test Results.
Automate future tests
Scripting gcloud commands with Test Lab
You can use shell scripts or batch files to automate mobile app testing commands that you would otherwise run using the gcloud command line. This sample bash script runs an XCTest with a two-minute timeout, and reports if the test run completed successfully:
if gcloud firebase test ios run --test MyTest.zip --timeout 2m then echo "Test matrix successfully finished" else echo "Test matrix exited abnormally with non-zero exit code: " $? fi
Script exit codes
Test Lab provides several exit codes that you can use to better understand the results of tests that you run using scripts or batch files.
Scripting exit codes for Test Lab
|0||All test executions passed.|
|1||A general failure occurred. Possible causes include: a filename that does not exist or an HTTP/network error.|
|2||Testing exited because unknown commands or arguments were provided.|
|10||One or more test cases (tested classes or class methods) within a test execution did not pass.|
|15||Firebase Test Lab could not determine if the test matrix passed or failed, because of an unexpected error.|
|19||The test matrix was canceled by the user.|
|20||A test infrastructure error occurred.|