Firebase Test Lab for Android Game Loop Test

Automating game testing is challenging because of the wide range of UI frameworks used for game development (some of which are engine-dependent), and the difficulties of automating UI navigation in games. To support game app testing, Test Lab now includes beta support for using a "demo mode" where the game app runs while simulating the actions of a player. This mode can include multiple loops (or scenarios), which can be logically organized using labels so that you can run related loops together.

This document provides guidelines on how to implement game loops so that you can easily use them to test your game during development. These guidelines apply whether you are running tests on a single test device, on a farm of test devices, or using Test Lab.

Game loop support is currently in beta release.

Get Started

To use the game loop test in Test Lab, your game must be modified to do the following:

  1. Launch the loop
  2. Run the loop
  3. (Optional) Close the game app

Launch the loop

When launching the game loop test, your game is triggered with a specific intent, so you need to modify your manifest and add a new intent filter to your activity, as shown in the following example code:

<activity android:name=".MyActivity">
   <intent-filter>
       <action android:name="com.google.intent.action.TEST_LOOP"/>
       <category android:name="android.intent.category.DEFAULT"/>
       <data android:mimeType="application/javascript"/>
   </intent-filter>
   <intent-filter>
      ... (other intent filters here)
   </intent-filter>
</activity>

Run the loop

Your activity, when launched, checks which intent launched it, as shown in the following example code:

launchIntent = getIntent();
if(launchIntent.getAction().equals("com.google.intent.action.TEST_LOOP")) {
   int scenario = launchIntent.getIntExtra("scenario", 0);
   // Code to handle your game loop here
}

We recommend running this code in your onCreate method, but you can run it later if you prefer (for example, after initially loading your game engine). You can then implement your game loop (or multiple loops, as described in Multiple game loops) however you like, depending on your game engine. For example, game loops can be used to:

  • Run a level of your game the same way an end user would play it. You can either script the input of the user, let the user idle, or replace the user with an AI if it makes sense in your game (for example, if you already have an AI implemented, like in a car racing game, and you can easily put an AI driver in charge of the user's input).
  • Run your game at the highest quality setting to see if devices support it.
  • Run a technical test (compile multiple shaders, execute them, check that the output is as expected, etc).

Test Loop Manager

To help you integrate game loops and run them on your local devices, as well as to help your Quality Assurance team run game loops on their devices, we provide an open-source app called Test Loop Manager.

To use Test Loop Manager, do the following:

  1. Download Test Loop Manager.
  2. Install Test Loop Manager on your device, using the following command:
    adb install testloopmanager.apk
  3. Open the app (named "Test Loop Apps") on your phone or tablet; you are presented with a list of apps on the device that contain game loops. If you do not see your app here, check to ensure that your intent filter matches the one described in Launch the loop.
  4. Select the game app that you want to test.
    1. After clicking the game app's name from the list, select the scenarios (i.e., the game loops) that your app implements. If your app implements multiple loops, see Multiple game loops.
    2. Click Run test, and watch as the scenarios are run.

Close the game app

At the end of your test, you should close the app using:

yourActivity.finish();

Generally the game loop test allows the UI framework to start the next loop. If you do not close the app, the UI framework running your loops won't know that the test has finished, and may terminate the app after some time has passed.

Run the game loop test in Test Lab

Go to Firebase console and create a project if you do not already have one. To use Test Lab at no charge, but with limited daily quota, use the Spark billing plan. To learn about using Test Lab with no usage limits, see Firebase Pricing Plans.

Use the Firebase console

Use the following instructions to run a game loop test using the Firebase console:

  1. In the Firebase console, click Test Lab in the left navigation panel.
  2. Click Run Your First Test (or Run a Test, if your project has previously run a test).
  3. Select Game Loop as the test type, and then click Continue.
  4. Click Browse, and then browse to your app's .apk file.
  5. (optional) To select a subset of available scenarios (game loops) do one of the following:

    • To choose specific scenarios, enter a list or range of scenario numbers in the Scenarios field.
    • To choose all scenarios with a specific label applied, enter one or more scenario labels in the Labels field.
  6. Click Continue.

  7. Select the physical devices to use to test your app.
  8. Click Start Tests.

For more information on the Firebase Console, see Use Firebase Test Lab for Android from the Firebase console.

Use the gcloud command-line environment

Use the following instructions to run a game loop test using the gcloud command-line environment (CLI):

  1. If you don't already have a Firebase project, go to the Firebase console and click Add project to create a project.
  2. Install the Google Cloud SDK, which includes the gcloud CLI.
  3. Login using a Google account:

    gcloud auth login

  4. Set your Firebase project using the following command, where PROJECT_ID is the Firebase project that you created in step 1:

    gcloud config set project PROJECT_ID
    

  5. Run your first test, as follows:

    gcloud beta firebase test android run \
    --type=game-loop --app=<path-to-apk> \
    --device model=herolte,version=23

  6. (optional) To select a subset of available scenarios (game loops) do one of the following when you run a game loop test:

    • To choose specific scenarios, specify a list of scenario numbers by using the --scenario-numbers flag (for example, --scenario-numbers=1,3,5).
    • To choose all scenarios with a specific label applied, specify one or more scenario labels by using the --scenario-labels flag (for example, --scenario-labels=performance,gpu).

For more information on using the gcloud CLI with Test Lab, see Using Firebase Test Lab for Android from the gcloud Command Line.

Optional features

This section describes how to use optional features, such as writing data to an output file, supporting multiple game loops, and labeling related loops so that you can easily test related loops in a single test matrix.

Write output data

Your game loop can write output to a file that is provided using the launchIntent.getData() method. This output data can be displayed by Test Lab on the test results page. For an example of a data output file, see Game loop test output file example.

Test Lab follows best practices for sharing a file between apps described at Sharing a File. In your Activity’s onCreate() method, where you are getting the intent, you can also check for that file using the following code:

Uri logFile = launchIntent.getData();
if (logFile != null) {
   Log.i(TAG, "Log file " + logFile.getEncodedPath());
   // ...
}
If you want to write to that file from the C++ side of your game app, you would pass the file descriptor, rather than the file path, as shown in the following example code:

Java:

Uri logFile = launchIntent.getData();
int fd = -1;
if (logFile != null) {
   Log.i(TAG, "Log file " + logFile.getEncodedPath());
   try {
       fd = getContentResolver()
               .openAssetFileDescriptor(logFile, "w")
               .getParcelFileDescriptor()
               .getFd();
   } catch (FileNotFoundException e) {
       e.printStackTrace();
       fd = -1;
   } catch (NullPointerException e) {
       e.printStackTrace();
       fd = -1;
   }
}
native_function(fd);  // C++ code invoked here.
C++:
#include <unistd.h>
JNIEXPORT void JNICALL
Java_my_package_name_MyActivity_native_function(JNIEnv *env, jclass type, jint log_file_descriptor) {
 // The file descriptor needs to be duplicated.
 int my_file_descriptor = dup(log_file_descriptor);
}

Multiple game loops

You can also support multiple game loops in your game app. For example, if you have multiple levels in your game, you might want to have one game loop to launch each level instead of having one loop that iterates through all of them.

That way, if your app crashes on level 32 you can directly launch that game loop to try and reproduce the crash and to test bug fixes.

For example if you have 5 game loops in your app, you only need to add one line to your app's manifest file, inside of the <application> element:

<application>
      …
  <meta-data android:name="com.google.test.loops" android:value="5" />
      …
</application>

Then, when you launch Test Loop Manager, can use a selection screen to select which loop to launch. If you select multiple loops to launch, it launches each loop in sequence after the preceding loop completes. With Test Lab, you select which loops to launch.

The launch intent contains the target loop as an integer parameter, in the range of 1 to the maximum number of loops supported.

You can read the extra from the intent in Java as shown in the following example code:

launchIntent = getIntent();
int scenario = launchIntent.getIntExtra("scenario", 0);

Then, you can pass this extra to your native C++ code to change the behavior of your loop based on the resulting int value.

Label game loops

You can label your game loop with one or more scenario labels to make it easy for your QA team to launch a set of related game loops (for example, "all compatibility game loops"). To do this, you need to add a meta-data line to your app's manifest file that is similar to the following example:

<meta-data android:name="com.google.test.loops.LABEL_NAME"
           android:value="1,3-5" />

You can create your own labels, and we also provide 4 predefined labels:

  • com.google.test.loops.player_experience: For loops used to reproduce a real user's experience when playing the game. The goal of testing with these loops is to find issues that a real user would face while playing the game.
  • com.google.test.loops.gpu_compatibility: For loops used to test GPU-related issues. The goal of testing with these loops is to execute GPU code that might not run properly run in production, to expose issues with hardware and drivers.
  • com.google.test.loops.compatibility: For loops used to test a broad range of compatibility issues, including I/O issues and OpenSSL issues.
  • com.google.test.loops.performance: For loops used to test the performance of the device. For example, a game might run at the most complex graphics settings to see how a new device behaves.

App licensing support

Test Lab supports apps that use the App Licensing service offered by Google Play. To successfully check licensing when testing your app with Test Lab, you must publish your app to the production channel in the Play store. To test your app in the alpha or beta channel using Test Lab, remove the licensing check before uploading your app to Test Lab.

Known issues

The Game loop test in Test Lab has the following known issues:

  • Limited support for the Khronos Vulkan API. Only the following devices available in Test Lab support the Vulkan API: Samsung Galaxy S7 (API level 24), Google Pixel (API level 25).
  • Some crashes do not support backtraces. For example, some release builds may suppress the output of the debuggerd process using prctl(PR_SET_DUMPABLE, 0). To learn more, see debuggerd.
  • API Level 19 is not currently supported due to file permission errors.

Game loop test output file example

You can use output data files formatted as shown in the example below to display game loop test results in the Firebase console. Areas shown as /.../ can contain any custom fields that you need, as long as they don't conflict with with the names of other fields used in this file:

{
  "name": "test name",
  "start_timestamp": 0, // Timestamp of the test start (in us).
                           Can be absolute or relative
  "driver_info": "...",
  "frame_stats": [
    {
      "timestamp": 1200000, // Timestamp at which this section was written
                               It contains value regarding the period
                               start_timestamp(0) -> this timestamp (1200000 us)
      "avg_frame_time": 15320, // Average time to render a frame in ns
      "nb_swap": 52, // Number of frame rendered
      "threads": [
        {
          "name": "physics",
          "Avg_time": 8030 // Average time spent in this thread per frame in us
        },
        {
          "name": "AI",
          "Avg_time": 2030 // Average time spent in this thread per frame in us
        }
      ],
      /.../ // Any custom field you want (vertices display on the screen, nb units …)
    },
    {
      // Next frame data here, same format as above
    }
  ],
  "loading_stats": [
    {
      "name": "assets_level_1",
      "total_time": 7850, // in us
      /.../
    },
    {
      "name": "victory_screen",
      "total_time": 554, // in us
      /.../
    }

  ],
  /.../, // You can add custom fields here
}

Send feedback about...

Need help? Visit our support page.