Use a TensorFlow Lite model for inference with ML Kit on Android

You can use ML Kit to perform on-device inference with a TensorFlow Lite model.

This API requires Android SDK level 16 (Jelly Bean) or newer.

See the ML Kit quickstart sample on GitHub for an example of this API in use, or try the codelab.

Before you begin

  1. If you have not already added Firebase to your app, do so by following the steps in the getting started guide.
  2. Include the dependencies for ML Kit in your app-level build.gradle file:
    dependencies {
      // ...
    
      implementation 'com.google.firebase:firebase-ml-model-interpreter:16.2.3'
    }
    
  3. Convert the TensorFlow model you want to use to TensorFlow Lite (tflite) format. See TOCO: TensorFlow Lite Optimizing Converter.

Host or bundle your model

Before you can use a TensorFlow Lite model for inference in your app, you must make the model available to ML Kit. ML Kit can use TensorFlow Lite models hosted remotely using Firebase, stored locally on the device, or both.

By both hosting the model on Firebase and storing the model locally, you can ensure that the most recent version of the model is used when it is available, but your app's ML features still work when the Firebase-hosted model isn't available.

Model security

Regardless of how you make your TensorFlow Lite models available to ML Kit, ML Kit stores them in the standard serialized protobuf format in local storage.

In theory, this means that anybody can copy your model. However, in practice, most models are so application-specific and obfuscated by optimizations that the risk is similar to that of competitors disassembling and reusing your code. Nevertheless, you should be aware of this risk before you use a custom model in your app.

On Android API level 21 (Lollipop) and newer, the model is downloaded to a directory that is excluded from automatic backup.

On Android API level 20 and older, the model is downloaded to a directory named com.google.firebase.ml.custom.models in app-private internal storage. If you enabled file backup using BackupAgent, you might choose to exclude this directory.

Host models on Firebase

To host your TensorFlow Lite model on Firebase:

  1. In the ML Kit section of the Firebase console, click the Custom tab.
  2. Click Add custom model (or Add another model).
  3. Specify a name that will be used to identify your model in your Firebase project, then upload the .tflite file.
  4. In your app's manifest, declare that INTERNET permission is required:
    <uses-permission android:name="android.permission.INTERNET" />
    

After you add a custom model to your Firebase project, you can reference the model in your apps using the name you specified. At any time, you can upload a new .tflite file for a model, and your app will download the new model and start using it when the app next restarts. You can define the device conditions required for your app to attempt to update the model (see below).

Make models available locally

To make your TensorFlow Lite model locally available, you can either bundle the model with your app, or download the model from your own server in your app.

To bundle your TensorFlow Lite model with your app, copy the .tflite file to your app's assets/ folder. (You might need to create the folder first by right-clicking the app/ folder, then clicking New > Folder > Assets Folder.)

Then, add the following to your project's build.gradle file:

android {

    // ...

    aaptOptions {
        noCompress "tflite"
    }
}

The .tflitefile will be included in the app package and available to ML Kit as a raw asset.

If you instead host the model on your own server, you can download the model to local storage at an appropriate point in your app. Then, the model will be available to ML Kit as a local file.

Load the model

To use a TensorFlow Lite model for inference, first specify the locations of the .tflite file.

If you hosted your model with Firebase, create a FirebaseCloudModelSource object, specifying the name you assigned the model when you uploaded it, and the conditions under which ML Kit should download the model initially and when an update is available.

FirebaseModelDownloadConditions.Builder conditionsBuilder =
        new FirebaseModelDownloadConditions.Builder().requireWifi();
if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.N) {
    // Enable advanced conditions on Android Nougat and newer.
    conditionsBuilder = conditionsBuilder
            .requireCharging()
            .requireDeviceIdle();
}
FirebaseModelDownloadConditions conditions = conditionsBuilder.build();

// Build a FirebaseCloudModelSource object by specifying the name you assigned the model
// when you uploaded it in the Firebase console.
FirebaseCloudModelSource cloudSource = new FirebaseCloudModelSource.Builder("my_cloud_model")
        .enableModelUpdates(true)
        .setInitialDownloadConditions(conditions)
        .setUpdatesDownloadConditions(conditions)
        .build();
FirebaseModelManager.getInstance().registerCloudModelSource(cloudSource);

If you bundled the model with your app, or downloaded the model from your own host at run time, create a FirebaseLocalModelSource object, specifying the filename of the .tflite model and whether the file is a raw asset (if bundled) or in local storage (if downloaded at run time).

FirebaseLocalModelSource localSource = new FirebaseLocalModelSource.Builder("my_local_model")
        .setAssetFilePath("mymodel.tflite")  // Or setFilePath if you downloaded from your host
        .build();
FirebaseModelManager.getInstance().registerLocalModelSource(localSource);

Then, create a FirebaseModelOptions object with the names of your Cloud source, local source, or both, and use it to get an instance of FirebaseModelInterpreter:

FirebaseModelOptions options = new FirebaseModelOptions.Builder()
        .setCloudModelName("my_cloud_model")
        .setLocalModelName("my_local_model")
        .build();
FirebaseModelInterpreter firebaseInterpreter =
        FirebaseModelInterpreter.getInstance(options);

If you specify both a Cloud model source and a local model source, the model interpreter will use the Cloud model if it's available, and fall back to the local model when it is not.

Specify the model's input and output

Next, you must specify the format of the model's input and output by creating a FirebaseModelInputOutputOptions object.

A TensorFlow Lite model takes as input and produces as output one or more multidimensional arrays. These arrays contain either byte, int, long, or float values. You must configure ML Kit with the number and dimensions ("shape") of the arrays your model uses.

For example, a quantized image classification model might take as input an Nx299x299x3 array of bytes, representing a batch of N 299x299 truecolor (24-bit) images, and produce as output a list of 1000 byte values, each representing the probability the image is a member of one of the 1000 categories the model predicts.

FirebaseModelInputOutputOptions inputOutputOptions =
    new FirebaseModelInputOutputOptions.Builder()
        .setInputFormat(0, FirebaseModelDataType.BYTE, new int[]{1, 299, 299, 3})
        .setOutputFormat(0, FirebaseModelDataType.BYTE, new int[]{1, 1000})
        .build();

Or, a floating-point image classification model might take as input an Nx299x299x3 array of floats, representing a batch of N 299x299 RGB images:

FirebaseModelInputOutputOptions inputOutputOptions =
    new FirebaseModelInputOutputOptions.Builder()
        .setInputFormat(0, FirebaseModelDataType.FLOAT32, new int[]{1, 299, 299, 3})
        .setOutputFormat(0, FirebaseModelDataType.FLOAT32, new int[]{1, 1000})
        .build();

Perform inference on input data

Finally, to perform inference using the model, get your input data (for example, by capturing an image with the device's camera):

// Quantized model:
byte[][][][] input = new byte[1][299][299][3];
input = getYourInputData();

If you are using a floating-point model with image data as input, you might need to convert the image data to floating point:

// Floating-point model:
byte[][][][] prenormalizedInput = new byte[1][299][299][3];
prenormalizedInput = getYourInputData();

float[][][][] input = new float[1][299][299][3];
for (int b = 0; b < 1; b++) {
    for (int x = 0; x < 299; x++) {
        for (int y = 0; y < 299; y++) {
            for (int ch = 0; ch < 3; ch++) {
                // Normalize channel values to [-1.0, 1.0]
                input[b][x][y][ch] =
                        (float) prenormalizedInput[b][x][y][ch] / 255.0f:
            }
        }
    }
}

Then, create a FirebaseModelInputs object with your input data, and pass it and the model's input and output specification to the model interpreter's run method:

FirebaseModelInputs inputs = new FirebaseModelInputs.Builder()
    .add(input)  // add() as many input arrays as your model requires
    .build();
Task<FirebaseModelOutputs> result =
    firebaseInterpreter.run(inputs, inputOutputOptions)
        .addOnSuccessListener(
          new OnSuccessListener<FirebaseModelOutputs>() {
            @Override
            public void onSuccess(FirebaseModelOutputs result) {
              // ...
            }
          })
        .addOnFailureListener(
          new OnFailureListener() {
            @Override
            public void onFailure(@NonNull Exception e) {
              // Task failed with an exception
              // ...
            }
          });

You can get the output by calling the getOutput() method of the object that is passed to the success listener. For example:

// Quantized model:
byte[][] output = result.<byte[][]>getOutput(0);
byte[] probabilities = output[0];

// Floating-point model:
float[][] output = result.<float[][]>getOutput(0);
float[] probabilities = output[0];

How you use the output depends on the model you are using. For example, if you are performing classification, as a next step, you might map the indexes of the result to the labels they represent.

Send feedback about...

Need help? Visit our support page.