Build hybrid experiences in Web apps with on-device and cloud-hosted models

You can build AI-powered Web apps and features with hybrid inference using Firebase AI Logic. Hybrid inference enables running inference using on-device models when available and seamlessly falling back to cloud-hosted models otherwise (and vice versa).

This page describes how to get started using the client SDK. After completing this standard setup, check out the additional configuration options and capabilities.

Note that on-device inference via Firebase AI Logic is supported for web apps using Firebase AI Logic SDK v12.5.0+ and running on Chrome on Desktop (version 139 or higher).

arrow_downward Jump to view code examples

Use cases and supported capabilities

Depending on your app's use cases, you can take advantage of several benefits of on-device or hybrid experiences:

• Benefits of using an on-device model for inference:

  • Enhanced privacy
  • Local context
  • Inference at no-cost
  • Offline functionality

• Benefits of using a hybrid approach:

  • Provide a similar app experience, regardless of the end-user's device
  • Improve availability of generative AI features, regardless of internet connectivity, quota limitations, or device capabilities

Supported on-device inference capabilities

On-device inference only supports single-turn text generation (not chat) (with streaming or non-streaming output), specifically the following capabilities:

• Generating text from text-only input.

• Generating text from text-and-image input, specifically input image types of JPEG and PNG.

• Generating structured output, including JSON and enums.

Make sure to review the list of not-yet-supported features for hybrid or on-device inference at the bottom of this page.

Supported APIs and platforms

The following sections describe the supported APIs, as well as the supported platforms and browsers.

Supported APIs for in-cloud vs on-device inference

• In-cloud inference uses your chosen Gemini API provider (either the Gemini Developer API or the Agent Platform Gemini API (formerly Vertex AI)).

• On-device inference uses the Prompt API from Chrome.

Supported platforms and browsers

For on-device inference (which uses the Prompt API from Chrome), the end-user of your app must be running Chrome on Desktop (version 139 or higher). Learn more in the Chrome documentation.

Get started on localhost

These get started steps describe the required general setup for any supported prompt request that you want to send.

You can learn more about using APIs on localhost in the Chrome documentation.

Step 1: Set up Chrome and the Prompt API for on-device inference

1. Make sure you're using a recent version of Chrome. Update in chrome://settings/help.
   On-device inference is available from Chrome v139 and higher.

2. Enable the on-device multimodal model by setting the following flag to Enabled:

   • chrome://flags/#prompt-api-for-gemini-nano-multimodal-input

3. Restart Chrome.

4. (Optional) Download the on-device model before the first request.

   The Prompt API is built into Chrome; however, the on-device model isn't available by default. If you haven't yet downloaded the model before your first request for on-device inference, the request will automatically start the model download in the background.

   View instructions to download the on-device model

   1. Open Developer Tools > Console.

   2. Run the following:

      await LanguageModel.availability();
      

   3. Make sure that the output is available, downloading, or downloadable.

   4. If the output is downloadable, start the model download by running:

      await LanguageModel.create();
      

   5. You can use the following monitor callback to listen for download progress and make sure that the model is available before making requests:

      const session = await LanguageModel.create({
        monitor(m) {
          m.addEventListener("downloadprogress", (e) => {
            console.log(`Downloaded ${e.loaded * 100}%`);
          });
        },
      });
      

Step 2: Set up a Firebase project and connect your app to Firebase

1. Sign into the Firebase console, and then select your Firebase project.

   Don't already have a Firebase project?

   If you don't already have a Firebase project, click the button to create a new Firebase project, and then use either of the following options:

   • Option 1: Create a wholly new Firebase project (and its underlying Google Cloud project automatically) by entering a new project name in the first step of the workflow.

   • Option 2: "Add Firebase" to an existing Google Cloud project by clicking Add Firebase to Google Cloud project (at bottom of page). In the first step of the workflow, start entering the project name of the existing project, and then select the project from the displayed list.

   Complete the remaining steps of the on-screen workflow to create a Firebase project. Note that when prompted, you do not need to set up Google Analytics to use the Firebase AI Logic SDKs.

2. In the Firebase console, go to AI Services > AI Logic.

3. Click Get started to launch a guided workflow that helps you set up the required APIs and resources for your project.

4. If prompted, follow the on-screen instructions to register your app and add your Firebase configuration to your app.

5. When asked to choose a "Gemini API provider", we recommend selecting the Gemini Developer API, which lets you get started quickly at no cost.

   At any point later, you can always set up the Agent Platform Gemini API (formerly Vertex AI) (and its requirement for billing).

6. Continue in the workflow to set up the required APIs and associated services for Firebase AI Logic.

   This stage of the workflow automatically enforces Firebase App Check for AI Logic, which is a critical service to help protect the Gemini API when it's directly accessed from your app. As part of getting started (see steps later in this guide), you'll need to configure the App Check debug provider for local development when App Check is enforced.

7. Continue to the next step in this guide to add the required SDKs to your app.

Step 3: Add the SDK

The Firebase library provides access to the APIs for interacting with generative models. The library is included as part of the Firebase JavaScript SDK for Web.

1. Install the Firebase JS SDK for Web using npm:

   npm install firebase
   

2. Initialize Firebase in your app:

   import { initializeApp } from "firebase/app";
   import { initializeAppCheck } from "firebase/app-check";
   
   // TODO(developer) Replace the following with your app's Firebase configuration
   // See: https://firebase.google.com/docs/web/learn-more#config-object
   const firebaseConfig = {
     // ...
   };
   
   // Initialize FirebaseApp
   const firebaseApp = initializeApp(firebaseConfig);
   

Step 4: Configure the App Check debug provider for local development

As part of the guided setup workflow for AI Logic in the console, Firebase App Check is automatically enforced to protect the Gemini API. For local development, you need to configure the App Check debug provider to bypass attestation while still maintaining the enforcement of App Check.

Here's how to use the debug provider while running your app from localhost interactively (for example, during local development):

1. In your debug build, enable debug mode by setting self.FIREBASE_APPCHECK_DEBUG_TOKEN to true before you initialize App Check. For example:

   self.FIREBASE_APPCHECK_DEBUG_TOKEN = true;
   initializeAppCheck(firebaseApp, { /* App Check options */ });
   

2. Visit your web app locally and open the browser's developer tools. In the debug console, you'll see a debug token:

   AppCheck debug token: "123a4567-b89c-12d3-e456-789012345678".
   You will need to safelist it in the Firebase console for it to work.
   

3. Register your debug token with App Check:

   1. In the Firebase console, go to the Security > App Check > Apps tab.

   2. Find your app, click the overflow menu (more_vert), and then select Manage debug tokens.

   3. Follow the on-screen instructions to register your debug token.

For details about the debug provider (including how to get a new debug token), check out the official App Check docs.

Step 5: Initialize the service and create a model instance

Set up the following before you send a prompt request to the model:

1. Initialize the service for your chosen API provider.

2. Create a GenerativeModel instance. Make sure to set the mode to one of:

   • PREFER_ON_DEVICE: Use the on-device model if it's available; otherwise, fall back to the cloud-hosted model.

   • ONLY_ON_DEVICE: Use the on-device model if it's available; otherwise, throw an exception.

   • PREFER_IN_CLOUD: Use the cloud-hosted model if it's available; otherwise, fall back to the on-device model.

   • ONLY_IN_CLOUD: Use the cloud-hosted model if it's available; otherwise, throw an exception.

import { initializeApp } from "firebase/app";
import { getAI, getGenerativeModel, GoogleAIBackend, InferenceMode } from "firebase/ai";

// TODO(developer) Replace the following with your app's Firebase configuration
// See: https://firebase.google.com/docs/web/learn-more#config-object
const firebaseConfig = {
  // ...
};

// Initialize FirebaseApp
const firebaseApp = initializeApp(firebaseConfig);

// Initialize the Gemini Developer API backend service
const ai = getAI(firebaseApp, { backend: new GoogleAIBackend() });

// Create a `GenerativeModel` instance
// Set the mode (for example, use the on-device model if it's available)
const model = getGenerativeModel(ai, { mode: InferenceMode.PREFER_ON_DEVICE });

Step 6: Initialize the on-device model

You must call initializeDeviceModel() after or on an end-user page interaction (like a button click) and before you send a prompt request to the model. Learn more about the user activation requirement in the Chrome documentation.

import { initializeApp } from "firebase/app";
import { getAI, getGenerativeModel, GoogleAIBackend, InferenceMode } from "firebase/ai";

// TODO(developer) Replace the following with your app's Firebase configuration
// See: https://firebase.google.com/docs/web/learn-more#config-object
const firebaseConfig = {
  // ...
};

// Initialize FirebaseApp
const firebaseApp = initializeApp(firebaseConfig);

// Initialize the Gemini Developer API backend service
const ai = getAI(firebaseApp, { backend: new GoogleAIBackend() });

// Create a `GenerativeModel` instance
// Set the mode (for example, use the on-device model if it's available)
const model = getGenerativeModel(ai, { mode: InferenceMode.PREFER_ON_DEVICE });

// `initializeDeviceModel` must be called:
// (1) after or on an end-user page interaction such as a button click
// and
// (2) before any queries to the model (such as `generateContent()`)
// You may want to `await` this promise if using `ONLY_ON_DEVICE` (see note below).
model.initializeDeviceModel((val) =>
  // Example: "Download progress: 72.62%""
  console.log(`Download progress: ${Math.round(val*10000) / 100}%`)
);

Step 7: Send a prompt request to a model

This section shows you how to send various types of input to generate different types of output, including:

• Generate text from text-only input
• Generate text from text-and-image (multimodal) input

If you want to generate structured output (like JSON or enums), then use one of the following "generate text" examples and additionally configure the model to respond according to a provided schema.

Generate text from text-only input

You can use generateContent() to generate text from a prompt that contains text:

// Imports + initialization of FirebaseApp and backend service + creation of model instance

// Wrap in an async function so you can use await
async function run() {
  // Provide a prompt that contains text
  const prompt = "Write a story about a magic backpack."

  // To generate text output, call `generateContent` with the text input
  const result = await model.generateContent(prompt);

  const response = result.response;
  const text = response.text();
  console.log(text);
}

run();

Note that Firebase AI Logic also supports streaming of text responses using generateContentStream (instead of generateContent).

Generate text from text-and-image (multimodal) input

You can use generateContent() to generate text from a prompt that contains text and image files—providing each input file's mimeType and the file itself.

The supported input image types for on-device inference are PNG and JPEG.

// Imports + initialization of FirebaseApp and backend service + creation of model instance

// Converts a File object to a Part object.
async function fileToGenerativePart(file) {
  const base64EncodedDataPromise = new Promise((resolve) => {
    const reader = new FileReader();
    reader.onloadend = () => resolve(reader.result.split(',')[1]);
    reader.readAsDataURL(file);
  });
  return {
    type: "inlineData",
    inlineData: { data: await base64EncodedDataPromise, mimeType: file.type },
  };
}

async function run() {
  // Provide a text prompt to include with the image
  const prompt = "Write a poem about this picture:";

  const fileInputEl = document.querySelector("input[type=file]");
  const imagePart = await fileToGenerativePart(fileInputEl.files[0]);

  // To generate text output, call `generateContent` with the text and image
  const result = await model.generateContent([prompt, imagePart]);

  const response = result.response;
  const text = response.text();
  console.log(text);
}

run();

Note that Firebase AI Logic also supports streaming of text responses using generateContentStream (instead of generateContent).

What else can you do?

You can use various additional configuration options and capabilities for your hybrid experiences:

• Set an inference mode.

• Determine whether on-device or in-cloud inference was used.

• Override the default cloud-hosted fallback model.

• Use model configuration to control responses.

• Generate structured output (like JSON and enums).

Features not yet supported

Not all the capabilities of the Web SDK or cloud-hosted models are available for hybrid or on-device inference.

Features not yet supported for hybrid implementations

The following features cannot be used with hybrid implementations (neither on-device nor via cloud fallback):

• Gemini Live API: Bi-directional real-time streaming is not supported.

• Server prompt templates: Managed server prompt templates are not supported.

• Count tokens: Count tokens shouldn't be relied upon because the count will differ between cloud-hosted and on-device models, so there's no intuitive fallback.

Features not yet supported for on-device inference

The following features are not yet supported for on-device inference, but they are available for cloud-hosted models.

• Generating images using Gemini Nano Banana models

  • Can fall back to the cloud-hosted model; however, ONLY_ON_DEVICE mode will throw an error.

• Generating text from image file input types other than JPEG and PNG

  • Can fall back to the cloud-hosted model; however, ONLY_ON_DEVICE mode will throw an error.

• Generating text from multimodal inputs of audio, video, and documents (PDFs)

  • Can fall back to the cloud-hosted model; however, ONLY_ON_DEVICE mode will throw an error.

• Providing files using URLs in multimodal requests. You must provide files as inline data to on-device models.

• Multi-turn chat

  • Can fall back to the cloud-hosted model; however, ONLY_ON_DEVICE mode will throw an error.

• Providing the model with tools to help it generate its response (like function calling, code execution, URL context, Grounding with Google Search, and Grounding with Google Maps)

• AI monitoring in the Firebase console does not show any data for on-device inference (including on-device logs). However, any inference that uses a cloud-hosted model can be monitored just like other inference via Firebase AI Logic.

Additional limitations

In addition to the above, on-device inference has the following limitations:

• The end-user of your app must be running Chrome on Desktop (version 139 or higher) with the required flags enabled.

• The on-device model requires sufficient storage space on the user's device for the initial model download.

Give feedback about your experience with Firebase AI Logic