Build AI-powered web apps with Firebase Extensions for the Gemini API

1. Before you begin

Firebase Extensions allow you to add pre-packaged functionality to your apps with minimal code – even AI-powered functionality. This codelab shows you how to integrate two Firebase Extensions in a web app so that you can leverage the Gemini API to generate image descriptions, summarizations, and even personalized recommendations based on provided context and end user input.

In this codelab, you'll learn how to build an AI-powered web app that provides compelling user experiences with Firebase Extensions.

Prerequisites

• Knowledge of Node.js, Next.js, and TypeScript.

What you'll learn

• How to use Firebase Extensions for the Gemini API to process language.
• How to use Cloud Functions for Firebase to compose an augmented context for your language model.
• How to use JavaScript to access output produced by Firebase Extensions.

What you'll need

• A browser of your choice, such as Google Chrome
• A development environment with a code editor and terminal
• A Google Account for the creation and management of your Firebase project

2. Review the web app, Firebase services, and extensions

In this section, you'll review the web app that you'll build with this codelab, as well learn about the Firebase services and Firebase Extensions that you'll use.

Web app

In this codelab, you'll build a web app called Friendly Conf.

The Friendly Conference staff decided to use AI to produce a delightful and personalized user experience for their attendees. The completed conference app provides attendees with a conversational AI chatbot that's powered by a multimodal generative AI model (also known as a large language model or LLM), and it can answer questions about general topics, tailored to the conference schedule and topics. The chatbot has historical context and knowledge about the current date/time and the Friendly Conf topics and schedule, so its responses can factor in all this context.

Firebase services

In this codelab, you'll use many Firebase services and features, and most of the starter code for them is provided for you. The following table contains the services that you'll use and the reasons for using them.

Firebase Extensions

The Firebase Extensions that you'll use in this codelab include the following:

• Build Chatbot with the Gemini API
• Multimodal Tasks with the Gemini API

Extensions are useful because they react to events that happen in your Firebase project. Both of the extensions used in this codelab respond when you create new documents in preconfigured collections in Cloud Firestore.

3. Set up your development environment

Verify your Node.js version

1. In your terminal, verify that you have Node.js version 20.0.0 or higher installed:

   node -v
   

2. If you don't have Node.js version 20.0.0 or higher, download the latest LTS version and install it.

Get the source code for the codelab

If you have a GitHub account:

1. Create a new repository using our template from github.com/FirebaseExtended/codelab-gemini-api-extensions
2. Clone the codelab's GitHub repository you just created:

   git clone https://github.com/<your-github-handle>/codelab-gemini-api-extensions
   

If you don't have git installed or prefer to not create a new repo:

Download the GitHub repository as a zip file.

Review the folder structure

The root folder includes a README.md file that offers a quick start to run the web app, using streamlined instructions. However, if you're a first-time learner, you should complete this codelab (instead of the quickstart) because the codelab contains the most comprehensive set of instructions.

If you're unsure of whether you correctly applied code as instructed throughout this codelab, you can find the solution code in the end git branch.

Install the Firebase CLI

1. Verify that you have the Firebase CLI installed and that it's version 13.6 or higher:

   firebase --version
   

2. If you have the Firebase CLI installed, but it's not version 13.6 or higher, update it:

   npm update -g firebase-tools
   

3. If you don't have the Firebase CLI installed, install it:

   npm install -g firebase-tools
   

If you're unable to update or install the Firebase CLI because of permission errors, see the npm documentation or use another installation option.

Log into Firebase

1. In your terminal, navigate to the codelab-gemini-api-extensions folder and log in to Firebase:

   cd codelab-gemini-api-extensions
   firebase login
   

   If your terminal says that you're already logged in to Firebase, you can skip to the Set up your Firebase project section of this codelab.
2. In your terminal, depending on whether you want Firebase to collect data, enter Y or N. (either option works for this codelab)
3. In your browser, select your Google Account and click Allow.

4. Set up your Firebase project

In this section, you'll set up a Firebase project and register a Firebase Web App in it. You'll also enable a few Firebase services used by the sample web app later in this codelab.

All of the steps in this section are performed in the Firebase console.

Create a Firebase project

1. Sign into the Firebase console using a Google Account.
2. Click Create a project, and then enter a project name (for example, AI Extensions Codelab).
   Remember the auto-assigned project ID for your Firebase project (or click the Edit icon to set your preferred project ID). You'll need this ID later to identify your Firebase project in the Firebase CLI. If you forget your ID, you can always find it later in the Project Settings.
3. Click Continue.
4. If prompted, review and accept the Firebase terms, and then click Continue.
5. For this codelab, you do not need Google Analytics, so toggle off the Google Analytics option.
6. Click Create project, wait for your project to provision, and then click Continue.

Upgrade your Firebase pricing plan

To use Firebase Extensions (and their underlying cloud services) as well as Cloud Storage for Firebase, your Firebase project needs to be on the pay-as-you go (Blaze) pricing plan, which means it's linked to a Cloud Billing account.

• A Cloud Billing account requires a payment method, like a credit card.
• If you're new to Firebase and Google Cloud, check if you're eligible for a $300 credit and a Free Trial Cloud Billing account.
• If you're doing this codelab as part of an event, ask your organizer if there are any Cloud credits available.

Also note that with billing enabled on your Firebase project, you'll be charged for the calls that the extension makes to the Gemini API (no matter which provider you choose, Google AI or Vertex AI). Learn more about pricing for Google AI and for Vertex AI.

To upgrade your project to the Blaze plan, follow these steps:

1. In the Firebase console, select to upgrade your plan.
2. Select the Blaze plan. Follow the on-screen instructions to link a Cloud Billing account to your project.
   If you needed to create a Cloud Billing account as part of this upgrade, you might need to navigate back to the upgrade flow in the Firebase console to complete the upgrade.

Add a web app to your Firebase project

1. Navigate to the Project Overview screen in your Firebase project, and then click Web.
2. In the App nickname text box, enter a memorable app nickname, such as My AI Extensions
3. Click Register app > Next > Next > Continue to console.
   You can skip all the steps related to "hosting" in the Web App flow because you'll be optionally setting up a hosting service later in this codelab.

Great! You have now registered a web app in your new Firebase project.

Set up Firebase Authentication

1. Navigate to Authentication using the left-navigation pane.
2. Click Get started.
3. In the Additional providers column, click Google > Enable.
4. In the Public-facing name for project text box, enter a useful name, such as My AI Extensions Codelab.
5. In the Support email for project menu, select your email address.
6. Click Save.

Set up Cloud Firestore

1. In the left-panel of the Firebase console, expand Build and then select Firestore database.
2. Click Create database.
3. Leave the Database ID set to (default).
4. Select a location for your database, then click Next.
   For a real app, you want to choose a location that's close to your users.
5. Click Start in test mode. Read the disclaimer about the security rules.
   Later in this codelab, you'll add Security Rules to secure your data. Do not distribute or expose an app publicly without adding Security Rules for your database.
6. Click Create.

Set up Cloud Storage for Firebase

1. In the left-panel of the Firebase console, expand Build and then select Storage.
2. Click Get started.
3. Select a location for your default Storage bucket.
   Buckets in US-WEST1, US-CENTRAL1, and US-EAST1 can take advantage of the "Always Free" tier for Google Cloud Storage. Buckets in all other locations follow Google Cloud Storage pricing and usage.
4. Click Start in test mode. Read the disclaimer about the security rules.
   Later in this codelab, you'll add security rules to secure your data. Do not distribute or expose an app publicly without adding Security Rules for your Storage bucket.
5. Click Create.

In the next section of this codelab, you'll install and configure the two Firebase Extensions that you'll use in the web app throughout this codelab.

5. Set up the "Build Chatbot with the Gemini API" extension

Install the "Build Chatbot with the Gemini API" extension

1. Navigate to the "Build Chatbot with the Gemini API" extension".
2. Click Install in the Firebase console.
3. Select your Firebase project, and then click Next.
4. In the Review APIs enabled and resources created section, click Enable next to any services that are suggested to you, and then click Next.
5. For any permissions that are suggested to you, select Grant, and then click Next.
6. Configure the extension:
   1. In the Gemini API Provider menu, select whether you want to use the Gemini API from Google AI or Vertex AI. For developers using Firebase, we recommend using Vertex AI.
   2. In the Firestore Collection Path text box, enter: users/{uid}/messages.
      In future steps of this codelab, adding documents to this collection will trigger the extension to call the Gemini API.
   3. In the Cloud Functions location menu, select your preferred location (like Iowa (us-central1) or the location that you previously specified for your Firestore database).
   4. Leave all other values as their default.
7. Click Install extension and wait for the extension to install.

Try out the "Build Chatbot with the Gemini API" extension

While the goal of this codelab is to interact with the "Build Chatbot with the Gemini API" extension through a web app, it's helpful to learn how the extension works by trying it out in the Firebase console first.

The extension is triggered whenever a Firestore document is created under the collection users/{uid}/discussion/{discussionId}/messages, which you can do in the Firebase console.

1. In the Firebase console, navigate to Firestore, and then click Start collection within the first column.
2. In the Collection ID text box, enter users, and then click Next.
3. In the Document ID text box, click Auto-ID, and then click Save.
4. In the users collection, click Start collection.
5. In the Collection ID text box, enter messages, and then click Next.
   1. In the Document ID text box, click Auto-ID.
   2. In the Field text box, enter prompt
   3. In the Value text box, enter Tell me 5 random fruits
6. Click Save and wait a few seconds.

When you added this document, it triggered the extension to call the Gemini API. The document that you just added to the messages collection now includes not only your prompt but also now the model's response to your query.

Trigger the extension again by adding another document to the messages collection:

1. In the messages collection, click Add document.
2. In the Document ID text box, click Auto-ID.
3. In the Field text box, enter prompt
4. In the Value text box, enter And now, vegetables
5. Click Save and wait a few seconds. The document you just added to the messages collection now includes a response to your query.
   
   When generating this response, the underlying Gemini model used the historical knowledge from your previous query.

6. Set up the web app

To run the web app, you'll need to run commands in your terminal and add code within your code editor.

Set up the Firebase CLI to run against your Firebase project

In your terminal, tell the CLI to use your Firebase project by running the following command:

firebase use YOUR_PROJECT_ID

Deploy Security Rules for Firestore and Cloud Storage

This codelab's codebase already has a set of Firestore Security Rules and Cloud Storage Security Rules written for you. Once you deploy these Security Rules, your Firebase services in your Firebase project are better protected from misuse.

1. To deploy Security Rules, run this command in your terminal:

   firebase deploy --only firestore:rules,storage
   

2. If asked whether to grant Cloud Storage the IAM Role to use cross-service rules, enter Y or N. (either option works for this codelab)

Connect your web app to your Firebase project

Your web app's codebase needs to know which Firebase project it should use for its database, storage, etc. You do this by adding your Firebase configuration to your app's codebase.

1. Obtain your Firebase configuration:
   1. In the Firebase console, navigate to Project settings within your Firebase project.
   2. Scroll down to the Your apps section, and select your registered web app.
   3. In the SDK setup and configuration pane, copy the full initializeApp code including the firebaseConfig const.
2. Add your Firebase configuration to your web app's codebase:
   1. In your code editor, open the src/app/lib/firebase/firebase.config.js file.
   2. Select all in the file, and replace with the code that you copied.
   3. Save the file.

Preview the web app in your browser

1. In your terminal, install dependencies and then run the web app:

   npm install
   npm run dev
   

2. In your browser, navigate to the locally hosted Hosting URL to view the web app. For example, in most cases, the URL is http://localhost:3000/ or something similar.

Use the web app's chatbot

1. In your browser, return to the tab with the locally running Friendly Conf web app.
2. Click Sign in with Google, and if necessary, select your Google Account.
3. After you have signed in, you'll see an empty chat window.
4. Type in a greeting (like hi), and then click Send.
5. Wait a few seconds for the chatbot to respond.

The chatbot in the app responds with a generic response.

Specialize the chatbot for the app

You need the underlying Gemini model used by your web app's chatbot to know conference-specific details when the model generates responses for attendees using the app. There are lots of ways to control and steer these responses, and in this codelab's sub-section, we're showing you a very basic way by providing "context" in the initial prompt (rather than only the input from the user of the web app).

1. In the web app in your browser, clear the conversation by clicking the red "x" button (next to the message in the chat history).
2. In your code editor, open the src/app/page.tsx file.
3. Scroll down and replace the code on or near line 77 that says prompt: userMsg with the following:
   prompt: preparePrompt(userMsg, messages),
4. Save the file.
5. Go back to the web app running in your browser.
6. Again, type in a greeting (like hi), and then click Send.
7. Wait a few seconds for the chatbot to respond.

The chatbot responds with knowledge guided by the context provided in src/app/lib/context.md. Even though you didn't type in a specific request, the underlying Gemini model generates a personalized recommendation based on this context as well as the current date/time. Now you can specify follow-up questions and dive deeper.

This expanded context is important for the chatbot, but you shouldn't show it to the user of the web app. Here's how to hide it:

1. In your code editor, open the src/app/page.tsx file.
2. Scroll down and replace the code on or near line 56 that says ...doc.data(), with the following:
   ...prepareMessage(doc.data()),
3. Save the file.
4. Go back to the web app running in your browser.
5. Reload the page.

You can also try out the ability to converse with the chatbot with historical context:

1. In the Type a message text box, ask a question like: Any other interesting talks about AI?The chatbot will return a response.
2. In the Type a message text box, ask a follow-up question that's related to the previous question: Give me a few more details about the last one.

The chatbot responds with historical knowledge. Since the chat history is now part of the context, the chatbot understands follow-up questions.

Bonus: Run the web app using the Firebase Local Emulator Suite

The Firebase Local Emulator Suite allows you to test locally most of the features of the web app.

1. In your terminal, make sure that you're at the root of the web app.
2. Run the following command to install and then run the Firebase Local Emulator Suite:

   firebase init emulators
   firebase emulators:start
   

7. Set up the "Multimodal Tasks with the Gemini API" extension

The "Multimodal Tasks with the Gemini API" extension calls the Gemini API with multimodal prompts that contain a text prompt as well as a supported file URL or Cloud Storage URL (note that even the Google AI Gemini API uses a Cloud Storage URL as its underlying file URL infrastructure). The extension also supports handlebars variables to substitute values from the Cloud Firestore document for customizing the text prompt.

In your app, whenever you upload an image to a Cloud Storage bucket, you can generate a URL and add that URL to a new Cloud Firestore document – thus triggering the extension to create a multimodal prompt and call the Gemini API. In this codelab's source code, we've already provided the code for uploading an image and writing the URL to a Firestore document.

Install the "Multimodal Tasks with the Gemini API" extension

1. Navigate to the "Multimodal Tasks with the Gemini API" extension".
2. Click Install in the Firebase console.
3. Select your Firebase project.
4. Click Next > Next > Next, until you reach the Configure extension section.
   1. In the Gemini API Provider menu, select whether you want to use the Gemini API from Google AI or Vertex AI. For developers using Firebase, we recommend using Vertex AI.
   2. In the Firestore Collection Path text box, enter: gallery
   3. In the Prompt text box, enter: Please describe the provided image; if there is no image, say "no image"
   4. In the Image field text box, enter: image
   5. In the Cloud Functions location menu, select your preferred location (like Iowa (us-central1) or the location that you previously specified for your Firestore database).
   6. Leave all other values as their default.
5. Click Install extension and wait for the extension to install.

Try out the "Multimodal Tasks with the Gemini API" extension

While the goal of this codelab is to interact with the "Multimodal Tasks with the Gemini API" extension through a web app, it's helpful to learn how the extension works by trying it out in the Firebase console first.

The extension is triggered whenever a Firestore document is created under the collection users/{uid}/gallery, which you can do in the Firebase console. The extension then takes the Cloud Storage image URL in the Cloud Firestore document and passes it as part of the multimodal prompt in a call to the Gemini API.

First, upload an image to a Cloud Storage bucket:

1. Navigate to Storage within your Firebase project.
2. Click Create folder.
3. In the Folder name text box, enter gallery
4. Click Add folder.
5. In the gallery folder, click Upload file.
6. Select a JPEG image file to upload.

Next, add the Cloud Storage URL for the image to a Firestore document (which is the trigger for the extension):

1. Navigate to Firestore within your Firebase project
2. Click Start collection within the first column.
3. In the Collection ID text box, enter: gallery, and then click Next.
4. Add a document to the collection:
   1. In the Document ID text box, click Auto-ID.
   2. In the Field text box, enter: image. In the Value box enter the Storage location uri of the image you just uploaded.
5. Click Add Field.
6. In the Field text box, enter: published. In the Type box select boolean. In the Value box select true.
7. Click Save and wait a few seconds.

The gallery collection now includes a document that contains a response to your query.

Use the web app's image gallery

1. In your browser, return to the tab with the locally running Friendly Conf web app.
2. Click on the Gallery navigation tab.
3. You will see a gallery of uploaded images and AI generated descriptions. It should have the image you uploaded earlier to the gallery folder in your Storage bucket.
4. Click the button to "Upload" and select another JPEG image.
5. Wait a few seconds for the image to show in the gallery. A few moments later, the AI generated description for the newly uploaded image will be displayed, as well.

If you want to understand the code for how this was implemented, see src/app/gallery/page.tsx in the web app's codebase.

8. Bonus: Deploy your application

Firebase offers several ways in which you can deploy a web application. For this codelab, choose one of the following options:

• Option 1: Firebase Hosting - Use this option if you decide to not create your own GitHub repo (and only have your source code stored locally on your machine).
• Option 2: Firebase App Hosting - Use this option if you want automatic deployment anytime you push changes to your own GitHub repo. This new Firebase service is built specifically to tailor to the needs of dynamic Next.js and Angular applications.

Option 1: Deploy using Firebase Hosting

Use this option if you decide to not create your own GitHub repo (and only have your source code stored locally on your machine).

1. In your terminal, initialize Firebase Hosting by running these commands:

   firebase experiments:enable webframeworks
   firebase init hosting
   

2. For the prompt: Detected an existing Next.js codebase in your current directory, should we use this?, press Y.
3. For the prompt: In which region would you like to host server-side content, if applicable?, select either the default location or the location that you used earlier in this codelab. Then press Enter (or return on macOS).
4. For the prompt: Set up automatic builds and deploys with GitHub?, press N.
5. Deploy your web app to Hosting by running this command:

   firebase deploy --only hosting
   

All done! If you update your app and want to deploy that new version, just rerun firebase deploy --only hosting and Firebase Hosting will build and re-deploy your app.

Option 2: Deploy using Firebase App Hosting

Use this option if you want automatic deployment anytime you push changes to your own GitHub repo.

1. Commit your changes to GitHub.
2. In the Firebase console, navigate to App Hosting within your Firebase project.
3. Click Get started > Connect to GitHub.
4. Select your GitHub account and Repository. Click Next.
5. In Deployment setting > Root directory, enter the name of the folder with your source code (if your package.json is not in the root directory of your repository).
6. For the Live branch, select the main branch of your GitHub repo. Click Next.
7. Enter an ID for your backend (for example, chatbot).
8. Click Finish and Deploy.

It will take a few minutes for your new deployment to be ready. You can check the deployment status in the App Hosting section of the Firebase console.

From this point forward, each time that you push a change to your GitHub repo, Firebase App Hosting will automatically build and deploy your app.

9. Conclusion

Congratulations! You achieved a lot in this codelab!

Installing and configuring extensions

You used the Firebase console to configure and install various Firebase Extensions that use generative AI. Using Firebase Extensions is convenient because you don't need to learn about and write lots of boilerplate code to handle authenticating with Google Cloud services or the backend Cloud Functions logic to listen to and interact with Firestore and Google Cloud services and APIs.

Trying out extensions using the Firebase console

Instead of jumping straight into code, you took time to understand how these genAI extensions work, based on an input that you provided through Firestore or Cloud Storage. This can be especially useful while debugging the output of an extension.

Building an AI-powered web app

You built an AI-powered web app that uses Firebase Extensions to access just a few capabilities of the Gemini model.

In the web app, you use the "Chatbot with Gemini API" extension to provide the user with an interactive chat interface, that includes app-specific and historical context in conversations - where each message is stored in a Firestore document that is scoped to a particular user.

The web app also used the "Multimodal Tasks with the Gemini API" extension to auto-generate image descriptions for uploaded images.

Next steps

• Experiment with prompts and take advantage of the large context window in Google AI Studio or Vertex AI Studio.
• Learn about AI retrieval augmented generation (RAG) search.
• Try a self-paced codelab showing how to add a chatbot to an existing Firebase app using Firebase Genkit (which is a new AI framework service).
• Learn about similarity search capabilities in Firestore and Cloud SQL for PostgreSQL.
• Teach your chatbot to call into your existing app with function calling.