Build AI-powered web apps with Firebase Extensions

1. Before you begin

In this codelab, you learn how to build AI-powered web apps that provide compelling user experiences with Firebase Extensions.

Prerequisites

• Knowledge of Node.js and JavaScript

What you'll learn

• How to use AI-related Extensions to process language and video input.
• How to use Cloud Functions for Firebase to form a pipeline between Extensions.
• How to use JavaScript to access output produced by Extensions.

What you'll need

• A browser of your choice, such as Google Chrome
• A development environment with a code editor and terminal
• A Google Account for the creation and management of your Firebase project

2. Review web apps and their Firebase services

This section describes the web apps that you'll build in this codelab and which Firebase you'll use to build them.

Reviewly app

A T-shirt company is overwhelmed by long reviews about one of their T-shirts and unsure of its overall rating. The completed Reviewly web app summarizes each review, provides a star rating for each review, and uses each review to infer an overall rating for the product. Users can also expand each summarized review to see the original review.

Chatbot app

A school's teaching staff is overwhelmed by repetitive questions about general topics, so they want to automate responses. The completed Chatbot app provides students with a conversational chatbot that's powered by a large language model (LLM) and can answer questions about general topics. The chatbot has historical context, so its responses can factor in previous questions that students asked in the same conversation.

Video Hint app

A government department wants its videos to provide audio descriptions to improve accessibility, but they have hundreds of videos to annotate and need a streamlined approach. The completed Video Hint app is a prototype that the department will review to assess its effectiveness.

These are the extensions used in the Video Hint app:

• Label Videos with Cloud Video AI extension — extract labels from each video uploaded to Storage.
• Language Tasks with PaLM API extension — summarize the labels into a text description.
• Convert Text to Speech extension — create an audio version of the video's description.

3. Set up your development environment

Verify your Node.js version

1. In your terminal, verify that you have Node.js v20.0.0 or higher installed:

   node -v
   

2. If you don't have Node.js v20.0.0 or higher, download and install it.

Download the repository

1. If you have git installed, clone the codelab's GitHub repository:

   git clone https://github.com/FirebaseExtended/ai-extensions-codelab.git
   

2. If you don't have git installed, download the GitHub repository as a zip file.

Review the folder structure

On your local machine, find the cloned repository and review the folder structure. The following table contains the folders and their descriptions:

Each folder includes a readme.md file that offers a quick start to run the respective web app, using streamlined instructions. However, if you're a first-time learner, you should complete this codelab because it contains the most comprehensive set of instructions.

If you're unsure of whether you correctly applied code as instructed throughout this codelab, you can find the solution code for the respective apps in the reviewly-end, chatbot-end, and video-hint-end folders.

Install the Firebase CLI

Run the following command to verify that you have the Firebase CLI installed and that it's v12.5.4 or higher:

firebase --version

• If you have the Firebase CLI installed, but it's not v12.5.4 or higher, update it:

  npm update -g firebase-tools
  

• If you don't have the Firebase CLI installed, install it:

  npm install -g firebase-tools
  

If you're unable to install the Firebase CLI because of permission errors, see the npm documentation or use another installation option.

Log in to Firebase

1. In your terminal, navigate to the ai-extensions-codelab folder and log in to Firebase:

   cd ai-extensions-codelab
   firebase login
   

2. If your terminal says that you're already logged in to Firebase, proceed to the Set up your Firebase project section of this codelab.
3. Depending on whether you want Firebase to collect data, enter Y or N.
4. In your browser, select your Google account, and then click Allow.

4. Set up your Firebase project

In this section, you'll set up a Firebase project and associate a Firebase web app with it. You'll also enable the Firebase services used by the sample web apps.

Create a Firebase project

1. In the Firebase console, click Create project.
2. In the Enter your project name text box, enter AI Extensions Codelab (or a project name of your choice), and then click Continue.
3. For this codelab, you don't need Google Analytics, so toggle off the Enable Google Analytics for this project option.
4. Click Create project.
5. Wait for your project to provision, and then click Continue.
6. In your Firebase project, go to Project Settings. Note your project ID because you need it later. This unique identifier is how your project is identified (for example, in the Firebase CLI).

Download a Firebase service account

Some of the web apps you'll build in this codelab use server side rendering with Next.js.

The Firebase Admin SDK for Node.js is used to ensure Security Rules are functional from server side code. To use APIs in Firebase Admin, you need to download a Firebase service account from the Firebase console.

1. In the Firebase console, navigate to the Service Accounts page in your Project settings.
2. Click Generate new private key > Generate Key.
3. After the file has downloaded to your filesystem, get the full path to that file.
   For example, if you downloaded the file to your Downloads folder, the full path might look like this: /Users/me/Downloads/my-project-id-firebase-adminsdk-123.json
4. In your terminal, set the GOOGLE_APPLICATION_CREDENTIALS environment variable to the path of your downloaded private key. In a Unix environment, the command might look like this:

   export GOOGLE_APPLICATION_CREDENTIALS="/Users/me/Downloads/my-project-id-firebase-adminsdk-123.json"
   

5. Keep this terminal open and use it for the remainder of this codelab, as your environment variable might be lost if you start a new terminal session.
   If you open a new terminal session, you must rerun the previous command.

Upgrade your Firebase pricing plan

To use Cloud Functions and Firebase Extensions, your Firebase project needs to be on the Blaze pricing plan, which means it's associated with a Cloud Billing account.

• A Cloud Billing account requires a payment method, like a credit card.
• If you're new to Firebase and Google Cloud, check if you're eligible for a $300 credit and a Free Trial Cloud Billing account.

However, note that completion of this codelab shouldn't incur any actual charges.

To upgrade your project to the Blaze plan, follow these steps:

1. In the Firebase console, select to upgrade your plan.
2. In the dialog, select the Blaze plan, and then follow the on-screen instructions to associate your project with a Cloud Billing account.
   If you needed to create a Cloud Billing account, you might need to navigate back to the upgrade flow in the Firebase console to complete the upgrade.

Set up Firebase services in the Firebase console

In this section, you'll provision and set up several Firebase services used by the web apps in this codelab. Note that not all of these services are used in each web app, but setting up all these services right now is a convenience for working through this codelab.

Set up Authentication

You'll use Authentication with both the Chatbot app and the Video Hint app. Remember, though, if you were building a real app, each app should have its own Firebase project.

1. In the Firebase console, navigate to Authentication.
2. Click Get started.
3. In the Additional providers column, click Google > Enable.

4. In the Public-facing name for project text box, enter a memorable name, such as My AI Extensions Codelab.
5. From the Support email for project drop-down, select your email address.
6. Click Save.
   

Set up Cloud Firestore

You'll use Firestore with all three apps. Remember, though, if you were building a real app, each app should have its own Firebase project.

1. In the Firebase console, navigate to Firestore.
2. Click Create database > Start in test mode > Next.
   Later in this codelab, you'll add Security Rules to secure your data. Do not distribute or expose an app publicly without adding Security Rules for your database.
3. Use the default location or select a location of your choice.
   For a real app, you want to choose a location that's close to your users. Note that this location cannot be changed later, and it will also automatically be the location of your default Cloud Storage bucket (next step).
4. Click Done.

Set up Cloud Storage for Firebase

You'll use Cloud Storage with the Video Hint app and to try out the Convert Text to Speech extension (next step of the codelab).

1. In the Firebase console, navigate to Storage.
2. Click Get started > Start in test mode > Next.
   Later in this codelab, you'll add Security Rules to secure your data. Do not distribute or expose an app publicly without adding Security Rules for your Storage bucket.
3. The location of your bucket should already be selected (due to setting up Firestore in the previous step).
4. Click Done.

In the next sections of this codelab, you'll install extensions and modify the codebases of each sample app in this codelab to get three different web apps working.

5. Set up the "Language Tasks with PaLM API" extension for the Reviewly app

Install the Language Tasks with PaLM API extension

1. Navigate to the Language Tasks with PaLM API extension.
2. Click Install in the Firebase console.
3. Select your Firebase project.
4. In the Review APIs enabled and resources created section, click Enable next to any services that are suggested to you:

5. Click Next > Next.
6. In the Collection path text box, enter bot.
7. In the Prompt text box, enter {{ input }}.
8. In the Variable fields text box, enter input.
9. In the Response field text box, enter text.
10. From the Cloud Functions location drop-down, select Iowa (us-central1) or the location that you previously selected for Firestore and Cloud Storage.
11. From the Language model drop-down, select text-bison-001.
12. Leave all other values as their default.
13. Click Install extension and wait for the extension to install.

Try out the Language Tasks with PaLM API extension

While the goal of this codelab is to interact with the Language Tasks with PaLM API extension through a web app, it's helpful to understand how the extension works by triggering the extension using the Firebase console. The extension triggers when a Cloud Firestore document is added to the bot collection.

To see how the extension works by using the Firebase console, follow these steps:

1. In the Firebase console, navigate to Firestore.
2. In the bot collection, click Add document.
3. In the Document ID text box, click Auto-ID.
4. In the Field text box, enter input.
5. In the Value text box, enter Tell me about the moon.
6. Click Save and wait a few seconds. Your document within the bot collection now includes a response to your query.

6. Set up the Reviewly app to use Firebase

To run the Reviewly app, you need to set up your app's code and the Firebase CLI to interact with your Firebase project.

Add Firebase services and configuration to your app's code

To use Firebase, your app's codebase needs the Firebase SDKs for the services you want to use and the Firebase configuration that tells those SDKs which Firebase project to use.

This codelab's sample app already includes all the necessary import and initialization code for the SDKs (see reviewly-start/js/reviews.js), so you don't need to add these. However, the sample app only has placeholder values for the Firebase configuration (see reviewly-start/js/firebase-config.js), so you need to register your app with your Firebase project to get the unique Firebase configuration values for your app.

1. In the Firebase console, in your Firebase project, navigate to the Project overview, and then click Web.
   
2. In the App nickname text box, enter a memorable app nickname, such as My Reviewly app.
3. Do not select the Also set up Firebase Hosting for this app checkbox. You'll do these steps later in the codelab.
4. Click Register app.
5. The console displays a code snippet for adding and initializing the Firebase SDK with an app-specific Firebase configuration object. Copy all the properties in the Firebase configuration object.
6. In your code editor, open the reviewly-start/js/firebase-config.js file.
7. Replace the placeholder values with the values you just copied. It's ok if you have properties and values for Firebase services that you don't use in the Reviewly app.
8. Save the file.
9. Back in the Firebase console, click Continue to console.

Set up your terminal to run Firebase CLI commands against your Firebase project

1. In your terminal, navigate to the ai-extensions-codelab folder that you downloaded earlier.
2. Navigate to the reviewly-start web app folder:

   cd reviewly-start
   

3. Make the Firebase CLI run commands against a specific Firebase project:

   firebase use YOUR_PROJECT_ID
   

Run and view the Reviewly web app

To run and view the web app, follow these steps:

1. In your terminal, install dependencies and then run the web app:

   npm install
   
   # Include the parentheses in the following command.
   (cd functions && npm install)
   
   npm run dev
   

2. In your browser, navigate to the URL shown in your terminal. For example: http://localhost:8080.

The page should load, but you'll notice that various features are missing. We'll add these in the next steps of this codelab.

7. Add functionality to the Reviewly app

In the last step of this codelab, you ran the Reviewly app locally, but it didn't have much functionality and didn't yet use the installed extension. In this step of the codelab, you'll add this functionality and use the web app to trigger the extension.

Deploy Security Rules

This codelab's sample app contains sets of Security Rules for Firestore and for Cloud Storage for Firebase. After you deploy these Security Rules to your Firebase project, the data in your database and your bucket are better protected from misuse.

You can view these rules in the firestore.rules and storage.rules files.

1. To deploy these Security Rules, run this command in your terminal:

   firebase deploy --only firestore:rules,storage
   

2. If you're asked: "Cloud Storage for Firebase needs an IAM Role to use cross-service rules. Grant the new role?", select Yes.

Update the app's code to trigger the extension

In the Reviewly app, a new review added to Firestore triggers the extension to summarize the review.

1. In your code editor, open the functions/add-mock-reviews.js file.
2. Replace the reviewWithPrompt variable with the following code, which prompts the language model for a shorter review.

   const reviewWithPrompt = `Here's a user supplied review. Give me a shorter summary of the review, and a rating out of 5, and a flag which indicates if there was a product defect.
   
   Product name: "Blue t-shirt with cat picture". Review: """${review}"""
   
   Craft your response as JSON with properties has_defect, summary and rating. Don't include any extra text.`;
   

3. After the reviewWithPrompt variable, replace the reviewDocument variable with the following code, which creates a review document so that it can be added to Firestore.

   const reviewDocument = {
           input: reviewWithPrompt,
           originalReview: review,
           timestamp: Timestamp.now(),
   };
   
   getFirestore().collection(DB_COLLECTION_NAME).add(reviewDocument);
   

4. Save the file.
5. In the js/reviews.js file, after the Insert code below, to import your Firebase Callable Cloud Function comment, import your Firebase HTTP callable function with the httpsCallable helper:

   const addMockReviews = httpsCallable(functions, "addMockReviews");
   

6. After the Insert code below, to invoke your Firebase Callable Cloud Function comment, invoke your Firebase HTTP callable function:

   await addMockReviews();
   

7. Save the file.

Deploy a function to add new reviews

The Reviewly web app uses a Cloud Function to add the reviews. But currently, the Cloud Function is not deployed.

To deploy your function with the Firebase CLI, follow these steps:

1. In your terminal while still in the reviewly-start folder, press Control+C to stop the server.
2. Deploy your function:

   firebase deploy --only functions
   

3. If you see a Permission denied while using the Eventarc Service Agent or similar error, wait a few minutes, and then retry the command.

You just deployed your first custom function with Cloud Functions. The Firebase console offers a dashboard where you can see all the functions that you deploy to your Firebase project.

Run and view the Reviewly web app again (now with functionality)

To run and view the now functional web app, follow these steps:

1. In your terminal, run the server again:

   npm run dev
   

2. In your browser, navigate to the URL shown in your terminal. For example: http://localhost:8080.
3. In the app, click Add some mock reviews and wait a few seconds for a few long reviews to appear.
   In the background, the Language Tasks with PaLM API extension reacts to the new document that represents the new review. The prompt you added earlier requests a shorter summary from the language model.
4. To view an entire review and the prompt used for the review, click one of the reviews, and then select Show PaLM prompt.

8. Set up the "Chatbot with PaLM API" extension for the Chatbot app

Install the Chatbot with PaLM API extension

1. Navigate to the Chatbot with PaLM API extension.
2. Click Install in Firebase console.
3. Select your Firebase project.
4. Click Next > Next > Next, until you reach the Configure extension section.
5. In the Collection path text box, enter users/{uid}/discussion/{discussionId}/messages.
6. From the Cloud Functions location drop-down, select Iowa (us-central1) or the location that you previously selected for Firestore and Cloud Storage.
7. From the Language model drop-down, select chat-bison.
8. Leave all other values as their default.
9. Click Install extension and wait for the extension to install.

Try out the Chatbot with PaLM API extension

While the goal of this codelab is to interact with the Chatbot with PaLM API extension through a web app, it's helpful to understand how the extension works by triggering the extension using the Firebase console. The extension triggers when a Cloud Firestore document is created in the users/{uid}/discussion/{discussionId}/messages collection.

1. In the Firebase console, navigate to Firestore.
2. Click Start collection.
   1. In the Collection ID text box, enter users, and then click Next.
   2. In the Document ID text box, click Auto-ID, and then click Save.
3. In the users collection, click Start collection.
   
   1. In the Collection ID text box, enter discussion, and then click Next.
   2. In the Document ID text box, click Auto-ID, and then click Save.
4. In the discussion collection, click Start collection.
   
   1. In the Collection ID text box, enter messages, and then click Next.
   2. In the Document ID text box, click Auto-ID.
   3. In the Field text box, enter prompt.
   4. In the Value text box, enter Tell me 5 random fruits.
   5. Click Save and wait a few seconds. The messages collection now includes a document that contains a response to your query.

5. In the messages collection, click Add document.
   1. In the Document ID text box, click Auto-ID.
   2. In the Field text box, enter prompt.
   3. In the Value text box, enter And now, vegetables.
   4. Click Save and wait a few seconds. The messages collection now includes a document that contains a response to your query.

9. Set up the Chatbot app to use Firebase

To run the Chatbot app, you need to set up your app's code and the Firebase CLI to interact with your Firebase project.

Add Firebase services and configuration to your app's code

To use Firebase, your app's codebase needs the Firebase SDKs for the services you want to use and the Firebase configuration that tells those SDKs which Firebase project to use.

This codelab's sample app already includes all the necessary import and initialization code for the SDKs (see chatbot-start/lib/firebase/firebase.js), so you don't need to add these. However, the sample app only has placeholder values for the Firebase configuration (see chatbot-start/lib/firebase/firebase-config.js), so you need to register your app with your Firebase project to get the unique Firebase configuration values for your app.

1. In the Firebase console, in your Firebase project, navigate to the Project overview, and then click Web (or click Add app if you've already registered an app with the project).
2. In the App nickname text box, enter a memorable app nickname, such as My Chatbot app.
3. Do not select the Also set up Firebase Hosting for this app checkbox. You'll do these steps later in the codelab.
4. Click Register app.
5. The console displays a code snippet for adding and initializing the Firebase SDK with an app-specific Firebase configuration object. Copy all the properties in the Firebase configuration object.
6. In your code editor, open the chatbot-start/lib/firebase/firebase-config.js file.
7. Replace the placeholder values with the values you just copied. It's ok if you have properties and values for Firebase services that you don't use in the Chatbot app.
8. Save the file.
9. Back in the Firebase console, click Continue to console.

Set up your terminal to run Firebase CLI commands against your Firebase project

1. In your terminal, press Control+C to stop the server from running the previous web app.
2. In your terminal, navigate to the chatbot-start web app folder:

   cd ../chatbot-start
   

3. Make the Firebase CLI run commands against a specific Firebase project:

   firebase use YOUR_PROJECT_ID
   

Set up your app's codebase to use framework-aware Firebase Hosting

This codelab uses web frameworks with Hosting (preview) with the Chatbot web app.

1. In your terminal, enable web frameworks with Firebase Hosting:

   firebase experiments:enable webframeworks
   

2. Initialize Firebase Hosting:

   firebase init hosting
   

3. When prompted with Detected an existing Next.js codebase in your current directory, should we use this?, press Y.
4. When prompted with In which region would you like to host server-side content, if applicable?, select either the default region or the location that you previously selected for Firestore and Cloud Storage, and then press Enter (or return on macOS).
5. When prompted with Set up automatic builds and deploys with GitHub?, press N.

Run and view the Chatbot web app

1. In your terminal, install dependencies and then run the web app:

   npm install
   firebase emulators:start --only hosting
   

2. In your browser, navigate to the locally hosted Hosting URL. In most cases, it's http://localhost:5000/ or something similar.

The page should load, but you'll notice that various features are missing. We'll add these in the next steps of this codelab.

Troubleshoot running the web app

If you see the error in the web page that starts like this: Error: Firebase session cookie has incorrect..., you need to delete all your cookies in your localhost environment. To do this, follow the instructions on delete cookies | DevTools Documentation..

10. Add functionality to the Chatbot app

In the last step of this codelab, you ran the Chatbot app locally, but it didn't have much functionality and didn't yet use the installed extension. In this step of the codelab, you'll add this functionality and use the web app to trigger the extension.

Deploy Security Rules

This codelab's sample app contains sets of Security Rules for Firestore and for Cloud Storage for Firebase. After you deploy these Security Rules to your Firebase project, the data in your database and your bucket are better protected from misuse.

You can view these rules in the firestore.rules and storage.rules files.

1. To deploy these Security Rules, run this command in your terminal:

   firebase deploy --only firestore:rules,storage
   

2. If you're asked: "Cloud Storage for Firebase needs an IAM Role to use cross-service rules. Grant the new role?", select Yes.

Update the code to add messages to Cloud Firestore

In the Chatbot app, a new message from a user is added to Firestore and that triggers the extension to generate a response.

1. In your code editor, open the lib/firebase/firestore.js file.
2. Near the end of the file, find the addNewMessage function, which handles the addition of new messages.
   The function already takes in the following object properties:
   

   Parameter	Description
   userId	The ID of the logged-in user
   discussionId	The discussion ID to which the message was added
   message	The message text content
   db	A Firestore database instance

   With these variables ready, you can add a Cloud Firestore document to represent the new message.
3. In the body of the addNewMessage function after the // Insert your code below ⬇️ comment, add the following code:

   await addDoc(
           collection(
                   db,
                   "users",
                   userId,
                   "discussion",
                   discussionId,
                   "messages"
           ),
           {
                   prompt: message,
                   createTime: serverTimestamp(),
           }
   );
   
   await updateDoc(doc(db, "users", userId, "discussion", discussionId), {
           latestMessage: message,
           updatedTime: serverTimestamp(),
   });
   

Update the code to construct a query to get messages

1. Still in the lib/firebase/firestore.js file, locate the getMessagesQuery function, which needs to return a Cloud Firestore query that locates messages stored in the users/{uid}/discussion/{discussionId}/messages collection path.
2. Replace the entire getMessagesQuery function with the following code:

   function getMessagesQuery(db, userId, discussionId) {
           if (!userId || !discussionId) {
                   return null;
           }
           const messagesRef = collection(
                   db,
                   "users",
                   userId,
                   "discussion",
                   discussionId,
                   "messages"
           );
           return query(messagesRef, orderBy("createTime", "asc"));
   }
   

Update the code to handle Cloud Firestore message documents

1. Still in the lib/firebase/firestore.js file, locate the handleMessageDoc function, which receives a Cloud Firestore document that represents a single message.
   This function needs to format and structure the data in a way that makes sense for the UI of the Chatbot app.
2. Replace the entire handleMessageDoc function with the following code:

   function handleMessageDoc(doc) {
           const data = doc.data();
           const item = {
                   prompt: data.prompt,
                   response: data.response,
                   id: doc.id,
                   createTime: formatDate(data.createTime.toDate()),
           };
   
           if (data?.status?.completeTime) {
                   item.completeTime = formatDate(data.status.completeTime.toDate());
           }
   
           return item;
   }
   

3. Save the file.

Run and view the Chatbot web app again (now with functionality)

To run and view the now functional web app, follow these steps:

1. In your browser, return to the tab with the Chatbot web app and reload the page.
2. Click Sign in with Google.
3. If necessary, select your Google account.
4. After you have signed in, reload the page.
5. In the Enter your message text box, enter a message, such as Tell me about space.
6. Click Send and wait a few seconds for the Chatbot web app to respond.

Another benefit of the Chatbot with PaLM API extension is its conversation history.

To see an example of its ability to converse with historical context, follow these steps:

1. In the Enter your message text box, ask a question, such as What are five random fruits?.
2. In the Enter your message text box, ask a follow-up question that's related to the previous question, such as And what about vegetables?.

The Chatbot web app responds with historical knowledge. Even though your last question didn't specify five random vegetables, the Chatbot with PaLM API extension understands follow-up questions.

11. Set up the "Convert Text to Speech" extension for the Video Hint app

Install the Convert Text to Speech extension

1. Navigate to the Convert Text to Speech extension.
2. Click Install in the Firebase console.
3. Select your Firebase project.
4. Click Next.
5. In the Review APIs enabled and resources created section, click Enable next to any services that are suggested to you:

6. Click Next, and then select Grant next to any permissions that are suggested to you.

7. Click Next.
8. In the Collection path text box, enter bot.
9. In the Storage path text box, enter tts.
10. Leave all other values as their default options.
11. Click Install extension and wait for the extension to install.

Try out the Convert Text to Speech extension

While the goal of this codelab is to interact with the Convert Text to Speech extension through a web app, it's helpful to understand how the extension works by triggering the extension using the Firebase console. The extension triggers when a Cloud Firestore document is created in the bot collection.

To see how the extension works by using the Firebase console, follow these steps:

1. In the Firebase console, navigate to Firestore
2. Click Start collection.
3. In the Collection ID text box, enter bot.
4. Click Next.

5. In the Document ID text box, click Auto-ID.
6. In the Field text box, enter text.
7. In the Value text box, enter The quick brown fox jumps over the lazy dog.
8. Click Save.

To see and hear the MP3 file that you created, follow these steps:

1. In the Firebase console, navigate to Storage.
2. In the pane where you can upload files, note your bucket name after the gs:// value. This is the name of your default bucket for this project. You need it at various tasks throughout this codelab.

3. In the Google Cloud Console, navigate to Cloud Storage.
4. Select your project.
   If you don't see your project within the recent list of projects, click Select Project to locate your project in the project picker.

5. Select your default Storage bucket.
6. Navigate to the tts/ folder.
7. Click the MP3 file.
8. At the end of the MP3 file, click and notice that your text is converted to speech.

12. Set up the "Label Videos with Cloud Video AI" extension for the Video Hint app

Install the Label Videos with Cloud Video AI extension

1. Navigate to the Label Videos with Cloud Video AI extension.
2. Click Install in Firebase console.
3. Select your Firebase project.
4. Click Next > Next > Next, until you reach the Configure extension section.
5. From the Cloud Functions location drop-down, select a supported location (either the location that you previously selected for Firestore and Cloud Storage or the nearest to it). For supported locations, see the location_id section in AnnotateVideoRequest.
6. From the Model drop-down, select Latest.
7. From the Stationary camera drop-down, select No.
8. Leave all other values as their default.
9. Click Install extension and wait for the extension to install.

Try out the Label Videos with Cloud Video AI extension

While the goal of this codelab is to interact with the Label Videos with Cloud Video AI extension through a web app, it's helpful to understand how the extension works by triggering the extension using the Firebase console. The extension triggers when a video file is uploaded to your Storage bucket.

To see how the extension works by using the Firebase console, follow these steps:

1. Navigate to Storage within your Firebase project > Create folder.
2. In the Folder name text box, enter video_annotation_input.

3. Click Add folder.
4. In the video_annotation_input folder, click Upload file.
5. In the ai-extensions-codelab/video-hint-start/public/videos folder that you cloned or downloaded earlier, select the first video file.
6. Back in your browser, in the Google Cloud Console, navigate to Cloud Storage.
7. Select your default Storage bucket, which you noted earlier.
8. Click the video_annotation_output folder.
   If you don't see the video_annotation_output folder, wait a few seconds and refresh the page because the Video Intelligence API might still be processing the video.

9. Notice a JSON file exists that follows a similar name to the file you uploaded earlier.
10. Click Download FILENAME.
11. Open the downloaded JSON file in your code editor. It contains the raw output from the Video Intelligence API, which includes detected labels of the video that you uploaded.

13. Set up the Video Hint app to use Firebase

To run the Video Hint app, you need to set up your app's code and the Firebase CLI to interact with your Firebase project.

Add Firebase services and configuration to your app's code

To use Firebase, your app's codebase needs the Firebase SDKs for the services you want to use and the Firebase configuration that tells those SDKs which Firebase project to use.

This codelab's sample app already includes all the necessary import and initialization code for the SDKs (see video-hint-start/lib/firebase/firebase.js), so you don't need to add these. However, the sample app only has placeholder values for the Firebase configuration (see video-hint-start/lib/firebase/firebase-config.js), so you need to register your app with your Firebase project to get the unique Firebase configuration values for your app.

1. In the Firebase console, in your Firebase project, navigate to the Project overview, and then click Web (or click Add app if you've already registered an app with the project).
2. In the App nickname text box, enter a memorable app nickname, such as My Video Hint app.
3. Do not select the Also set up Firebase Hosting for this app checkbox. You'll do these steps later in the codelab.
4. Click Register app.
5. The console displays a code snippet for adding and initializing the Firebase SDK with an app-specific Firebase configuration object. Copy all the properties in the Firebase configuration object.
6. In your code editor, open the video-hint-start/lib/firebase/firebase-config.js file.
7. Replace the placeholder values with the values you just copied. It's ok if you have properties and values for Firebase services that you don't use in the Video Hint app.
8. Save the file.
9. Back in the Firebase console, click Continue to console.

Set up your terminal to run Firebase CLI commands against your Firebase project

1. In your terminal, press Control+C to stop the server from running the previous web app.
2. In your terminal, navigate to the video-hint-start web app folder:

   cd ../video-hint-start
   

3. Make the Firebase CLI run commands against a specific Firebase project:

   firebase use YOUR_PROJECT_ID
   

Set up your app's codebase to use framework-aware Firebase Hosting

This codelab uses web frameworks with Hosting (preview) with the Video Hint web app.

1. In your terminal, enable web frameworks with Firebase Hosting:

   firebase experiments:enable webframeworks
   

2. Initialize Firebase Hosting:

   firebase init hosting
   

3. When prompted with Detected an existing Next.js codebase in your current directory, should we use this?, press Y.
4. When prompted with In which region would you like to host server-side content, if applicable?, select either the default region or the location that you previously selected for Firestore and Cloud Storage, and then press Enter (or return on macOS).
5. When prompted with Set up automatic builds and deploys with GitHub?, press N.

Run and view the Video Hint web app

1. In your terminal, install dependencies in the video-hint-start and functions folders, and then run the app:

   npm install
   
   # Include the parentheses in the following command.
   (cd functions && npm install)
   
   firebase emulators:start --only hosting
   

2. In your browser, navigate to the locally hosted Hosting URL. In most cases, it's http://localhost:5000/ or something similar.

The page should load, but you'll notice that various features are missing. We'll add these in the next steps of this codelab.

Troubleshoot running the web app

If you see an error message like Error: The query requires an index. You can create it here: https://console.firebase.google.com in the Console pane of DevTools, follow these steps:

1. Navigate to the provided URL.

2. Click Save and wait for the status to change from Building to Enabled.

14. Add functionality to the Video Hint app

In the last step of this codelab, you ran the Video Hint app locally, but it didn't have much functionality and didn't yet use the installed extension. In this step of the codelab, you'll add this functionality and use the web app to trigger the extension.

Deploy Security Rules

This codelab's sample app contains sets of Security Rules for Firestore and for Cloud Storage for Firebase. After you deploy these Security Rules to your Firebase project, the data in your database and your bucket are better protected from misuse.

You can view these rules in the firestore.rules and storage.rules files.

1. To deploy these Security Rules, run this command in your terminal:

   firebase deploy --only firestore:rules,storage
   

2. If you're asked: "Cloud Storage for Firebase needs an IAM Role to use cross-service rules. Grant the new role?", select Yes.

Update the code to combine the functions

1. In your code editor, expand the functions folder.
   This folder contains several functions, which combine to form an extension pipeline. The following table lists and describes each function:

   Function	Description
   functions/01-handle-video-upload.js	The first step in the extension pipeline. It processes the user's uploaded video file.
   functions/02-handle-video-labels.js	The second step in the extension pipeline. It processes the video-labels file, which was generated by the storage-label-videos extension.
   functions/03-handle-audio-file.js	The third step in the extension pipeline. It handles the transcribed audio file.

   However, you still need to add a file that groups these functions together.
2. In the functions/index.js file, add the following code:

   import { initializeApp } from "firebase-admin/app";
   
   export * from "./01-handle-video-upload.js";
   export * from "./02-handle-video-labels.js";
   export * from "./03-handle-audio-file.js";
   
   initializeApp();
   

This code uses JavaScript modules to import and export the functions from the index.js file so that there's one centralized location for all functions.

Update the code to handle video uploading

1. In your code editor, open the lib/firebase/storage.js file.
2. Locate the uploadVideo function.
   This function receives userId, filePath, and file parameters. This data is enough to upload a file to Cloud Storage.
3. In the body of the uploadVideo function, add the following code:

   const storageRef = ref(storage, `video_annotation_input/${filePath}`);
   
   const uploadTask = uploadBytesResumable(storageRef, file, {
           customMetadata: {
                   uid: userId,
           },
   });
   
   return uploadTask;
   

Deploy your functions

To deploy your functions with the Firebase CLI, follow these steps:

1. In your terminal while still in the video-hint-start folder, press Control+C to stop the current process.
2. Deploy your functions:

   firebase deploy --only functions
   

   If you are asked about deleting any previous Cloud Functions, select No.
3. If you see a message similar to Permission denied while using the Eventarc Service Agent, wait a few minutes and then retry the command.
4. After the command is finished, run the app locally again:

   firebase emulators:start --only hosting
   

Run and view the Video Hint web app again (now with functionality)

To run and view the now functional web app, follow these steps:

1. In your browser, find the tab where you navigated to http://localhost:5000.
2. If necessary, click Sign in with Google and select your Google account.
3. Click Upload Example video #1 and wait a few minutes to see the results of the video summary.
4. If you don't see any results after you upload the video, see Troubleshoot errors with Cloud Functions in the appendix of this codelab.

15. Conclusion

Congratulations! You achieved a lot in this codelab!

Installed and configured Firebase Extensions

You used the Firebase console to configure and install various AI extensions. Using extensions is convenient because you don't need to write lots of boilerplate code that deals with authenticating with Google Cloud services, reading and writing from Firestore and interacting with Google Cloud services - and the various nuances that are involved with those tasks.

Worked with extensions using the Firebase console

Instead of jumping straight into code, you took time to understand how the AI extensions work, based on an input that you provided via the console to Firestore or Cloud Storage. This type of interaction can be especially useful if you need to debug extension output.

Built three AI-powered web apps that use Firebase Extensions

Reviewly

In the Reviewly web app, you used the Language Tasks with PaLM API extension to summarize long reviews that users left for a T-shirt product. You also requested that the language model provide a JSON response to your query, where the JSON provided a star rating and a summarized review of the original long-form review.

Optional exercise: The T-shirt company is happy with the summarized reviews, but they've asked for an additional summary of the nature of the defect. Can you adjust the prompt to return a summary of the defect, and then include that summary within the user interface of the web app?

Chatbot

In the Chatbot web app, you used the Chatbot with PaLM API extension to provide the user with an interactive chat interface, that includes historical context in conversations - where each message is stored in a Firestore document that is scoped to a particular user.

Optional exercise: The students have been pleased with the chatbot, but the staff would like a few enhancements. Students should be given thought-provoking questions after their answer has been provided. For example:

Student asks: What is the ozone?
Response: The ozone is a molecule composed of...How do you think human activities can impact the ozone layer?

Hint: You could use a configurable context option to achieve this.

Video Hint

In the Video Hint web app, you used the Convert Text to Speech, the Language Tasks with PaLM API, and the Label Videos with Cloud Video extensions to form an extension pipeline that results in a text and audio description of a video.

Optional exercise: The government department found the prototype interesting, and they now would also like a user to be able to click on a text-based label to jump to the point in the video where the label is detected.

16. Appendix: Troubleshoot errors with Cloud Functions

If your web app isn't functioning as expected, and you think it might be due to functions, follow the steps in this troubleshooting page.

Allow public unauthenticated access

If you get any permission-related errors in the Console panel of Chrome DevTools, follow these steps:

1. Read the Authentication overview | Cloud Run page
2. Click the link to view and complete the required tasks to Allow public unauthenticated access to the function.

3. Navigate back to the Reviewly app. For example: http://localhost:8080.
4. Click Add some mock reviews and wait a few seconds.
   • If reviews appear: You don't need to continue with these troubleshooting steps, and can jump straight to Set up the Chatbot web app section in this codelab..
   • If reviews do not appear: Continue with this troubleshooting section.

Handle insufficient permission errors

1. In the Firebase console, navigate to Functions.
2. Hover over the addMockReviews function, and then click > View logs.

3. Scroll through the logs until you find an error similar to one of the following:

   Exception from a finished function: Error: 7 PERMISSION_DENIED: Missing or insufficient permissions.
   

   0001-compute@developer.gserviceaccount.com does not have storage.objects.get access to the Google Cloud Storage object. Permission 'storage.objects.get' denied on resource (or it may not exist).
   

4. In the Google Cloud Console, navigate to the IAM Permissions page, and then select your project.
5. In the table, find the Name column.
   In the IAM & Admin page, there is a table of users and roles. The name column in the table describes what the user, or principal, is for. You may have a principal with the name of Default compute service account.

If you see the Default compute service account, follow these steps:

1. Click Edit principal.

2. Continue with the Add roles to the Default compute service account section in this codelab.

If you don't see the Default compute service account, follow these steps:

1. Click Grant Access.
2. In the New principals text box, enter compute.
3. In the menu of autosuggestions that appears, select Default compute service account.

Add roles to the default compute service account

In the Assign roles section of the default compute service account:

1. Expand the Select a role menu.
2. In the Filter text box, enter Cloud Datastore User.
3. In the menu of autosuggestions that appears, select Cloud Datastore user.
4. Click Add another role.
   1. Expand the Select a role menu.
   2. In the Filter text box, enter Cloud Storage for Firebase Admin.
   3. In the menu of autosuggestions that appears, select Cloud Storage for Firebase Admin.
5. Click Add another role.
   1. Expand the Select a role menu.
   2. In the Filter text box, enter Cloud Storage for Firebase Service Agent.
   3. In the menu of autosuggestions that appears, select Cloud Storage for Firebase Service Agent.
6. Click Save.