Function calling using the Gemini API

Generative models are powerful at solving many types of problems. However, they are constrained by limitations like:

• They are frozen after training, leading to stale knowledge.
• They can't query or modify external data.

Function calling can help you overcome some of these limitations. Function calling is sometimes referred to as tool use because it allows a model to use external tools such as APIs and functions to generate its final response.

You can learn more about function calling in the Google Cloud documentation, including a helpful list of use cases for function calling.

Function calling is supported by Gemini 1.0 Pro, Gemini 1.5 Pro, and Gemini 1.5 Flash.

This guide shows you how you might implement a function call setup similar to the example described in the next major section of this page. At a high-level, here are the steps to set up function calling in your app:

1. Write a function that can provide the model with information that it needs to generate its final response (for example, the function can call an external API).

2. Create a function declaration that describes the function and its parameters.

3. Provide the function declaration during model initialization so that the model knows how it can use the function, if needed.

4. Set up your app so that the model can send along the required information for your app to call the function.

5. Pass the function's response back to the model so that the model can generate its final response.

Jump to code implementation

Overview of a function calling example

When you send a request to the model, you can also provide the model with a set of "tools" (like functions) that it can use to generate its final response. In order to utilize these functions and call them ("function calling"), the model and your app need to pass information back-and-forth to each other, so the recommended way to use function calling is through the multi-turn chat interface.

Imagine that you have an app where a user could enter a prompt like: What was the weather in Boston on October 17, 2024?.

The Gemini models may not know this weather information; however, imagine that you know of an external weather service API that can provide it. You can use function calling to give the Gemini model a pathway to that API and its weather information.

First, you write a function fetchWeather in your app that interacts with this hypothetical external API, which has this input and output:

When initializing the model, you tell the model that this fetchWeather function exists and how it can be used to process incoming requests, if needed. This is called a "function declaration". The model does not call the function directly. Instead, as the model is processing the incoming request, it decides if the fetchWeather function can help it respond to the request. If the model decides that the function can indeed be useful, the model generates structured data that will help your app call the function.

Look again at the incoming request: What was the weather in Boston on October 17, 2024?. The model would likely decide that the fetchWeather function can help it generate a response. The model would look at what input parameters are needed for fetchWeather and then generate structured input data for the function that looks roughly like this:

{
  functionName: fetchWeather,
  location: {
    city: Boston,
    state: Massachusetts  // the model can infer the state from the prompt
  },
  date: 2024-10-17
}

The model passes this structured input data to your app so that your app can call the fetchWeather function. When your app receives the weather conditions back from the API, it passes the information along to the model. This weather information allows the model to complete its final processing and generate its response to the initial request of What was the weather in Boston on October 17, 2024?

The model might provide a final natural-language response like: On October 17, 2024, in Boston, it was 38 degrees Fahrenheit with partly cloudy skies.

Implement function calling

Before you begin

If you haven't already, complete the getting started guide for the Vertex AI in Firebase SDKs. Make sure that you've done all of the following:

1. Set up a new or existing Firebase project, including using the Blaze pricing plan and enabling the required APIs.

2. Connect your app to Firebase, including registering your app and adding your Firebase config to your app.

3. Add the SDK and initialize the Vertex AI service and the generative model in your app.

After you've connected your app to Firebase, added the SDK, and initialized the Vertex AI service and the generative model, you're ready to call the Gemini API.

The remaining steps in this guide show you how to implement a function call setup similar to the workflow described in Overview of a function calling example (see the top section of this page).

You can view the complete code sample for this function calling example later on this page.

Step 1: Write the function

Imagine that you have an app where a user could enter a prompt like: What was the weather in Boston on October 17, 2024?. The Gemini models may not know this weather information; however, imagine that you know of an external weather service API that can provide it. The example in this guide rely on this hypothetical external API.

Write the function in your app that will interact with the hypothetical external API and provide the model with the information it needs to generate its final request. In this weather example, it will be a fetchWeather function that makes the call to this hypothetical external API.

// This function calls a hypothetical external API that returns
// a collection of weather information for a given location on a given date.
// `location` is an object of the form { city: string, state: string }
data class Location(val city: String, val state: String)

suspend fun fetchWeather(location: Location, date: String): JsonObject {

    // TODO(developer): Write a standard function that would call to an external weather API.

    // For demo purposes, this hypothetical response is hardcoded here in the expected format.
    return JsonObject(mapOf(
        "temperature" to JsonPrimitive(38),
        "chancePrecipitation" to JsonPrimitive("56%"),
        "cloudConditions" to JsonPrimitive("partlyCloudy")
    ))
}

// This function calls a hypothetical external API that returns
// a collection of weather information for a given location on a given date.
// `location` is an object of the form { city: string, state: string }
public JsonObject fetchWeather(Location location, String date) {

  // TODO(developer): Write a standard function that would call to an external weather API.

  // For demo purposes, this hypothetical response is hardcoded here in the expected format.
  return new JsonObject(Map.of(
        "temperature", JsonPrimitive(38),
        "chancePrecipitation", JsonPrimitive("56%"),
        "cloudConditions", JsonPrimitive("partlyCloudy")));
}

Step 2: Create a function declaration

Create the function declaration that you'll later provide to the model (next step of this guide).

In your declaration, include as much detail as possible in the descriptions for the function and its parameters.

The model uses the information in the function declaration to determine which function to select and how to provide parameter values for the actual call to the function. See Additional behaviors and options later on this page for how the model may choose among the functions, as well as how you can control that choice.

Note the following about the schema that you provide:

• You must provide function declarations in a schema format that's compatible with the OpenAPI schema. Vertex AI offers limited support of the OpenAPI schema.

  • The following attributes are supported: type, nullable, required, format, description, properties, items, enum.

  • The following attributes are not supported: default, optional, maximum, oneOf.

• By default, for Vertex AI in Firebase SDKs, all fields are considered required unless you specify them as optional in an optionalProperties array. For these optional fields, the model can populate the fields or skip them. Note that this is opposite from the default behavior for the Vertex AI Gemini API.

For best practices related to the function declarations, including tips for names and descriptions, see Best practices in the Google Cloud documentation.

Here's how you can write a function declaration:

val fetchWeatherTool = FunctionDeclaration(
    "fetchWeather",
    "Get the weather conditions for a specific city on a specific date.",
    mapOf(
        "location" to Schema.obj(
            mapOf(
                "city" to Schema.string("The city of the location."),
                "state" to Schema.string("The US state of the location."),
            ),
            description = "The name of the city and its state for which " +
                "to get the weather. Only cities in the " +
                "USA are supported."
        ),
        "date" to Schema.string("The date for which to get the weather." +
                                " Date must be in the format: YYYY-MM-DD."
        ),
    ),
)

FunctionDeclaration fetchWeatherTool = new FunctionDeclaration(
        "fetchWeather",
        "Get the weather conditions for a specific city on a specific date.",
        Map.of("location",
                Schema.obj(Map.of(
                        "city", Schema.str("The city of the location."),
                        "state", Schema.str("The US state of the location."))),
                "date",
                Schema.str("The date for which to get the weather. " +
                              "Date must be in the format: YYYY-MM-DD.")),
        Collections.emptyList());

Step 3: Provide the function declaration during model initialization

The maximum number of function declarations that you can provide with the request is 128. See Additional behaviors and options later on this page for how the model may choose among the functions, as well as how you can control that choice (using a toolConfig to set the function calling mode).

// Initialize the Vertex AI service and the generative model
// Use a model that supports function calling, like a Gemini 1.5 model
val model = Firebase.vertexAI.generativeModel(
    modelName = "gemini-1.5-flash",
    // Provide the function declaration to the model.
    tools = listOf(Tool.functionDeclarations(listOf(fetchWeatherTool)))
)

// Initialize the Vertex AI service and the generative model
// Use a model that supports function calling, like a Gemini 1.5 model
GenerativeModelFutures model = GenerativeModelFutures.from(
        FirebaseVertexAI.getInstance()
                .generativeModel("gemini-1.5-flash",
                        null,
                        null,
                        // Provide the function declaration to the model.
                        List.of(Tool.functionDeclarations(List.of(fetchWeatherTool)))));

Learn how to choose a Gemini model and optionally a location appropriate for your use case and app.

Step 4: Call the function to invoke the external API

If the model decides that the fetchWeather function can indeed help it generate a final response, your app needs to make the actual call to that function using the structured input data provided by the model.

Since information needs to be passed back-and-forth between the model and the app, the recommended way to use function calling is through the multi-turn chat interface.

The following code snippet shows how your app is told that the model wants to use the fetchWeather function. It also shows that the model has provided the necessary input parameter values for the function call (and its underlying external API).

In this example, the incoming request contained the prompt What was the weather in Boston on October 17, 2024?. From this prompt, the model inferred the input parameters that are required by the fetchWeather function (that is, city, state, and date).

val prompt = "What was the weather in Boston on October 17, 2024?"
val chat = model.startChat()
// Send the user's question (the prompt) to the model using multi-turn chat.
val result = chat.sendMessage(prompt)

val functionCalls = result.functionCalls
// When the model responds with one or more function calls, invoke the function(s).
val fetchWeatherCall = functionCalls.find { it.name == "fetchWeather" }

// Forward the structured input data prepared by the model
// to the hypothetical external API.
val functionResponse = fetchWeatherCall?.let {
    // Alternatively, if your `Location` class is marked as @Serializable, you can use
    // val location = Json.decodeFromJsonElement<Location>(it.args["location"]!!)
    val location = Location(
        it.args["location"]!!.jsonObject["city"]!!.jsonPrimitive.content,
        it.args["location"]!!.jsonObject["state"]!!.jsonPrimitive.content
    )
    val date = it.args["date"]!!.jsonPrimitive.content
    fetchWeather(location, date)
}

String prompt = "What was the weather in Boston on October 17, 2024?";
ChatFutures chatFutures = model.startChat();
// Send the user's question (the prompt) to the model using multi-turn chat.
ListenableFuture<GenerateContentResponse> response =
        chatFutures.sendMessage(new Content("user", List.of(new TextPart(prompt))));

ListenableFuture<JsonObject> handleFunctionCallFuture = Futures.transform(response, result -> {
    for (FunctionCallPart functionCall : result.getFunctionCalls()) {
        if (functionCall.getName().equals("fetchWeather")) {
            Map<String, JsonElement> args = functionCall.getArgs();
            JsonObject locationJsonObject =
                    JsonElementKt.getJsonObject(args.get("location"));
            String city =
                    JsonElementKt.getContentOrNull(
                            JsonElementKt.getJsonPrimitive(
                                    locationJsonObject.get("city")));
            String state =
                    JsonElementKt.getContentOrNull(
                            JsonElementKt.getJsonPrimitive(
                                    locationJsonObject.get("state")));
            Location location = new Location(city, state);

            String date = JsonElementKt.getContentOrNull(
                    JsonElementKt.getJsonPrimitive(
                            args.get("date")));
            return fetchWeather(location, date);
        }
    }
    return null;
}, Executors.newSingleThreadExecutor());

Step 5: Provide the function's output to the model to generate the final response

After the fetchWeather function returns the weather information, your app needs to pass it back to the model.

Then, the model performs its final processing, and generates a final natural-language response like: On October 17, 2024 in Boston, it was 38 degrees Fahrenheit with partly cloudy skies.

// Send the response(s) from the function back to the model
// so that the model can use it to generate its final response.
val finalResponse = chat.sendMessage(content("function") {
    part(FunctionResponsePart("fetchWeather", functionResponse!!))
})

// Log the text response.
println(finalResponse.text ?: "No text in response")

ListenableFuture<GenerateContentResponse> modelResponseFuture = Futures.transformAsync(
  handleFunctionCallFuture,
  // Send the response(s) from the function back to the model
  // so that the model can use it to generate its final response.
  functionCallResult -> chatFutures.sendMessage(new Content("function",
  List.of(new FunctionResponsePart(
          "fetchWeather", functionCallResult)))),
  Executors.newSingleThreadExecutor());

Futures.addCallback(modelResponseFuture, new FutureCallback<GenerateContentResponse>() {
@Override
public void onSuccess(GenerateContentResponse result) {
  if (result.getText() != null) {
      // Log the text response.
      System.out.println(result.getText());
  }
}

@Override
public void onFailure(Throwable t) {
  // handle error
}
}, Executors.newSingleThreadExecutor());

Additional behaviors and options

Here are some additional behaviors for function calling that you need to accommodate in your code and options that you can control.

The model may ask to call a function again or another function.

If the response from one function call is insufficient for the model to generate its final response, then the model may ask for an additional function call, or ask for a call to an entirely different function. The latter can only happen if you provide more than one function to the model in your function declaration list.

Your app needs to accommodate that the model may ask for additional function calls.

The model may ask to call multiple functions at the same time.

You can provide up to 128 functions in your function declaration list to the model. Given this, the model may decide that multiple functions are needed to help it generate its final response. And it might decide to call some of these functions at the same time – this is called parallel function calling.

Your app needs to accommodate that the model may ask for multiple functions running at the same time, and your app needs to provide all the responses from the functions back to the model.

Parallel function calling is supported by Gemini 1.5 Pro and Gemini 1.5 Flash.

You can control how and if the model can ask to call functions.

You can place some constraints on how and if the model should use the provided function declarations. This is called setting the function calling mode. Here are some examples:

• Instead of allowing the model to choose between an immediate natural language response and a function call, you can force it to always use function calls. This is called forced function calling.

• If you provide multiple function declarations, you can restrict the model to using only a subset of the functions provided.

You implement these constraints (or modes) by adding a tool configuration (toolConfig) along with the prompt and the function declarations. In the tool configuration, you can specify one of the following modes. The most useful mode is ANY.

Function calling mode is supported by Gemini 1.5 Pro and Gemini 1.5 Flash.

What else can you do?

Try out other capabilities of the Gemini API

• Build multi-turn conversations (chat).
• Generate text from text-only prompts.
• Generate text from multimodal prompts (including text, images, PDFs, video, and audio).

Learn how to control content generation

• Understand prompt design, including best practices, strategies, and example prompts.
• Configure model parameters like temperature and maximum output tokens.
• Use safety settings to adjust the likelihood of getting responses that may be considered harmful.

Learn more about the Gemini models

Give feedback about your experience with Vertex AI in Firebase