> [!WARNING] > **Preview** : Using the Firebase AI Logic SDKs with the Gemini Live API is a feature that's in Preview, which means that it isn't subject to any SLA or deprecation policy and could change in backwards-incompatible ways.
Even with the basic implementation for the Live API, you can build engaging and powerful interactions for your users. You can optionally customize the experience even more by using the following configuration options: - [Response voice and language](https://firebase.google.com/docs/ai-logic/live-api/configuration.md.txt#change-voice-and-language) - [Transcriptions for audio input and output](https://firebase.google.com/docs/ai-logic/live-api/configuration.md.txt#transcriptions) - [Voice activity detection (VAD)](https://firebase.google.com/docs/ai-logic/live-api/configuration.md.txt#vad) (for example, to handle interruptions) - [Session management](https://firebase.google.com/docs/ai-logic/live-api/configuration.md.txt#session-management) > [!NOTE] > **Note:** Firebase AI Logic does *not yet* support the following when using the Live API (but they're coming soon!): > Adding a thinking configuration or setting the input media resolution.
*** ** * ** *** ## Response voice and language You can make the model [respond in a specific voice](https://firebase.google.com/docs/ai-logic/live-api/configuration.md.txt#specify-response-voice) and influence the model to [respond in different languages](https://firebase.google.com/docs/ai-logic/live-api/configuration.md.txt#influence-response-language). ### Specify a response voice |---| | *Click your Gemini API provider to view provider-specific content and code on this page.* | The Live API supports 30 different synthisized HD voices, each with distinct characteristics. You can [view a list of response voice options and hear demos of each voice](https://firebase.google.com/docs/ai-logic/live-api/configuration.md.txt#response-voices-list) by expanding the section below. If you don't specify a response voice, the default is `Puck`.
**View a list of response voice options and hear demos**
| Voice name | Description | Gender | Demo | |---|---|---|---| | `Achernar` | Soft | Female | Your browser doesn't support the audio element. | | `Achird` | Friendly | Male | Your browser doesn't support the audio element. | | `Algenib` | Gravelly | Male | Your browser doesn't support the audio element. | | `Algieba` | Smooth | Male | Your browser doesn't support the audio element. | | `Alnilam` | Firm | Male | Your browser doesn't support the audio element. | | `Aoede` | Breezy | Female | Your browser doesn't support the audio element. | | `Autonoe` | Bright | Female | Your browser doesn't support the audio element. | | `Callirrhoe` | Easy-going | Female | Your browser doesn't support the audio element. | | `Charon` | Informative | Male | Your browser doesn't support the audio element. | | `Despina` | Smooth | Female | Your browser doesn't support the audio element. | | `Enceladus` | Breathy | Male | Your browser doesn't support the audio element. | | `Erinome` | Clear | Female | Your browser doesn't support the audio element. | | `Fenrir` | Excitable | Male | Your browser doesn't support the audio element. | | `Gacrux` | Mature | Female | Your browser doesn't support the audio element. | | `Iapetus` | Clear | Male | Your browser doesn't support the audio element. | | `Kore` | Firm | Female | Your browser doesn't support the audio element. | | `Laomedeia` | Upbeat | Female | Your browser doesn't support the audio element. | | `Leda` | Youthful | Female | Your browser doesn't support the audio element. | | `Orus` | Firm | Male | Your browser doesn't support the audio element. | | `Puck` | Upbeat | Male | Your browser doesn't support the audio element. | | `Pulcherrima` | Forward | Female | Your browser doesn't support the audio element. | | `Rasalgethi` | Informative | Male | Your browser doesn't support the audio element. | | `Sadachbia` | Lively | Male | Your browser doesn't support the audio element. | | `Sadaltager` | Knowledgeable | Male | Your browser doesn't support the audio element. | | `Schedar` | Even | Male | Your browser doesn't support the audio element. | | `Sulafat` | Warm | Female | Your browser doesn't support the audio element. | | `Umbriel` | Easy-going | Male | Your browser doesn't support the audio element. | | `Vindemiatrix` | Gentle | Female | Your browser doesn't support the audio element. | | `Zephyr` | Bright | Female | Your browser doesn't support the audio element. | | `Zubenelgenubi` | Casual | Male | Your browser doesn't support the audio element. |

To specify a response voice, set the voice name within the `speechConfig` object as part of the [model configuration](https://firebase.google.com/docs/ai-logic/model-parameters.md.txt#config-gemini-live-api). ### Swift ```swift // ... let liveModel = FirebaseAI.firebaseAI(backend: .googleAI()).liveModel( modelName: "GEMINI_LIVE_API_MODEL_NAME", // Configure the model to use a specific voice for its audio response. generationConfig: LiveGenerationConfig( responseModalities: [.audio], speech: SpeechConfig(voiceName: "VOICE_NAME") ) ) // ... ``` ### Kotlin ```kotlin // ... val model = Firebase.ai(backend = GenerativeBackend.googleAI()).liveModel( modelName = "GEMINI_LIVE_API_MODEL_NAME", // Configure the model to use a specific voice for its audio response. generationConfig = liveGenerationConfig { responseModality = ResponseModality.AUDIO speechConfig = SpeechConfig(voice = Voice("VOICE_NAME")) } ) // ... ``` ### Java ```java // ... LiveGenerativeModel lm = FirebaseAI.getInstance(GenerativeBackend.googleAI()).liveModel( "GEMINI_LIVE_API_MODEL_NAME", // Configure the model to use a specific voice for its audio response. new LiveGenerationConfig.Builder() .setResponseModality(ResponseModality.AUDIO) .setSpeechConfig(new SpeechConfig(new Voice("VOICE_NAME"))) .build() ); // ... ``` ### Web ```javascript // ... const ai = getAI(firebaseApp, { backend: new GoogleAIBackend() }); const liveModel = getLiveGenerativeModel(ai, { model: "GEMINI_LIVE_API_MODEL_NAME", // Configure the model to use a specific voice for its audio response. generationConfig: { responseModalities: [ResponseModality.AUDIO], speechConfig: { voiceConfig: { prebuiltVoiceConfig: { voiceName: "VOICE_NAME" }, }, }, }, }); // ... ``` ### Dart ```dart // ... final _liveModel = FirebaseAI.googleAI().liveGenerativeModel( model: 'GEMINI_LIVE_API_MODEL_NAME', // Configure the model to use a specific voice for its audio response. liveGenerationConfig: LiveGenerationConfig( responseModalities: [ResponseModalities.audio], speechConfig: SpeechConfig(voiceName: 'VOICE_NAME'), ), ); // ... ``` ### Unity ```c# // ... var liveModel = FirebaseAI.GetInstance(FirebaseAI.Backend.GoogleAI()).GetLiveModel( modelName: "GEMINI_LIVE_API_MODEL_NAME", // Configure the model to use a specific voice for its audio response liveGenerationConfig: new LiveGenerationConfig( responseModalities: new[] { ResponseModality.Audio }, speechConfig: SpeechConfig.UsePrebuiltVoice("VOICE_NAME") ) ); // ... ``` ### Influence the response language > [!IMPORTANT] > **Important:** The Live API models do *not* support explicitly setting a response *language* in the configuration. Instead, these models automatically choose the appropriate language for their responses. The Live API models automatically choose the appropriate language for their responses.
View list of supported languages
##### Languages supported by all audio-generating models | Language | BCP-47 Code | Language | BCP-47 Code | |---|---|---|---| | Arabic (Egyptian) | ar-EG | German (Germany) | de-DE | | English (US) | en-US | Spanish (US) | es-US | | French (France) | fr-FR | Hindi (India) | hi-IN | | Indonesian (Indonesia) | id-ID | Italian (Italy) | it-IT | | Japanese (Japan) | ja-JP | Korean (Korea) | ko-KR | | Portuguese (Brazil) | pt-BR | Russian (Russia) | ru-RU | | Dutch (Netherlands) | nl-NL | Polish (Poland) | pl-PL | | Thai (Thailand) | th-TH | Turkish (Turkey) | tr-TR | | Vietnamese (Vietnam) | vi-VN | Romanian (Romania) | ro-RO | | Ukrainian (Ukraine) | uk-UA | Bengali (Bangladesh) | bn-BD | | English (India) | en-IN \& hi-IN bundle | Marathi (India) | mr-IN | | Tamil (India) | ta-IN | Telugu (India) | te-IN | ##### Additional languages supported by audio-generating 3.x models | Language | BCP-47 Code | Language | BCP-47 Code | |---|---|---|---| | Afrikaans | af | Filipino | fil | | Albanian | sq | Finnish | fi | | Amharic | am | Galician | gl | | Armenian | hy | Georgian | ka | | Azerbaijani | az | Greek | el | | Basque | eu | Gujarati | gu | | Belarusian | be | Haitian Creole | ht | | Bulgarian | bg | Hebrew | he | | Burmese | my | Hungarian | hu | | Catalan | ca | Icelandic | is | | Cebuano | ceb | Javanese | jv | | Chinese, Mandarin | cmn | Kannada | kn | | Croatian | hr | Konkani | kok | | Czech | cs | Lao | lo | | Danish | da | Latin | la | | Estonian | et | Latvian | lv | | Lithuanian | lt | Luxembourgish | lb | | Macedonian | mk | Maithili | mai | | Malagasy | mg | Malay | ms | | Malayalam | ml | Mongolian | mn | | Nepali | ne | Norwegian, Bokmål | nb | | Norwegian, Nynorsk | nn | Odia | or | | Pashto | ps | Persian | fa | | Punjabi | pa | Serbian | sr | | Sindhi | sd | Sinhala | si | | Slovak | sk | Slovenian | sl | | Swahili | sw | Swedish | sv | | Urdu | ur | | |

If you want the model to respond in a non-English language or always in a specific language, you can use influence the model's responses by using [system instructions](https://firebase.google.com/docs/ai-logic/system-instructions.md.txt) like these examples: - Reinforce to the model that a non-English language may be appropriate ```text Listen to the speaker carefully. If you detect a non-English language, respond in the language you hear from the speaker. You must respond unmistakably in the speaker's language. ``` - Tell the model to always respond in a specific language ```text RESPOND IN LANGUAGE. YOU MUST RESPOND UNMISTAKABLY IN LANGUAGE. ```
*** ** * ** *** ## Transcriptions for audio input and output |---| | *Click your Gemini API provider to view provider-specific content and code on this page.* | As part of the model's response, you can receive transcriptions of the audio input and the model's audio response. You set this configuration as part of the [model configuration](https://firebase.google.com/docs/ai-logic/model-parameters.md.txt#config-gemini-live-api). - For transcription of the audio input, add `inputAudioTranscription`. - For transcription of the model's audio response, add `outputAudioTranscription`. Note the following: - You can configure the model to return transcriptions of both input and output (as shown in the following example), or you can configure it to return only one or the other. - The transcripts are streamed along with the audio, so it's best to collect them like you do text parts with each turn. - The transcription language is inferred from the audio input and the model's audio response. ### Swift ```swift // ... let liveModel = FirebaseAI.firebaseAI(backend: .googleAI()).liveModel( modelName: "GEMINI_LIVE_API_MODEL_NAME", // Configure the model to return transcriptions of the audio input and output. generationConfig: LiveGenerationConfig( responseModalities: [.audio], inputAudioTranscription: AudioTranscriptionConfig(), outputAudioTranscription: AudioTranscriptionConfig() ) ) var inputTranscript: String = "" var outputTranscript: String = "" do { let session = try await liveModel.connect() for try await response in session.responses { if case let .content(content) = response.payload { if let inputText = content.inputAudioTranscription?.text { // Handle transcription text of the audio input. inputTranscript += inputText } if let outputText = content.outputAudioTranscription?.text { // Handle transcription text of the audio output. outputTranscript += outputText } if content.isTurnComplete { // Log the transcripts after the current turn is complete. print("Input audio: \(inputTranscript)") print("Output audio: \(outputTranscript)") // Reset the transcripts for the next turn. inputTranscript = "" outputTranscript = "" } } } } catch { // Handle error } // ... ``` ### Kotlin ```kotlin // ... val liveModel = Firebase.ai(backend = GenerativeBackend.googleAI()).liveModel( modelName = "GEMINI_LIVE_API_MODEL_NAME", // Configure the model to return transcriptions of the audio input and output. generationConfig = liveGenerationConfig { responseModality = ResponseModality.AUDIO inputAudioTranscription = AudioTranscriptionConfig() outputAudioTranscription = AudioTranscriptionConfig() } ) val liveSession = liveModel.connect() fun handleTranscription(input: Transcription?, output: Transcription?) { input?.text?.let { text -> // Handle transcription text of the audio input. println("Input Transcription: $text") } output?.text?.let { text -> // Handle transcription text of the audio output. println("Output Transcription: $text") } } liveSession.startAudioConversation(null, ::handleTranscription) // ... ``` ### Java ```java // ... ExecutorService executor = Executors.newFixedThreadPool(1); LiveGenerativeModel lm = FirebaseAI.getInstance(GenerativeBackend.googleAI()).liveModel( "GEMINI_LIVE_API_MODEL_NAME", // Configure the model to return transcriptions of the audio input and output. new LiveGenerationConfig.Builder() .setResponseModality(ResponseModality.AUDIO) .setInputAudioTranscription(new AudioTranscriptionConfig()) .setOutputAudioTranscription(new AudioTranscriptionConfig()) .build() ); LiveModelFutures liveModel = LiveModelFutures.from(lm); ListenableFuture sessionFuture = liveModel.connect(); Futures.addCallback(sessionFuture, new FutureCallback() { @Override public void onSuccess(LiveSessionFutures ses) { LiveSessionFutures session = ses; session.startAudioConversation((Transcription input, Transcription output) -> { if (input != null) { // Handle transcription text of the audio input. System.out.println("Input Transcription: " + input.getText()); } if (output != null) { // Handle transcription text of the audio output. System.out.println("Output Transcription: " + output.getText()); } return null; }); } @Override public void onFailure(Throwable t) { // Handle exceptions t.printStackTrace(); } }, executor); // ... ``` ### Web ```javascript // ... const ai = getAI(firebaseApp, { backend: new GoogleAIBackend() }); const liveModel = getLiveGenerativeModel(ai, { model: 'GEMINI_LIVE_API_MODEL_NAME', // Configure the model to return transcriptions of the audio input and output. generationConfig: { responseModalities: [ResponseModality.AUDIO], inputAudioTranscription: {}, outputAudioTranscription: {}, }, }); const liveSession = await liveModel.connect(); liveSession.sendAudioRealtime({ data, mimeType: "audio/pcm" }); const messages = liveSession.receive(); for await (const message of messages) { switch (message.type) { case 'serverContent': if (message.inputTranscription) { // Handle transcription text of the audio input. console.log(`Input transcription: ${message.inputTranscription.text}`); } if (message.outputTranscription) { // Handle transcription text of the audio output. console.log(`Output transcription: ${message.outputTranscription.text}`); } else { // Handle other message types (modelTurn, turnComplete, interruption). } default: // Handle other message types (toolCall, toolCallCancellation). } } // ... ``` ### Dart ```dart // ... final _liveModel = FirebaseAI.googleAI().liveGenerativeModel( model: 'GEMINI_LIVE_API_MODEL_NAME', // Configure the model to return transcriptions of the audio input and output. liveGenerationConfig: LiveGenerationConfig( responseModalities: [ResponseModalities.audio], inputAudioTranscription: AudioTranscriptionConfig(), outputAudioTranscription: AudioTranscriptionConfig(), ), ); final LiveSession _session = await _liveModel.connect(); await for (final response in _session.receive()) { if (response.message case LiveServerContent message) { if (message.inputTranscription?.text case final inputText?) { // Handle transcription text of the audio input. print('Input: $inputText'); } if (message.outputTranscription?.text case final outputText?) { // Handle transcription text of the audio output. print('Output: $outputText'); } } } // ... ``` ### Unity ```c# // ... var liveModel = FirebaseAI.GetInstance(FirebaseAI.Backend.GoogleAI()).GetLiveModel( modelName: "GEMINI_LIVE_API_MODEL_NAME", // Configure the model to return transcriptions of the audio input and output liveGenerationConfig: new LiveGenerationConfig( responseModalities: new[] { ResponseModality.Audio }, inputAudioTranscription: new AudioTranscriptionConfig(), outputAudioTranscription: new AudioTranscriptionConfig() ) ); try { var session = await liveModel.ConnectAsync(); var stream = session.ReceiveAsync(); await foreach (var response in stream) { if (response.Message is LiveSessionContent sessionContent) { if (!string.IsNullOrEmpty(sessionContent.InputTranscription?.Text)) { // handle transcription text of input audio } if (!string.IsNullOrEmpty(sessionContent.OutputTranscription?.Text)) { // handle transcription text of output audio } } } } catch (Exception e) { // Handle error } // ... ```
*** ** * ** *** ## Voice activity detection (VAD) Voice activity detection (VAD) lets the model recognize when a person is speaking. This is essential for creating natural conversations, as it lets you interrupt the model at any time. **VAD is enabled by default.** You can [customize the behavior of VAD](https://firebase.google.com/docs/ai-logic/live-api/configuration.md.txt#vad-configure), including [handling interruptions](https://firebase.google.com/docs/ai-logic/live-api/configuration.md.txt#vad-handle-interruptions). You can alternatively [disable automatic VAD](https://firebase.google.com/docs/ai-logic/live-api/configuration.md.txt#vad-disable) to take complete manual control over conversational turns (such as for a push-to-talk interface). ### Configure voice activity detection (VAD) |---| | *Click your Gemini API provider to view provider-specific content and code on this page.* | The following example shows how you can customize the behavior of VAD by setting `realtimeInputConfig` as part of the [model configuration](https://firebase.google.com/docs/ai-logic/model-parameters.md.txt#config-gemini-live-api). Learn about [available settings for this configuration](https://firebase.google.com/docs/ai-logic/live-api/configuration.md.txt#vad-configuration-settings) in the table later in this section. ### Swift ```swift // ... let liveModel = FirebaseAI.firebaseAI(backend: .googleAI()).liveModel( modelName: "GEMINI_LIVE_API_MODEL_NAME", generationConfig: LiveGenerationConfig( responseModalities: [.audio], // Configure voice activity detection (VAD), interruption handling, and turn coverage. // IMPORTANT: Example values shown here. Make sure to update for your use case. realtimeInputConfig: RealtimeInputConfig( automaticActivityDetection: ActivityDetectionConfig( startSensitivity: .low, endSensitivity: .low, prefixPadding: .milliseconds(20), silenceDuration: .milliseconds(500) ), activityHandling: .interrupt, turnCoverage: .onlyActivity ) ) ) ``` ### Kotlin ```kotlin // ... val liveModel = Firebase.ai(backend = GenerativeBackend.googleAI()).liveModel( modelName = "GEMINI_LIVE_API_MODEL_NAME", generationConfig = liveGenerationConfig { responseModality = ResponseModality.AUDIO // Configure voice activity detection (VAD), interruption handling, and turn coverage. // IMPORTANT: Example values shown here. Make sure to update for your use case. realtimeInputConfig = realtimeInputConfig { automaticActivityDetection = ActivityDetectionConfig.builder() .setStartSensitivity(ActivityDetectionConfig.Sensitivity.LOW) .setEndSensitivity(ActivityDetectionConfig.Sensitivity.LOW) .setPrefixPaddingMs(20) .setSilenceDurationMs(500) .build() activityHandling = RealtimeInputConfig.ActivityHandling.INTERRUPT turnCoverage = RealtimeInputConfig.TurnCoverage.ONLY_ACTIVITY } } ) ``` ### Java ```java // ... LiveGenerativeModel lm = FirebaseAI.getInstance(GenerativeBackend.googleAI()).liveModel( "GEMINI_LIVE_API_MODEL_NAME", new LiveGenerationConfig.Builder() .setResponseModality(ResponseModality.AUDIO) // Configure voice activity detection (VAD), interruption handling, and turn coverage. // IMPORTANT: Example values shown here. Make sure to update for your use case. .setRealtimeInputConfig(RealtimeInputConfig.builder() .setAutomaticActivityDetection(ActivityDetectionConfig.builder() .setStartSensitivity(ActivityDetectionConfig.Sensitivity.LOW) .setEndSensitivity(ActivityDetectionConfig.Sensitivity.LOW) .setPrefixPaddingMs(20) .setSilenceDurationMs(500) .build()) .setActivityHandling(RealtimeInputConfig.ActivityHandling.INTERRUPT) .setTurnCoverage(RealtimeInputConfig.TurnCoverage.ONLY_ACTIVITY) .build()) .build() ); ``` ### Web ```javascript Not yet supported for Web apps - check back soon! ``` ### Dart ```dart // ... final _liveModel = FirebaseAI.googleAI().liveGenerativeModel( model: 'GEMINI_LIVE_API_MODEL_NAME', liveGenerationConfig: LiveGenerationConfig( responseModalities: [ResponseModalities.audio], // Configure voice activity detection (VAD), interruption handling, and turn coverage. // IMPORTANT: Example values shown here. Make sure to update for your use case. realtimeInputConfig: RealtimeInputConfig( automaticActivityDetection: ActivityDetectionConfig( startSensitivity: Sensitivity.low, endSensitivity: Sensitivity.low, prefixPaddingMS: 20, silenceDurationMS: 500, ), activityHandling: ActivityHandling.interrupt, turnCoverage: TurnCoverage.onlyActivity, ), ), ); ``` ### Unity ```c# // ... var liveModel = FirebaseAI.GetInstance(FirebaseAI.Backend.GoogleAI()).GetLiveModel( modelName: "GEMINI_LIVE_API_MODEL_NAME", liveGenerationConfig: new LiveGenerationConfig( responseModalities: new[] { ResponseModality.Audio }, // Configure voice activity detection (VAD), interruption handling, and turn coverage. // IMPORTANT: Example values shown here. Make sure to update for your use case. realtimeInputConfig: new RealtimeInputConfig( automaticActivityDetection: new ActivityDetectionConfig( startSensitivity: ActivityDetectionConfig.Sensitivity.Low, endSensitivity: ActivityDetectionConfig.Sensitivity.Low, prefixPaddingMS: 20, silenceDurationMS: 500 ), activityHandling: RealtimeInputConfig.ActivityHandling.Interrupt, turnCoverage: RealtimeInputConfig.TurnCoverage.OnlyActivity ) ) ); ``` #### Available VAD configuration settings | Setting | Supported values | Description | |---|---|---| | Automatic activity detection (`automaticActivityDetection`) ||| | `startSensitivity` | - `LOW` (default) - `HIGH` | Determines how readily the server detects when a user *begins* speaking: - `LOW`: Less sensitive to faint sounds. This helps to prevent ambient noise from inadvertently starting a speech turn. - `HIGH`: More sensitive. This helps to quickly detect quieter speech or soft-spoken words. | | `endSensitivity` | - `LOW` (default) - `HIGH` | Determines how readily the server detects when a user *stops* speaking: - `LOW`: Less sensitive to trailing pauses. The server accommodates natural hesitations or quieter words at the end of a sentence. - `HIGH`: More sensitive. The server concludes the speech turn promptly once voice levels drop. | | `prefixPadding` | Duration in milliseconds (default: None (0 ms)) | Defines the amount of audio to include *before* speech is detected. This "look-back" buffer ensures the model captures the full onset of speech, including the first syllable which often starts before the detector triggers. - The default value of `0` can cause the beginning of words to be clipped. - **Recommended**: 20-50 milliseconds | | `silenceDuration` | Duration in milliseconds (default: \~800 ms) | Defines how long the server waits through silence before ending a speech turn. This determines how tolerant the system is of natural mid-sentence pauses (such as thinking, breathing, or clause boundaries). - **Recommended (500-800 ms)**: Provides a good balance---the model receives complete, contextually rich audio while keeping response latency reasonable. - **Too low (below 200 ms)**: Ends speech turns during natural pauses, splitting utterances into small fragments that can degrade transcription and response quality. - **Too high (above 2000 ms)**: Adds noticeable delay before the model begins responding. | | Interruption behavior (`activityHandling`) ||| | `activityHandling` | - `INTERRUPT` (default) - `NO_INTERRUPT` | Defines how the server handles incoming user activity while generating a response: - `INTERRUPT`: Immediately cancels the model's current response when user activity is detected. - `NO_INTERRUPT`: Continues generating the response without interruption. | | Turn context coverage (`turnCoverage`) ||| | `turnCoverage` | - `ONLY_ACTIVITY` (default) - `ALL_INPUT` - `AUDIO_ACTIVITY_AND_ALL_VIDEO` | Defines what streamed input is included in the conversational turn context: - `ONLY_ACTIVITY`: Excludes silence, retaining only segments where user activity was detected. - `ALL_INPUT`: Retains all streamed input, including audio silence. - `AUDIO_ACTIVITY_AND_ALL_VIDEO`: Retains all streamed video frames, but only audio segments where activity was detected. | ### Handle interruptions If the server detects incoming user speech while the model is generating a response and `activityHandling` is set to `INTERRUPT` (the default), be aware of the following behaviors: - **Generation is canceled**: The model immediately halts its response and discards any pending tool calls. - **Session history is truncated**: Only the content already transmitted to your app before the interruption is preserved in the session context window. - **An interruption signal is emitted** : The server sends a response message indicating that generation was interrupted (such as `wasInterrupted` in Swift or `interrupted` in Kotlin, Java, Web, Flutter, and Unity). > [!IMPORTANT] > **Important:** If your app buffers streamed audio for real-time playback, check for the interruption signal in your response listener. If an interruption signal is detected, your app should immediately stop audio playback and clear any queued audio buffers so the user doesn't hear stale audio overlapping with the model's new turn. ### Disable automatic voice activity detection (VAD) To take complete manual control over conversational turns (such as for a push-to-talk interface), you can disable automatic voice activity detection (VAD) instead of configuring sensitivity parameters. When automatic detection is disabled, the server won't automatically detect speech or trigger responses based on audio silence. Instead, your app must manually signal user activity boundaries during the active session. #### 1. Disable automatic VAD in the model configuration |---| | *Click your Gemini API provider to view provider-specific content and code on this page.* | ### Swift ```swift // ... let liveModel = FirebaseAI.firebaseAI(backend: .googleAI()).liveModel( modelName: "GEMINI_LIVE_API_MODEL_NAME", generationConfig: LiveGenerationConfig( responseModalities: [.audio], // Disable automatic voice activity detection (VAD). realtimeInputConfig: RealtimeInputConfig( automaticActivityDetection: .disabled() ) ) ) ``` ### Kotlin ```kotlin // ... val liveModel = Firebase.ai(backend = GenerativeBackend.googleAI()).liveModel( modelName = "GEMINI_LIVE_API_MODEL_NAME", generationConfig = liveGenerationConfig { responseModality = ResponseModality.AUDIO // Disable automatic voice activity detection (VAD). realtimeInputConfig = realtimeInputConfig { automaticActivityDetection = ActivityDetectionConfig.disabled() } } ) ``` ### Java ```java // ... LiveGenerativeModel lm = FirebaseAI.getInstance(GenerativeBackend.googleAI()).liveModel( "GEMINI_LIVE_API_MODEL_NAME", new LiveGenerationConfig.Builder() .setResponseModality(ResponseModality.AUDIO) // Disable automatic voice activity detection (VAD). .setRealtimeInputConfig(RealtimeInputConfig.builder() .setAutomaticActivityDetection(ActivityDetectionConfig.disabled()) .build()) .build() ); ``` ### Web ```javascript Not yet supported for Web apps - check back soon! ``` ### Dart ```dart // ... final _liveModel = FirebaseAI.googleAI().liveGenerativeModel( model: 'GEMINI_LIVE_API_MODEL_NAME', liveGenerationConfig: LiveGenerationConfig( responseModalities: [ResponseModalities.audio], // Disable automatic voice activity detection (VAD). realtimeInputConfig: RealtimeInputConfig( automaticActivityDetection: ActivityDetectionConfig.disabled(), ), ), ); ``` ### Unity ```c# // ... var liveModel = FirebaseAI.GetInstance(FirebaseAI.Backend.GoogleAI()).GetLiveModel( modelName: "GEMINI_LIVE_API_MODEL_NAME", liveGenerationConfig: new LiveGenerationConfig( responseModalities: new[] { ResponseModality.Audio }, // Disable automatic voice activity detection (VAD). realtimeInputConfig: new RealtimeInputConfig( automaticActivityDetection: ActivityDetectionConfig.Disabled() ) ) ); ``` #### 2. Manually signal user activity boundaries If you disable VAD, your app must manually demarcate the boundaries of each user activity turn: 1. **Signal activity start** : Call `sendStartActivityRealtime` before sending audio data (for example, when a user presses a push-to-talk button). 2. **Stream user input** : Continuously stream audio (or video) data using `sendAudioRealtime`. 3. **Signal activity end** : Call `sendStopActivityRealtime` when the user finishes speaking (for example, when the user releases the push-to-talk button). This signals the end of the user's turn and triggers the model's response. > [!NOTE] > **Note:** When automatic VAD is disabled, the server acts immediately on `sendStopActivityRealtime` with no additional wait or silence tolerance. If you use an automated client-side voice detector instead of a physical push-to-talk button, ensure your detector uses a silence threshold of at least 500 ms before calling `sendStopActivityRealtime` to avoid cutting off speech during natural pauses. ### Swift ```swift // If you disable automatic voice activity detection (VAD), you need to manually demarcate user activity. // Signal the start of user activity (for example, on push-to-talk button press). await session.sendStartActivityRealtime() // Stream user audio data while the user speaks. await session.sendAudioRealtime(audioData) // Signal the end of user activity (for example, on button release) to trigger model response. await session.sendStopActivityRealtime() ``` ### Kotlin ```kotlin // If you disable automatic voice activity detection (VAD), you need to manually demarcate user activity. // Signal the start of user activity (for example, on push-to-talk button press). session.sendStartActivityRealtime() // Stream user audio data while the user speaks. session.sendAudioRealtime(audioData) // Signal the end of user activity (for example, on button release) to trigger model response. session.sendStopActivityRealtime() ``` ### Java ```java // If you disable automatic voice activity detection (VAD), you need to manually demarcate user activity. // Signal the start of user activity (for example, on push-to-talk button press). session.sendStartActivityRealtime(); // Stream user audio data while the user speaks. session.sendAudioRealtime(audioData); // Signal the end of user activity (for example, on button release) to trigger model response. session.sendStopActivityRealtime(); ``` ### Web ```javascript Not yet supported for Web apps - check back soon! ``` ### Dart ```dart // If you disable automatic voice activity detection (VAD), you need to manually demarcate user activity. // Signal the start of user activity (for example, on push-to-talk button press). await session.sendStartActivityRealtime(); // Stream user audio data while the user speaks. await session.sendAudioRealtime(audioData); // Signal the end of user activity (for example, on button release) to trigger model response. await session.sendStopActivityRealtime(); ``` ### Unity ```c# // If you disable automatic voice activity detection (VAD), you need to manually demarcate user activity. // Signal the start of user activity (for example, on push-to-talk button press). await session.SendStartActivityRealtimeAsync(); // Stream user audio data while the user speaks. await session.SendAudioRealtimeAsync(audio: audioData); // Signal the end of user activity (for example, on button release) to trigger model response. await session.SendStopActivityRealtimeAsync(); ```
*** ** * ** *** ## Session management Learn about the following sessions-related topics: - Advanced capabilities, including: - [Updating system instructions mid-session](https://firebase.google.com/docs/ai-logic/live-api/sessions.md.txt#update-system-instructions-mid-session) - [Adding incremental content updates](https://firebase.google.com/docs/ai-logic/live-api/sessions.md.txt#add-incremental-content-updates) - [Session-related limits](https://firebase.google.com/docs/ai-logic/live-api/sessions.md.txt#limits), including connection and session length limits, session context window limits, and rate limits. - Options for handling session limits, including: - [Compressing the context window](https://firebase.google.com/docs/ai-logic/live-api/sessions.md.txt#compress-context) - [Resuming a session](https://firebase.google.com/docs/ai-logic/live-api/sessions.md.txt#resume-session)