即使只是基本实现 Live API,您也可以为用户打造极具吸引力且功能强大的互动体验。 您可以选择使用以下配置选项进一步自定义体验:
语音活动检测 (VAD)(例如,用于处理打断)
回答语音和语言
指定回答语音
|
点击您的 Gemini API 提供商,以查看此页面上特定于提供商的内容和代码。 |
Live API 支持 30 种不同的合成高清语音,每种语音都有不同的特点。您可以展开下面的部分,查看回答语音选项列表并试听每种语音。
如果您未指定回答语音,则默认使用 Puck。
如需指定回答语音,请在 speechConfig 对象中设置语音名称,作为模型配置的一部分。
Swift
// ...
let liveModel = FirebaseAI.firebaseAI(backend: .googleAI()).liveModel(
modelName: "GEMINI_LIVE_API_MODEL_NAME",
// Configure the model to use a specific voice for its audio response.
generationConfig: LiveGenerationConfig(
responseModalities: [.audio],
speech: SpeechConfig(voiceName: "VOICE_NAME")
)
)
// ...
Kotlin
// ...
val model = Firebase.ai(backend = GenerativeBackend.googleAI()).liveModel(
modelName = "GEMINI_LIVE_API_MODEL_NAME",
// Configure the model to use a specific voice for its audio response.
generationConfig = liveGenerationConfig {
responseModality = ResponseModality.AUDIO
speechConfig = SpeechConfig(voice = Voice("VOICE_NAME"))
}
)
// ...
Java
// ...
LiveGenerativeModel lm = FirebaseAI.getInstance(GenerativeBackend.googleAI()).liveModel(
"GEMINI_LIVE_API_MODEL_NAME",
// Configure the model to use a specific voice for its audio response.
new LiveGenerationConfig.Builder()
.setResponseModality(ResponseModality.AUDIO)
.setSpeechConfig(new SpeechConfig(new Voice("VOICE_NAME")))
.build()
);
// ...
Web
// ...
const ai = getAI(firebaseApp, { backend: new GoogleAIBackend() });
const liveModel = getLiveGenerativeModel(ai, {
model: "GEMINI_LIVE_API_MODEL_NAME",
// Configure the model to use a specific voice for its audio response.
generationConfig: {
responseModalities: [ResponseModality.AUDIO],
speechConfig: {
voiceConfig: {
prebuiltVoiceConfig: { voiceName: "VOICE_NAME" },
},
},
},
});
// ...
Dart
// ...
final _liveModel = FirebaseAI.googleAI().liveGenerativeModel(
model: 'GEMINI_LIVE_API_MODEL_NAME',
// Configure the model to use a specific voice for its audio response.
liveGenerationConfig: LiveGenerationConfig(
responseModalities: [ResponseModalities.audio],
speechConfig: SpeechConfig(voiceName: 'VOICE_NAME'),
),
);
// ...
Unity
// ...
var liveModel = FirebaseAI.GetInstance(FirebaseAI.Backend.GoogleAI()).GetLiveModel(
modelName: "GEMINI_LIVE_API_MODEL_NAME",
// Configure the model to use a specific voice for its audio response
liveGenerationConfig: new LiveGenerationConfig(
responseModalities: new[] { ResponseModality.Audio },
speechConfig: SpeechConfig.UsePrebuiltVoice("VOICE_NAME")
)
);
// ...
影响回答语言
Live API 模型会自动为其回答选择适当的语言。
如果您希望模型以非英语语言回答,或始终以特定语言回答,可以使用系统指令来影响模型的回答,如以下示例所示:
向模型强调,非英语语言可能更合适
Listen to the speaker carefully. If you detect a non-English language, respond in the language you hear from the speaker. You must respond unmistakably in the speaker's language.让模型始终以特定语言回答
RESPOND IN LANGUAGE. YOU MUST RESPOND UNMISTAKABLY IN LANGUAGE.
音频输入和输出的转写
|
点击您的 Gemini API 提供商,以查看此页面上特定于提供商的内容和代码。 |
作为模型回答的一部分,您可以接收音频输入和模型音频回答的转写。您可以在模型配置中设置此配置。
如需转写音频输入,请添加
inputAudioTranscription。如需转写模型的音频回答,请添加
outputAudioTranscription。
请注意以下几点:
您可以将模型配置为返回输入和输出的转录内容(如以下示例所示),也可以将其配置为仅返回其中一个。
转写内容会与音频一起流式传输,因此最好在每轮对话中像收集文本部分一样收集转写内容。
转写语言是根据音频输入和模型的音频回答推断出来的。
Swift
// ...
let liveModel = FirebaseAI.firebaseAI(backend: .googleAI()).liveModel(
modelName: "GEMINI_LIVE_API_MODEL_NAME",
// Configure the model to return transcriptions of the audio input and output.
generationConfig: LiveGenerationConfig(
responseModalities: [.audio],
inputAudioTranscription: AudioTranscriptionConfig(),
outputAudioTranscription: AudioTranscriptionConfig()
)
)
var inputTranscript: String = ""
var outputTranscript: String = ""
do {
let session = try await liveModel.connect()
for try await response in session.responses {
if case let .content(content) = response.payload {
if let inputText = content.inputAudioTranscription?.text {
// Handle transcription text of the audio input.
inputTranscript += inputText
}
if let outputText = content.outputAudioTranscription?.text {
// Handle transcription text of the audio output.
outputTranscript += outputText
}
if content.isTurnComplete {
// Log the transcripts after the current turn is complete.
print("Input audio: \(inputTranscript)")
print("Output audio: \(outputTranscript)")
// Reset the transcripts for the next turn.
inputTranscript = ""
outputTranscript = ""
}
}
}
} catch {
// Handle error
}
// ...
Kotlin
// ...
val liveModel = Firebase.ai(backend = GenerativeBackend.googleAI()).liveModel(
modelName = "GEMINI_LIVE_API_MODEL_NAME",
// Configure the model to return transcriptions of the audio input and output.
generationConfig = liveGenerationConfig {
responseModality = ResponseModality.AUDIO
inputAudioTranscription = AudioTranscriptionConfig()
outputAudioTranscription = AudioTranscriptionConfig()
}
)
val liveSession = liveModel.connect()
fun handleTranscription(input: Transcription?, output: Transcription?) {
input?.text?.let { text ->
// Handle transcription text of the audio input.
println("Input Transcription: $text")
}
output?.text?.let { text ->
// Handle transcription text of the audio output.
println("Output Transcription: $text")
}
}
liveSession.startAudioConversation(null, ::handleTranscription)
// ...
Java
// ...
ExecutorService executor = Executors.newFixedThreadPool(1);
LiveGenerativeModel lm = FirebaseAI.getInstance(GenerativeBackend.googleAI()).liveModel(
"GEMINI_LIVE_API_MODEL_NAME",
// Configure the model to return transcriptions of the audio input and output.
new LiveGenerationConfig.Builder()
.setResponseModality(ResponseModality.AUDIO)
.setInputAudioTranscription(new AudioTranscriptionConfig())
.setOutputAudioTranscription(new AudioTranscriptionConfig())
.build()
);
LiveModelFutures liveModel = LiveModelFutures.from(lm);
ListenableFuture<LiveSessionFutures> sessionFuture = liveModel.connect();
Futures.addCallback(sessionFuture, new FutureCallback<LiveSessionFutures>() {
@Override
public void onSuccess(LiveSessionFutures ses) {
LiveSessionFutures session = ses;
session.startAudioConversation((Transcription input, Transcription output) -> {
if (input != null) {
// Handle transcription text of the audio input.
System.out.println("Input Transcription: " + input.getText());
}
if (output != null) {
// Handle transcription text of the audio output.
System.out.println("Output Transcription: " + output.getText());
}
return null;
});
}
@Override
public void onFailure(Throwable t) {
// Handle exceptions
t.printStackTrace();
}
}, executor);
// ...
Web
// ...
const ai = getAI(firebaseApp, { backend: new GoogleAIBackend() });
const liveModel = getLiveGenerativeModel(ai, {
model: 'GEMINI_LIVE_API_MODEL_NAME',
// Configure the model to return transcriptions of the audio input and output.
generationConfig: {
responseModalities: [ResponseModality.AUDIO],
inputAudioTranscription: {},
outputAudioTranscription: {},
},
});
const liveSession = await liveModel.connect();
liveSession.sendAudioRealtime({ data, mimeType: "audio/pcm" });
const messages = liveSession.receive();
for await (const message of messages) {
switch (message.type) {
case 'serverContent':
if (message.inputTranscription) {
// Handle transcription text of the audio input.
console.log(`Input transcription: ${message.inputTranscription.text}`);
}
if (message.outputTranscription) {
// Handle transcription text of the audio output.
console.log(`Output transcription: ${message.outputTranscription.text}`);
} else {
// Handle other message types (modelTurn, turnComplete, interruption).
}
default:
// Handle other message types (toolCall, toolCallCancellation).
}
}
// ...
Dart
// ...
final _liveModel = FirebaseAI.googleAI().liveGenerativeModel(
model: 'GEMINI_LIVE_API_MODEL_NAME',
// Configure the model to return transcriptions of the audio input and output.
liveGenerationConfig: LiveGenerationConfig(
responseModalities: [ResponseModalities.audio],
inputAudioTranscription: AudioTranscriptionConfig(),
outputAudioTranscription: AudioTranscriptionConfig(),
),
);
final LiveSession _session = await _liveModel.connect();
await for (final response in _session.receive()) {
if (response.message case LiveServerContent message) {
if (message.inputTranscription?.text case final inputText?) {
// Handle transcription text of the audio input.
print('Input: $inputText');
}
if (message.outputTranscription?.text case final outputText?) {
// Handle transcription text of the audio output.
print('Output: $outputText');
}
}
}
// ...
Unity
// ...
var liveModel = FirebaseAI.GetInstance(FirebaseAI.Backend.GoogleAI()).GetLiveModel(
modelName: "GEMINI_LIVE_API_MODEL_NAME",
// Configure the model to return transcriptions of the audio input and output
liveGenerationConfig: new LiveGenerationConfig(
responseModalities: new[] { ResponseModality.Audio },
inputAudioTranscription: new AudioTranscriptionConfig(),
outputAudioTranscription: new AudioTranscriptionConfig()
)
);
try
{
var session = await liveModel.ConnectAsync();
var stream = session.ReceiveAsync();
await foreach (var response in stream) {
if (response.Message is LiveSessionContent sessionContent) {
if (!string.IsNullOrEmpty(sessionContent.InputTranscription?.Text)) {
// handle transcription text of input audio
}
if (!string.IsNullOrEmpty(sessionContent.OutputTranscription?.Text)) {
// handle transcription text of output audio
}
}
}
}
catch (Exception e)
{
// Handle error
}
// ...
语音活动检测 (VAD)
语音活动检测 (VAD) 可让模型识别用户何时在说话。这对于创建自然对话至关重要,因为这让您可以随时中断模型。VAD 默认处于启用状态。
您可以自定义 VAD 的行为,包括处理中断。您也可以停用自动 VAD,以完全手动控制对话轮次(例如,对于一键通界面)。
配置语音活动检测 (VAD)
|
点击您的 Gemini API 提供商,以查看此页面上特定于提供商的内容和代码。 |
以下示例展示了如何通过将 realtimeInputConfig 设置为模型配置的一部分来自定义 VAD 的行为。请参阅本部分后面表格中此配置的可用设置。
Swift
// ...
let liveModel = FirebaseAI.firebaseAI(backend: .googleAI()).liveModel(
modelName: "GEMINI_LIVE_API_MODEL_NAME",
generationConfig: LiveGenerationConfig(
responseModalities: [.audio],
// Configure voice activity detection (VAD), interruption handling, and turn coverage.
// IMPORTANT: Example values shown here. Make sure to update for your use case.
realtimeInputConfig: RealtimeInputConfig(
automaticActivityDetection: ActivityDetectionConfig(
startSensitivity: .low,
endSensitivity: .low,
prefixPadding: .milliseconds(20),
silenceDuration: .milliseconds(500)
),
activityHandling: .interrupt,
turnCoverage: .onlyActivity
)
)
)
Kotlin
// ...
val liveModel = Firebase.ai(backend = GenerativeBackend.googleAI()).liveModel(
modelName = "GEMINI_LIVE_API_MODEL_NAME",
generationConfig = liveGenerationConfig {
responseModality = ResponseModality.AUDIO
// Configure voice activity detection (VAD), interruption handling, and turn coverage.
// IMPORTANT: Example values shown here. Make sure to update for your use case.
realtimeInputConfig = realtimeInputConfig {
automaticActivityDetection = ActivityDetectionConfig.builder()
.setStartSensitivity(ActivityDetectionConfig.Sensitivity.LOW)
.setEndSensitivity(ActivityDetectionConfig.Sensitivity.LOW)
.setPrefixPaddingMs(20)
.setSilenceDurationMs(500)
.build()
activityHandling = RealtimeInputConfig.ActivityHandling.INTERRUPT
turnCoverage = RealtimeInputConfig.TurnCoverage.ONLY_ACTIVITY
}
}
)
Java
// ...
LiveGenerativeModel lm = FirebaseAI.getInstance(GenerativeBackend.googleAI()).liveModel(
"GEMINI_LIVE_API_MODEL_NAME",
new LiveGenerationConfig.Builder()
.setResponseModality(ResponseModality.AUDIO)
// Configure voice activity detection (VAD), interruption handling, and turn coverage.
// IMPORTANT: Example values shown here. Make sure to update for your use case.
.setRealtimeInputConfig(RealtimeInputConfig.builder()
.setAutomaticActivityDetection(ActivityDetectionConfig.builder()
.setStartSensitivity(ActivityDetectionConfig.Sensitivity.LOW)
.setEndSensitivity(ActivityDetectionConfig.Sensitivity.LOW)
.setPrefixPaddingMs(20)
.setSilenceDurationMs(500)
.build())
.setActivityHandling(RealtimeInputConfig.ActivityHandling.INTERRUPT)
.setTurnCoverage(RealtimeInputConfig.TurnCoverage.ONLY_ACTIVITY)
.build())
.build()
);
Web
Not yet supported for Web apps - check back soon!
Dart
// ...
final _liveModel = FirebaseAI.googleAI().liveGenerativeModel(
model: 'GEMINI_LIVE_API_MODEL_NAME',
liveGenerationConfig: LiveGenerationConfig(
responseModalities: [ResponseModalities.audio],
// Configure voice activity detection (VAD), interruption handling, and turn coverage.
// IMPORTANT: Example values shown here. Make sure to update for your use case.
realtimeInputConfig: RealtimeInputConfig(
automaticActivityDetection: ActivityDetectionConfig(
startSensitivity: Sensitivity.low,
endSensitivity: Sensitivity.low,
prefixPaddingMS: 20,
silenceDurationMS: 500,
),
activityHandling: ActivityHandling.interrupt,
turnCoverage: TurnCoverage.onlyActivity,
),
),
);
Unity
// ...
var liveModel = FirebaseAI.GetInstance(FirebaseAI.Backend.GoogleAI()).GetLiveModel(
modelName: "GEMINI_LIVE_API_MODEL_NAME",
liveGenerationConfig: new LiveGenerationConfig(
responseModalities: new[] { ResponseModality.Audio },
// Configure voice activity detection (VAD), interruption handling, and turn coverage.
// IMPORTANT: Example values shown here. Make sure to update for your use case.
realtimeInputConfig: new RealtimeInputConfig(
automaticActivityDetection: new ActivityDetectionConfig(
startSensitivity: ActivityDetectionConfig.Sensitivity.Low,
endSensitivity: ActivityDetectionConfig.Sensitivity.Low,
prefixPaddingMS: 20,
silenceDurationMS: 500
),
activityHandling: RealtimeInputConfig.ActivityHandling.Interrupt,
turnCoverage: RealtimeInputConfig.TurnCoverage.OnlyActivity
)
)
);
可用的 VAD 配置设置
| 设置 | 支持的值 | 说明 |
|---|---|---|
自动活动检测 (automaticActivityDetection)
|
||
startSensitivity |
|
确定服务器检测用户开始说话的难易程度:
|
endSensitivity |
|
确定服务器检测用户何时停止说话的灵敏度:
|
prefixPadding |
时长(以毫秒为单位) (默认:无 [0 毫秒]) |
定义检测到语音之前要包含的音频量。 此“回溯”缓冲区可确保模型捕获完整的语音开头,包括通常在检测器触发之前开始的第一个音节。
|
silenceDuration |
时长(以毫秒为单位) (默认值:约 800 毫秒) |
定义服务器在结束语音轮次之前等待静默的时间。这决定了系统对自然句中停顿(例如思考、呼吸或子句边界)的容忍度。
|
中断行为 (activityHandling)
|
||
activityHandling |
|
定义服务器在生成响应时如何处理传入的用户活动:
|
转向情境覆盖率 (turnCoverage)
|
||
turnCoverage |
|
定义对话回合上下文中包含哪些流式输入:
|
处理中断
如果服务器在模型生成回答时检测到用户正在说话,并且 activityHandling 设置为 INTERRUPT(默认值),请注意以下行为:
生成已取消:模型会立即停止回答,并舍弃所有待处理的工具调用。
会话历史记录被截断:只有在中断之前已传输到应用的内容才会保留在会话上下文窗口中。
发出中断信号:服务器发送响应消息,指明生成过程已中断(例如 Swift 中的
wasInterrupted,或 Kotlin、Java、Web、Flutter 和 Unity 中的interrupted)。
停用自动语音活动检测 (VAD)
如需完全手动控制对话轮次(例如,对于一键通界面),您可以停用自动语音活动检测 (VAD),而不是配置灵敏度参数。
停用自动检测功能后,服务器不会自动检测语音,也不会根据音频静音触发响应。相反,您的应用必须在活动会话期间手动发出用户活动边界信号。
1. 在模型配置中停用自动 VAD
|
点击您的 Gemini API 提供商,以查看此页面上特定于提供商的内容和代码。 |
Swift
// ...
let liveModel = FirebaseAI.firebaseAI(backend: .googleAI()).liveModel(
modelName: "GEMINI_LIVE_API_MODEL_NAME",
generationConfig: LiveGenerationConfig(
responseModalities: [.audio],
// Disable automatic voice activity detection (VAD).
realtimeInputConfig: RealtimeInputConfig(
automaticActivityDetection: .disabled()
)
)
)
Kotlin
// ...
val liveModel = Firebase.ai(backend = GenerativeBackend.googleAI()).liveModel(
modelName = "GEMINI_LIVE_API_MODEL_NAME",
generationConfig = liveGenerationConfig {
responseModality = ResponseModality.AUDIO
// Disable automatic voice activity detection (VAD).
realtimeInputConfig = realtimeInputConfig {
automaticActivityDetection = ActivityDetectionConfig.disabled()
}
}
)
Java
// ...
LiveGenerativeModel lm = FirebaseAI.getInstance(GenerativeBackend.googleAI()).liveModel(
"GEMINI_LIVE_API_MODEL_NAME",
new LiveGenerationConfig.Builder()
.setResponseModality(ResponseModality.AUDIO)
// Disable automatic voice activity detection (VAD).
.setRealtimeInputConfig(RealtimeInputConfig.builder()
.setAutomaticActivityDetection(ActivityDetectionConfig.disabled())
.build())
.build()
);
Web
Not yet supported for Web apps - check back soon!
Dart
// ...
final _liveModel = FirebaseAI.googleAI().liveGenerativeModel(
model: 'GEMINI_LIVE_API_MODEL_NAME',
liveGenerationConfig: LiveGenerationConfig(
responseModalities: [ResponseModalities.audio],
// Disable automatic voice activity detection (VAD).
realtimeInputConfig: RealtimeInputConfig(
automaticActivityDetection: ActivityDetectionConfig.disabled(),
),
),
);
Unity
// ...
var liveModel = FirebaseAI.GetInstance(FirebaseAI.Backend.GoogleAI()).GetLiveModel(
modelName: "GEMINI_LIVE_API_MODEL_NAME",
liveGenerationConfig: new LiveGenerationConfig(
responseModalities: new[] { ResponseModality.Audio },
// Disable automatic voice activity detection (VAD).
realtimeInputConfig: new RealtimeInputConfig(
automaticActivityDetection: ActivityDetectionConfig.Disabled()
)
)
);
2. 手动发出用户活动边界信号
如果您停用 VAD,您的应用必须手动划分每个用户活动轮次的边界:
发出活动开始信号:在发送音频数据之前(例如,当用户按下“一键通”按钮时)调用
sendStartActivityRealtime。流式传输用户输入:使用
sendAudioRealtime持续流式传输音频(或视频)数据。发出活动结束信号:当用户说完话时(例如,当用户松开一键通按钮时),调用
sendStopActivityRealtime。这表示用户轮次结束,并触发模型的回答。
Swift
// If you disable automatic voice activity detection (VAD), you need to manually demarcate user activity.
// Signal the start of user activity (for example, on push-to-talk button press).
await session.sendStartActivityRealtime()
// Stream user audio data while the user speaks.
await session.sendAudioRealtime(audioData)
// Signal the end of user activity (for example, on button release) to trigger model response.
await session.sendStopActivityRealtime()
Kotlin
// If you disable automatic voice activity detection (VAD), you need to manually demarcate user activity.
// Signal the start of user activity (for example, on push-to-talk button press).
session.sendStartActivityRealtime()
// Stream user audio data while the user speaks.
session.sendAudioRealtime(audioData)
// Signal the end of user activity (for example, on button release) to trigger model response.
session.sendStopActivityRealtime()
Java
// If you disable automatic voice activity detection (VAD), you need to manually demarcate user activity.
// Signal the start of user activity (for example, on push-to-talk button press).
session.sendStartActivityRealtime();
// Stream user audio data while the user speaks.
session.sendAudioRealtime(audioData);
// Signal the end of user activity (for example, on button release) to trigger model response.
session.sendStopActivityRealtime();
Web
Not yet supported for Web apps - check back soon!
Dart
// If you disable automatic voice activity detection (VAD), you need to manually demarcate user activity.
// Signal the start of user activity (for example, on push-to-talk button press).
await session.sendStartActivityRealtime();
// Stream user audio data while the user speaks.
await session.sendAudioRealtime(audioData);
// Signal the end of user activity (for example, on button release) to trigger model response.
await session.sendStopActivityRealtime();
Unity
// If you disable automatic voice activity detection (VAD), you need to manually demarcate user activity.
// Signal the start of user activity (for example, on push-to-talk button press).
await session.SendStartActivityRealtimeAsync();
// Stream user audio data while the user speaks.
await session.SendAudioRealtimeAsync(audio: audioData);
// Signal the end of user activity (for example, on button release) to trigger model response.
await session.SendStopActivityRealtimeAsync();
会话管理
了解以下与会话相关的主题:
高级功能,包括:
与会话相关的限制,包括连接和会话时长限制、会话上下文窗口限制以及速率限制。
处理会话限制的选项,包括: