Many of your users will have interacted with large language models for the first time through chatbots. Although LLMs are capable of much more than simulating conversations, it remains a familiar and useful style of interaction. Even when your users will not be interacting directly with the model in this way, the conversational style of prompting is a powerful way to influence the output generated by an AI model.
To support this style of interaction, Genkit provides a set of interfaces and abstractions that make it easier for you to build chat-based LLM applications.
Before you begin
Before reading this page, you should be familiar with the content covered on the Generating content with AI models page.
If you want to run the code examples on this page, first complete the steps in the Getting started guide. All of the examples assume that you have already installed Genkit as a dependency in your project.
Chat session basics
Here is a minimal, console-based, chatbot application:
import { genkit } from "genkit";
import { googleAI, gemini15Flash } from "@genkit-ai/googleai";
import { createInterface } from "node:readline/promises";
const ai = genkit({
plugins: [googleAI()],
model: gemini15Flash,
});
(async () => {
const chat = ai.chat();
console.log("You're chatting with Gemini. Ctrl-C to quit.\n");
const readline = createInterface(process.stdin, process.stdout);
while (true) {
const userInput = await readline.question("> ");
const { text } = await chat.send(userInput);
console.log(text);
}
})();
A chat session with this program looks something like the following example:
You're chatting with Gemini. Ctrl-C to quit.
> hi
Hi there! How can I help you today?
> my name is pavel
Nice to meet you, Pavel! What can I do for you today?
> what's my name?
Your name is Pavel! I remembered it from our previous interaction.
Is there anything else I can help you with?
As you can see from this brief interaction, when you send a message to a chat session, the model can make use of the session so far in its responses. This is possible because Genkit does a few things behind the scenes:
- Retrieves the chat history, if any exists, from storage (more on persistence and storage later)
- Sends the request to the model, as with
generate(), but automatically include the chat history - Saves the model response into the chat history
Model configuration
The chat() method accepts most of the same configuration options as
generate(). To pass configuration options to the model:
const chat = ai.chat({
model: gemini15Pro,
system:
"You're a pirate first mate. Address the user as Captain and assist " +
"them however you can.",
config: {
temperature: 1.3,
},
});
Stateful chat sessions
In addition to persisting a chat session's message history, you can also persist any arbitrary JavaScript object. Doing so can let you manage state in a more structured way then relying only on information in the message history.
To include state in a session, you need to instantiate a session explicitly:
interface MyState {
userName: string;
}
const session = ai.createSession<MyState>({
initialState: {
userName: 'Pavel',
},
});
You can then start a chat within the session:
const chat = session.chat();
To modify the session state based on how the chat unfolds, define tools and include them with your requests:
const changeUserName = ai.defineTool(
{
name: 'changeUserName',
description: 'can be used to change user name',
inputSchema: z.object({
newUserName: z.string(),
}),
},
async (input) => {
await ai.currentSession<MyState>().updateState({
userName: input.newUserName,
});
return `changed username to ${input.newUserName}`;
}
);
const chat = session.chat({
model: gemini15Pro,
tools: [changeUserName],
});
await chat.send('change user name to Kevin');
Multi-thread sessions
A single session can contain multiple chat threads. Each thread has its own message history, but they share a single session state.
const lawyerChat = session.chat('lawyerThread', {
system: 'talk like a lawyer',
});
const pirateChat = session.chat('pirateThread', {
system: 'talk like a pirate',
});
Session persistence (EXPERIMENTAL)
When you initialize a new chat or session, it's configured by default to store the session in memory only. This is adequate when the session needs to persist only for the duration of a single invocation of your program, as in the sample chatbot from the beginning of this page. However, when integrating LLM chat into an application, you will usually deploy your content generation logic as stateless web API endpoints. For persistent chats to work under this setup, you will need to implement some kind of session storage that can persist state across invocations of your endpoints.
To add persistence to a chat session, you need to implement Genkit's
SessionStore interface. Here is an example implementation that saves session
state to individual JSON files:
class JsonSessionStore<S = any> implements SessionStore<S> {
async get(sessionId: string): Promise<SessionData<S> | undefined> {
try {
const s = await readFile(`${sessionId}.json`, { encoding: 'utf8' });
const data = JSON.parse(s);
return data;
} catch {
return undefined;
}
}
async save(sessionId: string, sessionData: SessionData<S>): Promise<void> {
const s = JSON.stringify(sessionData);
await writeFile(`${sessionId}.json`, s, { encoding: 'utf8' });
}
}
This implementation is probably not adequate for practical deployments, but it illustrates that a session storage implementation only needs to accomplish two tasks:
- Get a session object from storage using its session ID
- Save a given session object, indexed by its session ID
Once you've implemented the interface for your storage backend, pass an instance of your implementation to the session constructors:
// To create a new session:
const session = ai.createSession({
store: new JsonSessionStore(),
});
// Save session.id so you can restore the session the next time the
// user makes a request.
// If the user has a session ID saved, load the session instead of creating
// a new one:
const session = await ai.loadSession(sessionId, {
store: new JsonSessionStore(),
});