Step-by-Step Guide to Building an AI-Powered Chatbot with Gemini
The gap between a chatbot demo and a chatbot in production is not the model — it is everything wrapped around the model call. A demo sends one message and prints one reply. A production bot has to remember the last ten turns, decide when to call a real API instead of guessing an answer, stream tokens so the UI doesn't sit frozen for four seconds, and still respond sensibly when the model call times out mid-stream.
This guide builds one concrete bot — an order-status assistant that answers "where is order A1092" by calling a real lookup function instead of hallucinating a tracking number — using the Gemini API. Each section adds one production concern on top of the last: first a single-turn call, then conversation history, then a function-calling tool, then the streaming and failure-handling that make the difference between a toy and something you'd point real users at. For the retrieval-augmented variant of this pattern, see ChatGPT-style RAG with LangChain.
The smallest working call
Before adding history, tools, or streaming, get one request working end to end: a system instruction that defines the bot's persona and boundaries, one user message, one generated reply. This is the skeleton every later section extends — get the request/response shape right here and the rest is additive.
The system instruction matters more than most demos suggest. Without an explicit boundary ("you do not have access to order data unless a tool result is provided"), the model will happily invent a plausible-looking tracking number rather than say it doesn't know — a hallucination that is worse than silence because it looks authoritative.
Quick reference
- Set the system instruction on every call, not just the first — some SDKs default to per-request state.
- Treat "the model refuses to guess" as the correct behavior to test for, not a bug to prompt away.
gemini-2.5-flashfavors latency; use a Pro-tier model only where the answer quality gap is measured, not assumed.- Log the raw request/response pair in development — most "the model is wrong" bugs are actually a missing or malformed system instruction.
Remember this
A chatbot without an explicit "you don't know unless a tool told you" boundary will hallucinate plausible answers instead of admitting uncertainty — that boundary is a prompt-engineering decision, not a model capability.
Conversation state and the session boundary
A chatbot needs memory across turns, but that memory has an ownership question that demos skip: who stores it, and what happens when the process restarts? For a single-user CLI demo, an in-memory array is fine. For a multi-user web app, conversation history belongs in a session store keyed by user ID, with a bounded window — sending the entire history on every call both costs tokens and eventually exceeds the context window.
Gemini's SDK models a chat session that appends turns for you, but that convenience hides the ownership question rather than answering it: if the process restarts, an in-memory chat session's history is gone. Decide explicitly whether that's acceptable (support bot, low stakes) or not (an agent mid-way through a multi-step task).
Quick reference
- Store conversation history server-side keyed by session/user ID — never trust a client to send an accurate history back.
- Bound the window (e.g. last 10 turns or last N tokens) and summarize older turns instead of growing unbounded.
- Decide the restart contract up front: acceptable data loss vs. a persisted store (Redis, Postgres) that survives a redeploy.
- A stale session (user closed the tab three days ago) should expire, not silently resume with context the user has forgotten giving.
- For the token-budget side of this same window-and-summarize trade-off, see Context window management for LLM apps.
Remember this
Conversation memory is a state-ownership decision, not a convenience API — decide who stores history, how far back it extends, and what a process restart is allowed to lose before writing the first line of the chat loop.
Function calling: answering with real data
The order-status bot only becomes trustworthy once it can call a real function instead of describing what an order status API might return. Gemini's function calling works by declaring a tool schema; the model emits a functionCall part instead of text when it decides a tool is needed, your code executes the real function, and you send the result back for the model to compose into a final reply.
The failure case to design for up front is a tool that times out or throws. If the handler doesn't catch that, the chat loop hangs waiting for a functionResponse that never arrives — from the user's perspective, indistinguishable from the app being broken.
Quick reference
- Always send a
functionResponseback, even on failure — an explicit error object, never a silent drop. - Bound every tool call with a timeout; an unbounded external call becomes an unbounded chat response time.
- Validate
functionCallarguments before calling your API — the model can pass a malformed order ID. - Log every tool call and its result for debugging — most "the bot answered wrong" reports are actually a tool that returned bad data.
Remember this
An unhandled tool timeout doesn't produce a wrong answer — it produces a hung stream, which is worse; every tool call needs a bounded timeout and an explicit error path back to the model.
Streaming and the user-facing failure path
Streaming tokens as they arrive is what makes a four-second response feel instant rather than frozen — the user sees words appearing immediately instead of staring at a spinner. Gemini's streaming API yields chunks you forward to the client over Server-Sent Events or a WebSocket; the harder part is what happens when the stream drops mid-response, which happens in production far more often than in a local demo.
A dropped stream should not leave the user with half a sentence and no error. Buffer enough state to detect a stall (no chunk for N seconds) and send an explicit "connection lost, retry" signal rather than letting the UI hang on a half-finished word.
Quick reference
- Stream via SSE for simplicity; use a WebSocket only if you need bidirectional messages mid-response.
- Detect stalls with a chunk-arrival timeout (e.g. 8s) — a silent stream is not the same as a finished one.
- On stall or error mid-stream, send an explicit terminal event the client can render as "try again," not a dangling ellipsis.
- Test the stall path deliberately: kill the network mid-response in development and confirm the UI shows a retry, not a hang.
Remember this
A stream that silently stops looks identical to a frozen app from the user's side — an explicit stall timeout and a terminal error event are what turn a network blip into a visible, recoverable "try again."
Key takeaway
Build the order-status bot end to end: single-turn call, then bounded conversation history, then the order_status function-calling tool against a stub API, then streaming with a stall detector. Verify success by asking "where is order A1092?" and confirming the reply cites the tool's returned status, not an invented one.
Then break it twice, deliberately: point the tool's fetch URL at an unreachable host and confirm the bot apologizes instead of hanging or fabricating a status; then kill the SSE connection mid-stream and confirm the client shows a retry prompt within your stall timeout instead of a frozen half-sentence. Pass criterion: both failures produce a visible, user-facing message within the timeout window you set, with no silent hang and no invented data.
Related Articles
Explore this topic