How are you adding real-time chat to AI applications?

I’m curious how developers here are adding chat functionality to their AI applications.

If you’re building an AI assistant or an app powered by LLMs, handling the model responses is one thing, but managing conversations is another. You need to think about message history, real-time updates, and what happens when a user disconnects and comes back later.

I’m wondering how others are approaching this.

Are you building the chat functionality from scratch, or using an existing API or SDK? And if you already shipped something, what was the most challenging part, handling real time messages, storing conversation history or scaling it as more users join?

I’d be interested to hear what you’re using and what you’ve learned from building it.

The three problems you list are easier to handle separately.

  1. Transport: plain SSE is enough for streaming tokens from the model to one client, and browsers reconnect on their own. Switch to WebSocket only if the client needs to send things mid-generation (stop, steer, voice).
  2. History: store every message in your own database as it completes, keyed by conversation ID, and load it when the user comes back. Don’t rely on the stream for this.
  3. Reconnect during generation: this is the hard part, since an SSE stream is tied to one request. The usual fix is to give each generation a stream ID and buffer the chunks (Redis Streams works) so the client can resume from the last offset instead of re-running inference. Vercel’s AI SDK has this pattern built in for page reloads.

What bites people in production is usually not the code but the infrastructure around it: proxy buffering and serverless or CDN timeouts that cut long streams, which work fine locally. Managed realtime services only really pay off once you need multi-device sync or presence. Until then, SSE + a database + a small resume buffer is enough.