Skip to main content
All work
Status: Built At Oracom Intelligence

Reliable real-time AI conversations: generation and connection lifecycles

Hands-on work on two layers of real-time AI conversations: tracking, cancelling, and cleaning up overlapping generations in the backend, and handling WebSocket connection races and errors in an avatar application.

  • WebSockets
  • Cooperative cancellation
  • Async lifecycle management
  1. Step 1: A request starts a generation
  2. Step 2: The generation is tracked as active
  3. Step 3: A newer request or a cancel arrives
  4. Step 4: The old generation stops and is cleaned up
The backend lifecycle of one generation, conceptually. The avatar app's connection handling is a separate layer. No internals.

Context

A real-time AI conversation runs several lifecycles at once: the request, the generation the model produces for it, and the connection the answer travels over. People interrupt, send a new message before the last answer is finished, or drop and reconnect. If nothing coordinates those lifecycles, the system depends on timing instead of on explicit rules.

My role

I implemented both changes described here myself, alongside leading the team. They sit in two different layers, the backend that runs generations and an avatar application that holds the connection, and they are separate pieces of work, not one fix. The wider voice and avatar stack is team work; this case covers only my part of it.

What I built and decided

  • Backend: tracking of active generations, so overlapping chat requests are known to the system instead of running side by side unnoticed.
  • Backend: cooperative cancellation, so a superseded generation stops at a safe point rather than being cut off mid-operation.
  • Backend: cleanup when a generation finishes or is cancelled, so state from an earlier request is released.
  • Avatar app: handling of WebSocket connection race conditions and errors, so opening, closing, and failing connections are dealt with explicitly rather than left to timing.
  • The judgment behind both: treat asynchronous work as something with a lifecycle that has to be owned, started, cancelled, and cleaned up, instead of letting the interface hide overlap or treating a successful generation as the only case that matters.

Outcome

In the components I changed, overlapping requests, cancellation, and connection changes are now handled explicitly and more predictably. Both changes are merged into their codebases. That is the boundary of the claim: I have no before-and-after latency or incident numbers, and this does not make every channel of the platform fully reliable.

What I'd do next

Add representative interaction and regression tests that replay interruptions, overlapping turns, and reconnects, then measure what the person in the conversation notices, such as the time to a fresh answer after an interruption.