FRESH

Hacker News

How to make SSE token streams resumable, cancellable, and multi-device

74 points by zknill

by e1g

1 subcomments

In JS land, this problem (streaming, resuming, recovering, multi-client, etc) has been fully solved by https://durablestreams.com - and it can be self-hosted, or managed via Cloudflare DO.

by jhancock

1 subcomments

I built this Clojure lib for robust high scale LLM calls wherein the consumer is usually a http request waiting on an SSE stream. https://github.com/jhancock/aimee
The article states: "Most applications are built on an architecture like the one above, where there are a number of stateless horizontally scaleable server replicas that can handle client requests."
Using the library I built, I have yet to worry about this as Clojure core.async, http libs and Java VM are so rock solid, I don't have a fragile set of stateless servers. Sure, at some point there are rare edge cases but it's nice to get very far along without worrying about them.

by the_gipsy

1 subcomments

> Stop reading here if you just wanted the how-to. Because I’m going to talk about what I think is better, and that is probably too ‘commercial’ for some folks.
> I work for Ably, and I’m building a dedicated transport for AI applications that...

by _pdp_

1 subcomments

This is way too complex!
We have developed a simple API that can produce tokens and events in various formats like jsonl, sse, even csv. Cancelation can happen when the socket is closed when streaming - fully automated - or when you push an event from another endpoint so that we stop the stream midway.
Background task are also subscriable and cancelable.
see https://cbk.ai

by ekojs

0 subcomment

> HTTP is just not a good transport for streaming LLM tokens and for building async agentic applications
I don't know if I agree if this is a problem with SSE or HTTP. Something like a Redis Streams-backed SSE would solve most of the 'challenges' presented in the post.

by TodorGrudev

0 subcomment

Cancellation was the painful one for me. I have SSE streaming in a React Native app and there's no proper AbortController — ended up with a ref + interval hack to detect when the user navigates away mid-stream. Still don't have a good answer for "connection dropped, show partial response and let them retry from where it left off." Would've loved something like this.