https://lucumr.pocoo.org/2025/11/3/absurd-workflows/

https://github.com/earendil-works/absurd

https://earendil-works.github.io/absurd/

I've not used it, but it's worth comparing to other options

  Restate.dev:
    for payment integrations on northflank since its faster than cf workflows, independent of cf and its downtime and self-hostable vendor-lock-in free,
  Cloudflare workflows:
    for non critical stuff like csv/pdf report generations since it's very cheap.
  DBOS.dev:
    for workflows that need atomic messaging tied to a postgres db transaction for 100% reliabilty/durabilty(for example populating a materialized row or sending out critical email/push to a merchant).

It also has VirtualObject which is a nice vendor-lock-in-free OSS alternative to CF's single threaded DurableObject.

Where DBOS absolutely shines is

1) Atomic messaging in the same db tx as your business logic via dbos.enqueue_workflow! This is often the most brittle part of any solution and doing it atomically and durably with same tx that ran your business logic drastically reduces lots of complexity.

2) Since DBOS stores workflow state in db it should be easy to build dashboard for observability from metabase/looker(I wish restate exposed its rocksdb instance so it could be hooked up to metabase).

I have used Temporal in the past, works really good, my only problem with it was some limits on request payload or event sizes, created some inconveniences to us when building solutions. It also enforces good engineering practices, but sometimes you don't want to write special logic if your CSV file is larger than 2Mb, upload it to S3, pass link, then download it in the workflow.

What is your experience with DBOS? How does it compare to Temporal in terms of operational complexity, feature parity and anything else

https://github.com/agentspan-ai/agentspan which is essentially an agentic SDK layer for Conductor can convert any of your langgraph, openAI, vercel, or ADK agent and makes it durable and adds orchestration with no code changes.

A dedicated external orchestrator decouples application servers' API and implementation from that of the orchestrator. This comes with some advantages. It becomes easier and more natural for the application servers to organize their APIs and implementation around a stable set of domain areas (instead of operating along two different layers of abstraction -- the workflow orchestration plane, and the underlying application domain logic). It also becomes easier to understand and update workflow logic when it's managed separately as a first-class citizen as opposed it being fragmented and diffused across multiple application servers. In my experience, these advantages tend to matter more in highly distributed system involving multiple semi-independent teams owning multiple application servers and data-stores. After a certain point in terms of the size and complexity of the distributed system, the aggregate cost of handling orchestration and checkpointing workflows often starts to provide stronger justification for having dedicated centralized orchestrator.

So while I agree that letting go of the central orchestrator (and letting the application servers and data-store do that work on their own) can sometimes be the pragmatic/preferred option, I'd argue that the fit is context dependent and there doesn't seem to be a one-size-fits-all solution available.

I recently developed a distributed queue and it works really great - benchmarks great too, with no race conditions or conflicts. I used SKIP LOCKED so that workers can compete safely.

You can also have multiple workers across nodes avoid conflict by using session wide mutexes i.e. pg advisory lock.

That said, my gamer-brain wants to call this "Save-scumming at scale." Which is to say, a lot of people already know that this approach works, but maybe they haven't made the connection to abstract CS stuff.

Another strategy that can be used to build robustness is to build your workflow out of idempotent operations. That can be useful for situations where the workflow state is too large to back up. Instead, you just run the job from the top and it's a bunch of no-ops until you start making progress again.

External Spanner provides changes streams. Internal spanner is different, mostly because of the extreme scaling requirements in some cases (and a healthy dose of "because it already works" mixed with "arbitrary change streams are scary").

Internal Spanner allows any transaction to write queue entries, where queues are (more or less) tables with some special time awareness. You can schedule delivery. Entries get pushed from queues to a handler which can also do writes to the DB within the dequeue transaction. And all of the same scaling is there.

Rails recently introduced a 'continuable' concept, allowing you to checkpoint and resume steps within a job, but it still feels like the convention is too keep jobs with a single responsibility, so it feels odd to use them for true workflows.

Has anyone else experienced this or found a solution to it?

It supports pipelines, batched pipelines, and basic runners, as well as idempotent keys (including batching them). It also lets you "partition" a queue into multiple sub-queues so that you can easily segregate your jobs within your application without a lot of setup on the outside. For example, you create a root queue talking to PostgreSQL and pass it around to subsystems that then each create their own sub-queue off that to enqueue entries into and their own workers that dequeue them.

It's only used internally right now but I've been thinking about creating a separate package (with documentation) with it for others to use as well. Any feedback or pull requests would be appreciated !

[0] https://github.com/KeetaNetwork/anchor/blob/main/src/lib/que...

[1] https://github.com/KeetaNetwork/anchor/blob/main/src/lib/que...

The main benefit is centralizing all the data in one place so we don't need to worry about copying data in between multiple systems. Once something becomes the bottleneck, you can eventually migrate to a purpose specific tool to scale out.To be honest, LISTEN/NOTIFY in my opinion is the most fragile part of PG but it's fine as start until you scale out.

Here's a an example computing a Fibonacci sequence (very inefficiently, with lots of spawned sub-tasks and message passing) [2]

[1] https://github.com/estuary/flow/tree/master/crates/automatio... [2] https://github.com/estuary/flow/blob/master/crates/automatio...

Not sure where the NIH ends and where you're actually better off with a supported orchestration approach. I suppose if you expect your program to be around a while (or need advanced features), maybe think about using something a bit more battle tested?

Strong correctness guarantee is something that should not be undermine. Even more important than availability.

The examples on the website is simple but heavily undermines the importance of correctness. Anyone who implement similar pseudo-code directly will eventually suffer from data correctness issue in crashes.

  @DBOS.workflow()
  def checkout_workflow(items: Items):
      order = create_order()
      reserve_inventory(order, items)
      payment_status = process_payment(order, items)

      if payment_status == 'paid':
          fulfill_order(order)
      else:
          undo_reserve_inventory(order, items)
          cancel_order(order)

I also recently started experimenting with https://github.com/earendil-works/absurd which is also Postgres and even simpler than DBOS. Their comparison is a great read:

https://earendil-works.github.io/absurd/comparison/

But for operational reasons I've started using sqlite for durable workflows instead. Porting the database concepts from either DBOS or absurd PG to SQLite is remarkably easy these days. A small polling loop instead of notify/listen feels fine for smaller workloads.

It questions the initial wisdom of creating lots of little independent distributed apps, without regards to interaction between them. Let’s build ever more necessary plumbing and schemes just to enable their interaction.

I am arguing that durable workflows should be a last resort for boundaries you must cross, not a default pattern for every business process.

For that particular usage, the volume we process and business criticality make it a good choice for inventing here - but for other durable processes we just use off the shelf tools since the cost of maintenance would quickly outstrip the value.

Postgres is a great tool to use and far more powerful than most people give it credit for - but there's always the balance of in-house maintenance vs. paying rent for someone else's solution.

Postgres is not cheap to run in the cloud at scale. We went for the cheapest infra, which is basically the disk storage.

Possibility one: There is one index on the table, and it is the created_at TS. This query has to scan 10,000 jobs/sec * 60 seconds * 60 minutes * 24 hours * 31 days * 1024 bytes / job = 25,543 GB.

A KV store would scan exactly that much.

Possibility two: The primary key is refined to (state, timestamp). Assume a 1% failure rate. Now, we "only" scan and return 255 GB. A key value store would scan exactly that much. (This is probably the right physical design).

Possibility three: The primary key is (timestamp), and there's a secondary index on state. I guess we do an index join, where one side of the join is 25,543 GB, and the other side is one unsorted bucket with 255GB * number of months the system has been in operation in it.

A KV store wouldn't let you express that.

Now, what other ad hoc queries are we supposed to efficiently support over a one month lookback? Also, what does PG do if you tell it to scan 25TB at the same time as it's inserting 10MB/sec at 10K TPS? How is vacuuming configured?

PG is just a detail of implementation, you need a good library to build reliable flows.

Typescript: https://www.pgflow.dev

Elixir: https://github.com/agoodway/pgflow/blob/main/docs/COMPARISON...

In any case there can be more to durable workflows than just saving the current step, and not all intermediate steps are serializable thus I don't get where's the postgres magic that more mature solutions don't have.

Given the above, it would seem that durable workflow software is pushed forward by those who have a surplus of VC money to spend. As for the vendors, there is no shortage of people trying to sell you things that you don't need.

Once you need retries, backoff, timeouts, cancellation, versioning, visibility, task routing, rate limits, leases, heartbeats, stuck-worker detection, replay/debugging semantics, workflow migration, fanout/fanin, long timers, audit trails, and operator tooling, the “just use a database” story becomes “build a poor copy of a workflow engine plus a bunch of workers.” pretty quick.

That may still be a good tradeoff for many applications, especially if Postgres is already the core operational dependency. But the comparison shouldn’t be “database vs overcomplicated orchestrator.” It’s more like “what complexity do you want to own, and what do you want to buy / offload to a professional system?”