by killingtime74
1 subcomments
- I got Claude to analyze the code and it's not really comparable to SKIP LOCKED queues. It's more like Kafka. There's no job queue semantics with acks, workers taking from same job pool.
It's Kafka like one event stream and multiple independent worker cursors.
It's more SNS than SQS or Kafka than Rabbitmq/Nats
by mind-blight
0 subcomment
- The vacuum pressure is real. Using a system with the skip locked technique + polling caused massive DB perf issues as the queue depth grew. The query to see the current jobs in the queue ended up being the main performance bottleneck, which cause slower throughput, which caused a larger queue depth, which etc.
Scaling the workers sometimes exacerbates the problem because you run into connection limits or polling hammering the DB.
I love the idea of pg as a queue, but I'm a more skeptical of it after dealing with it in production
- I don't understand the latency graph. It says it has 0.25ms consumer latency.
Then in the latency tradeof section it says end to end latency is between 1-2 seconds.
Is this under heavy load or always? How does this compare to pgmq end to end latency?
- Postgres durability without having to run Kafka or RabbitMQ clusters seems pretty enticing. May reach for it when I next need an outbox pattern or small fan out.
by andrewstuart
0 subcomment
- Postgres is not the only database that does queues.
Any database that supports SKIP LOCKED is fine including MySQL, MSSQL, Oracle etc.
Even SQLite makes a fine queue not via skip locked but because writes are atomic.
- I think it's great that projects like this exist where people are building middleware in different ways than others. Still, as someone who routinely uses shared memory queues, the idea of considering a queue built inside a database to be "zero bloat" leaves me scratching my head a bit. I can see why someone would want that, but once person's feature is bloat to someone else.
- So if I understand this correctly, there are three main approaches:
1. SKIP LOCKED family
2. Partition-based + DROP old partitions (no VACUUM required)
3. TRUNCATE family (PgQue’s approach)
And the benefit of PgQue is the failure mode, when a worker gets stuck:
- Table grows indefinitely, instead of
- VACUUM-starved death spiral
And a table growing is easier to reason about operationally?
- Cool
Hacker News