by JakaJancar
2 subcomments
- Assuming you're fine with keeping the queue in postgres, I've used Procrastinate and it's great:
https://procrastinate.readthedocs.io/en/stable/index.html
Core is not Django-specific, but it has an optional integration. Sync and async, retries/cancellation/etc., very extensible, and IMO super clean architecture and well tested.
IIRC think the codebase is like one-tenth that of Celery.
- Celery is such garbage to run/maintain at any sort of scale. Very excited for this. Rq/temporal also seem to solve this well.
Anyone here done the migration off of celery to another thing? Any wisdom?
by adamcharnock
2 subcomments
- Something about queuing systems that often gets me is that they can start to seem like the wrong abstraction as soon as one has tasks that enqueue additional tasks. Particularly when features start growing, and double particularly when modelling business processes.
This is because the code enqueuing the task needs to be aware of what happens next, which breaks separation of concerns. Why should the user sign-up code have to know that a report generation job now needs queuing?
Really what starts to make more sense to me is to fire off events. Code can say, "this thing just happened", and let other code decide if it wants to listen. When then makes it an event stream rather than a queue, with consumer groups at al.
I made the (now unmaintained) project https://lightbus.org around this, and it did work really well for our use case. Hopefully someone has now created something better.
So I'd say this: before grabbing for a task queue, take a moment to think about what you're actually modelling. But be careful of the event streaming rabbit-hole!
- Really cool to see a batteries‑included option in Django for background jobs.
For folks who’ve used Celery/Procrastinate/Chancy: how does retry/ACK behavior feel in real projects? Any rough edges?
What about observability — dashboards, tracing, metrics — good enough out of the box, or did you bolt on extra stuff?
Also, any gotchas with type hints or decorator-style tasks when refactoring? I’ve seen those bite before.
And lastly, does swapping backends for tests actually feel seamless, or is that more of a “works in the demo” thing?
by vforgione
1 subcomments
- I’ve been using the django-tasks library in production for about a year. The database backend and simple interface have been great. It definitely isn’t intended to replace all of celery, but for a simple task queue that doesn’t require additional infrastructure it works quite well.
by matsemann
3 subcomments
- How is the typing support? We just had downtime because a change to a celery task didn't trigger mypy to complain for all call sites until runtime. Too many python decorators aren't written with pretty weak typing support.
by Grollicus
2 subcomments
- That's intersting to see.
Does the api support progress reporting? ("30 % done")
Of course one could build this manually when you're building the worker implementation but I'd love to have it reflected in the api somewhere. Celery also seems to be missing api for that.
Anyone sees a reason why that's missing? I don't think it complicates the api much and seems such an obvious thing for longer-running background tasks.
by delbronski
0 subcomment
- I’ve tried Celery/Procrastinate/Chancy and others in the past. Nothing has really felt “right.” So I always end up using GCPs cloud task jobs and scheduling. It’s just so simple for small to mid size projects.
Really want to give this a try though.
by the__alchemist
2 subcomments
- This is great! The prev recommendation was usually a lib called celery that I wasn't able to get working. I don't remember the details, but it had high friction points or compatibility barriers I wasn't able to overcome. This integration fits Django's batteries included approach.
I've been handling this, so far, with separate standalone scripts that hook into Django's models and ORM. You have to use certain incantations in an explicit order at the top of the module to make this happen.
- Django this is about 10 years too late. It's frustrating because we use all manner of hacks to work around this being part of the builtin story.
- This is an exciting development. I imagine I'll continue using Celery in most cases but being able to transparently swap back-ends for testing, CI, etc. is very compelling.
I haven't looked into this in any detail but I wonder if the API or defaults will shave off some of the rough edges in Celery, like early ACKs and retries.
by leonkenneth
0 subcomment
- Very excited to see these kind of improvements. Coming from Rails to Django I've been a bit disappointed in the amount of things provided out of the box for a SaaS app (the main exception being of course the admin). Even in the ecosystem, I find packages to be a bit more messy and less production-ready than in Rails. Granted I'm fairly new to it, and the design philosophies are different
- Really you want automatic transpilation to go. A good Christmas project for someone.
by fatih-erikli-cg
0 subcomment
- [dead]
Hacker News