FRESH

Hacker News

Ask HN: How do you handle duplicate side effects when jobs, workflows retry?

8 points by shineDaPoker

by codebitdaily

0 subcomment

Idempotency is the only sustainable answer here. Whether it's at the database level using unique constraints or implementing idempotency keys in your API headers, you have to design for the 'at-least-once' delivery reality. I usually implement a 'processed_requests' table that stores the unique ID of the job. Before the worker executes any side effect (like a payment or email), it checks if that ID exists. If it does, it skips the execution and returns the previous result. It adds a bit of latency, but it's much cheaper than dealing with double-billing or corrupted data

by babelfish

1 subcomments

by moomoo11

1 subcomments

You proxy those api calls yourself and have idempotency to cover you for those APIs that don’t have it. If you architect it right you won’t have more than a ms latency addition. You can avoid the race condition issues by using atomic records so if something else tries they’d see it’s in progress and exit.

by stephenr

1 subcomments

I think the answer is probably like most things: it depends.
- If the external service supports idempotent operations, use that option.
- If the external service doesn't, but has a "retrieval" feature (i.e. lookup if the thing already exists, e.g fetch refunds on a given payment), use that first.
- If the system has neither, assess how critical it is to avoid duplicates.