FRESH

Hacker News

Open sourcing Dicer: Databricks's auto-sharder

80 points by vivek-jain

by charleshn

1 subcomments

> Application pods learn the current assignment through a library called the Slicelet (S for server side). The Slicelet maintains a local cache of the latest assignment by fetching it from the Dicer service and watching for updates. When it receives an updated assignment, the Slicelet notifies the application via a listener API.
For a critical control plane component like this, I tend to prefer a constant work pattern [0], to avoid metastable failures [1], e.g. periodically pull the data instead of relying on notifications.
[0] https://aws.amazon.com/builders-library/reliability-and-cons...
[1] https://brooker.co.za/blog/2021/05/24/metastable.html

by khaki54

1 subcomments

Seems weird to call it sharding since it's not sharding indexed datasets or anything like that. Is this just a tool to mitigate Databricks’ internal service-scaling challenges?

by ayf

3 subcomments

Does anyone else have something similar?
What are some use cases that you found are useful?

0 subcomment

by vivek-jain

0 subcomment