FRESH

Hacker News

The end of the road for Kafka-delta-ingest

16 points by alex_hirner

by nevi-me

0 subcomment

It's great to see my name mentioned here, cheered me up as I'm not having a good weekend. It feels like a lifetime ago since I got to work on this and parquet-rs.
Yeah, I agree with the conclusion on Kafka here. Kafka is a very resource-heavy application that's worth replacing with leaner options.
Oxbow seems very interesting, I presume it's designed to forward data that trickles into some S3 bucket as parquet data?

by rmnclmnt

0 subcomment

If only for the initiative driving the work behind delta-rs, I’d say it was well worth it!

by redwood

0 subcomment

by raverbashing

0 subcomment

> The big argument against kafka-delta-ingest was Apache Kafka. If an organization has Kafka for other reasons, then kafka-delta-ingest can be a useful “sidecar” process to persist data flowing through Kafka. If however the organization is running Kafka just for ingestion, there are cheaper options available. As the organization evolved, the other consumers of Kafka drifted away, driving the value proposition of kafka-delta-ingest lower and lower.
Sounds like how most of Kafka stories end.