We’ve done some fairly extensive testing internally recently and found that Garage is somewhat easier to deploy in comparison to our existing use of MinIO, but is not as performant at high speeds. IIRC we could push about 5 gigabits of (not small) GET requests out of it, but something blocked it from reaching the 20-25 gigabits (on a 25g NIC) that MinIO could reach (also 50k STAT requests/s, over 10 nodes)

I don’t begrudge it that. I get the impression that Garage isn’t necessarily focussed on this kind of use case.

---

In addition:

Next time we come to this we are going to look at RustFS [1], as well as Ceph/Rook [2].

We can see we're going to have to move away from MinIO in the foreseeable future. My hope is that the alternatives get a boost of interest given the direction MinIO is now taking.

[0]: https://news.ycombinator.com/item?id=46140342

[1]: https://rustfs.com/

[2]: https://rook.io/

> For the metadata storage, Garage does not do checksumming and integrity verification on its own, so it is better to use a robust filesystem such as BTRFS or ZFS. Users have reported that when using the LMDB database engine (the default), database files have a tendency of becoming corrupted after an unclean shutdown (e.g. a power outage), so you should take regular snapshots to be able to recover from such a situation.

It seems like you can also use SQLite, but a default database that isn't robust against power failure or crashes seems suprising to me.

https://www.repoflow.io/blog/benchmarking-self-hosted-s3-com... was useful.

RustFS also looks interesting but for entirely non-technical reasons we had to exclude it.

Anyone have any advice for swapping this in for Minio?

Garage looks really nice: I've evaluated it with test code and benchmarks and it looks like a winner. Also, very straightforward deployment (self contained executable) and good docs.

But no tags on objects is a pretty big gap, and I had to shelve it. If Garage folk see this: please think on this. You obviously have the talent to make a killer application, but tags are table stakes in the "cloud" API world.

It's a really cool system for hyper converged architecture where storage requests can pull data from the local machine and only hit the network when needed.

* no lifecycle management of any kind - if you're using it for backups you can't set "don't delete versions for 3 months", so if anyone takes hold of your key, you backups are gone. I relied on minio's lifecycle management for that but it's feature missing in garage (and to be fair, most other) S3

* no automatic mirroring (if you want to have second copy in something other than garage or just don't want to have a cluster but rather have more independent nodes)

* ACLs for access are VERY limited - can't make a key access only sub-path, can't make a "master key" (AFAIK, couldn't find an option) that can access all the buckets so the previous point is also harder - I can't easily use rclone to mirror entire instance somewhere else unless I write scrip iterating over buckets and adding them bucket by bucket to key ACK

* Web hosting features are extremely limited so you won't be able to say set CORS headers for the bucket

* No ability to set keys - you can only generate on inside garage or import garage-formatted one - which means you can't just migrate storage itself, you have to re-generate every key. It also makes automating it harder, in case of minio you can pre-generate key and pass then fed it to clients and to the minio key command, here you have to do the dance of "generate with tool" -> "scrape and put into DB" -> put onto clients.

Overall I like the software a lot but if you have setup that uses those features, beware.

[0] https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/1052

[1] https://github.com/Barre/ZeroFS

Previously I used LocalStack S3 but ultimately didn't like the lack of persistance thats not available on the OSS verison. MinIO OSS is apparently no longer maintained? Also looked at SeaweedFS and RustFS but from a quick reading into them this once was the easiest to set up.

https://garagehq.deuxfleurs.fr/blog/2022-ipfs/

Let's talk!

this is the reliability question no?

Does anyone know a good open source S3 alternarive that's easily extendable with custom storage backends?

For example, AWS offers IA and Glacier in addition to the defaults.

This is used for ransomware resistant backups.