- Wow. Wasn’t expecting to see this here… I am the author…
by divyekapoor
0 subcomment
- Why do you need a cryptographic hash function for detecting duplicates? xxHash, rapidHash or anything that’s fast will do the job. There’s no need to be secure against inverses.
- I'm not sure exactly how it compares, but a similar tool you can use that also does phashes of images/videos to find potential duplicate media files is czkawka [0].
[0]: https://github.com/qarmin/czkawka
- I have a thing like this - one thing I have found useful is also going up the directory tree to find entire duplicated directories. (So dir hash is hash of the concatenation of the subdirs and the files).
I also used go, it excels at just this sort of thing. Although I saturate the disk to memory bandwidth (for the SHA) way sooner than CPU on my laptop, so it still takes a while to run.
Nice.
- Here's a tip, if you are making regular compressed backups of the configuration of your network devices, if you use gzip, make sure to set the mtime to 0 when writing the file as this can throw off deduplication. This also applies to the name field.
https://docs.python.org/3/library/gzip.html#gzip.GzipFile.mt...
- Reflink conversion would be another nice feature to have. It allows userspace apps to create filesystem-native copy-on-write clones of files, so to the next app they work exactly like another copy of the same data. But the data is on disk only once.
- > …finds duplicates across large disks instantly
:D
by akoboldfrying
1 subcomments
- Interested to know what algorithm you use for --fuzzy. Presumably not any of the worst-case-quadratic diff algorithms... MinHash on n-grams?
by jauntywundrkind
1 subcomments
- imo fclones is the one to beat,
https://github.com/pkolaczk/fclones#benchmarks
nice that there are actually some ok interfaces here. with fclones, i tend to generate a file of candidates then do a little review to make super certain everything is square / as expected. it's very built for intermediary files a core pattern, which is very unix, very convenient. but it did take me a little while to settle on this, and it felt like i wasn't being offered a ton of options for management out of box. looks like some real attempts to be more user friendly here.
by daflkfdslkfds
0 subcomment
- [dead]
Hacker News