One of them appears to be broken [1]. No big deal, this is what RAIDs are for, I go and try to find one and now they're going anywhere between 2-4x that price, for a used one! It's not going to bankrupt me (and having a home server is a privilege in the first place, that's not lost on me), but I really hope that the others survive, at least until this storage crunch is over. If it ever does end...sigh.
I guess I didn't realize that even relatively slow storage like spinner drives was going to be affected too.
[1] I think, I am really hoping it's just a bad connection or something but I haven't fully diagnosed it yet.
ETA: Looks like at least in my case it was actually just a bad SATA cable. The drive is reading properly and resilvering now. Phew.
I bought 8x 4TB NAS RED 5400RPM HDD and paid in total AUD$1.4k, I am not even joking. My pool is only 8TB mirrorer and I am keeping the other 16TB as spare because things will get worse.
I bought them brand new because used ones cannot be trusted but even that brings risks.
WD has been selling 7200RPM labelled as 5400RPM disks.
WD was selling SMR disks instead of CMR disks.
Seagate disks used into crypto farms are being sold as brand new, so folks are finding our their brand new disks have 65k hours.
Now, with what has happened with memory chip prices, it almost seems like they got lucky (the Micron facility is doing commercial shipments now).
Obama used to talk about having "spooky" good luck. I think Modi has some of that too.
We are trully doomed.