pool: blackhole
state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
repaired.
scan: scrub repaired 0B in 02:01:59 with 0 errors on Tue Jul 11 04:02:09 2023
config:
NAME STATE READ WRITE CKSUM
blackhole DEGRADED 0 0 0
raidz1-0 DEGRADED 0 0 0
ata-WDC_WD120EDAZ-11F3RA0_5PG8DYKC ONLINE 0 0 0
ata-WDC_WD120EFBX-68B0EN0_5QKJ6M8B ONLINE 0 0 0
ata-WDC_WD120EFBX-68B0EN0_5QKJTT8B FAULTED 51 0 0 too many errors
errors: No known data errors
I only got the drive 6 months ago, well within WD’s 3 year warranty so I opened a support case, but do errors like this basically always mean the drive is its way out or is it possible to have false positives?
Look at your SMART data and run some tests. Reseat the SATA cables too. I’ve had that cause problems. Even had a SATA cable go bad a few times over the years. That will cause you to get these kinds of errors too. ZFS is pretty paranoid about data loss. So if it even gets a small hint that something is wrong it does t his.
At this point it really comes down to how valuable the data is to you. Most of the time when I see this error I’m not seeing anything on SMART that would lead me to believe that there’s a problem. So I’ll clear the error and watch it. If I start to get the same problem with the same drive I’ll usually replace it when I can. That being said I have pretty good backups so it would inconvenience me a lot but it’s most likely not going to be the end of the world if my drive dies on me. YMMV
Thanks, I’ll try some of those things out and see if a second scrub says the same.
That being said I have pretty good backups
Out of curiosity, what do you do for backups? The initial cost of 3x12TB drives was enough to make me not want to spend a bunch more money on backup stuff at the time, but now that I’m seeing errors I’m willing to spend a bit of money again and should look into my options.
I use Backblaze B2 and restic. Just a simple systemd unit & timer setup to kick off the backup. I also have a restic repo setup on a external drive for my most important things eg family photos. I try to follow the 3,2,1 rule as much as possible. Fedora magazine has two articles on the site about setting up restic like that. They’re pretty helpful if you need pointers.
If there’s something I want to share or don’t need/want encrypted I’ll just use rclone to sync it to B2.
Backups are very cheap with B2, restores can be costly. But cost less than something like AWS Glacier. So it’s my last resort for restoring things. Mostly I rely on snapshots in the case that I delete something by accident. (Check out SANOID)
I also have all my other systems using B2. I have a bucket setup for all of my family’s laptops that they backup too as well. Keeps everyone’s data safe.
If you have some data that required putting on an eyepatch & tricorn (Yarr me matey) to acquire and you don’t care about loosing it then don’t back that up.
Edit: I also keep my important data on set of mirrored pairs. It’s not space efficient but it does the job of keeping things performant and safe. Eventually I’ll expand that past 3 pairs but for right now it’s 3 pairs (6 drives) of 10TB disks.
Anything else that isn’t important is just on a small Z1 array. I put all my older drives on that array because they would just be on a shelf doing nothing otherwise so I don’t care about wasting storage on that array. Not a recommended practice at all. So do as I say not as I do kinda thing.
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !selfhosted@lemmy.world
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don’t control.
Rules:
Be civil: we’re here to support and learn from one another. Insults won’t be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it’s not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don’t duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
Look at your SMART data and run some tests. Reseat the SATA cables too. I’ve had that cause problems. Even had a SATA cable go bad a few times over the years. That will cause you to get these kinds of errors too. ZFS is pretty paranoid about data loss. So if it even gets a small hint that something is wrong it does t his.
At this point it really comes down to how valuable the data is to you. Most of the time when I see this error I’m not seeing anything on SMART that would lead me to believe that there’s a problem. So I’ll clear the error and watch it. If I start to get the same problem with the same drive I’ll usually replace it when I can. That being said I have pretty good backups so it would inconvenience me a lot but it’s most likely not going to be the end of the world if my drive dies on me. YMMV
Thanks, I’ll try some of those things out and see if a second scrub says the same.
Out of curiosity, what do you do for backups? The initial cost of 3x12TB drives was enough to make me not want to spend a bunch more money on backup stuff at the time, but now that I’m seeing errors I’m willing to spend a bit of money again and should look into my options.
I use Backblaze B2 and restic. Just a simple systemd unit & timer setup to kick off the backup. I also have a restic repo setup on a external drive for my most important things eg family photos. I try to follow the 3,2,1 rule as much as possible. Fedora magazine has two articles on the site about setting up restic like that. They’re pretty helpful if you need pointers.
If there’s something I want to share or don’t need/want encrypted I’ll just use rclone to sync it to B2.
Backups are very cheap with B2, restores can be costly. But cost less than something like AWS Glacier. So it’s my last resort for restoring things. Mostly I rely on snapshots in the case that I delete something by accident. (Check out SANOID)
I also have all my other systems using B2. I have a bucket setup for all of my family’s laptops that they backup too as well. Keeps everyone’s data safe.
If you have some data that required putting on an eyepatch & tricorn (Yarr me matey) to acquire and you don’t care about loosing it then don’t back that up.
Edit: I also keep my important data on set of mirrored pairs. It’s not space efficient but it does the job of keeping things performant and safe. Eventually I’ll expand that past 3 pairs but for right now it’s 3 pairs (6 drives) of 10TB disks.
Anything else that isn’t important is just on a small Z1 array. I put all my older drives on that array because they would just be on a shelf doing nothing otherwise so I don’t care about wasting storage on that array. Not a recommended practice at all. So do as I say not as I do kinda thing.
Wow that’s pretty substantial, thanks for the tips! Wow yeah Backblaze does seem pretty affordable.
I’ve had this happen when I had ram issues. You can try doing a memory test if you want to take that out of the equation.