I really want to run ceph because it fits a number of criteria I have: gradually adding storage, mismatched disks, fault tolerance, erasure encoding, encryption, support out-of-the-box from other software (like Incus).
But then I look at the hardware suggestions, and they seem like an up-front investment and ongoing cost to keep at least three machines evenly matched on RAM and physical storage. I also want more of a single-box NAS.
Would it be idiotic to put a ceph setup all on one machine? I could run three mons on it with separate physical device backing each so I don’t lose everything from a disk failure with those. I’m not too concerned about speed or network partitioning, this would be lukewarm storage for me.
Since you are talking mismatched disks, I have gone to unraid after running a ceph cluster. I found it easy to keep adding and upgrading disks in unraid where it made more sense than maintaining or adding nodes. While I like the concept of being able to add nodes for very large storage arrays. My current unraid server is 180tb.
It is super simple to add/upgrade the storage one disk at a time.
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !selfhosted@lemmy.world
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don’t control.
Rules:
Be civil: we’re here to support and learn from one another. Insults won’t be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it’s not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don’t duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
Ceph is a huge amount of overhead, both engineering and compute resources, for this usecase.
Since you are talking mismatched disks, I have gone to unraid after running a ceph cluster. I found it easy to keep adding and upgrading disks in unraid where it made more sense than maintaining or adding nodes. While I like the concept of being able to add nodes for very large storage arrays. My current unraid server is 180tb.
It is super simple to add/upgrade the storage one disk at a time.
Oh, neat, I’ll have to look into that more. It’s able to have some redundancy and does some sort of rebalancing on disk failures?
It has parity disks, which always need to be the largest disks in your array. You can run with either a single one double parity disk.
It seems to work well, as that’s how I’ve had to replace a dozen disks in the last year upgrading from 8tb disks to 18 or 22tb disks.
Found an interesting read regarding the matter here:
https://old.reddit.com/r/ceph/comments/mppwas/single_node_ceph_vs_zfsbtrfs/
Most seem to recommend going for ZFS instead if using a single machine but there is a person discussing his first hand experience with single node Ceph.
This was really neat, kinda boils down to “you don’t want to deal with the complexity and it’s horrifically slow.”
Neat! Thank you
I did it for a while and last everything. Go for it if your have adequate backups.