Cloud storage for mobile devices of close family members on and off site
Complications:
Electricity cost is a scam where I live (~0.40$ per Kwh)
I have an ageing NAS whose capacity is getting to its limits storage and horsepower wise. And since I have to do work on my setup anyways, I was wondering whether you might give me some inputs on what I could/should use to achieve my goals.
My NAS is getting to its teenage years and I am getting worried about its eol. Buying some old server HW is out of the question because of power usage and availability. What are my best options? Build something myself with current hardware? Buy a new NAS? What is a good way to migrate data to a new system? From a power consumption standpoint are SSD’s better than HDD’s?
I have an off site which i visit regularly where I could either place backup drives or put a system in a rack. What would be a good option for an offsite backup solution?
I have gotten my aunt (77) a tablet during covid so she could video call us. In recent months a smartphone has entered the ring because daily life is getting impacted when you don’t have one. Now she is all into taking pictures and videos and the storage on her phone is not enough. What are my options? I’ve experimented with Nextcloud but I am uncertain whether it is the right solution, especially from a usability perspective. (I want to avoid third party services for storage)
I will very much appreciate your input since I’m not working in the field and am getting to the edge of my own knowledge at this point.
I’m still a fan of synology because it does a lot of what you want out of the box without you needing to constantly manage and setup all these services from scratch. I’ve upgraded through several synology units over the years, currently using a 6TBx8 unit for much of what you mention. Since drives are so big these days, you could get a newer 4bay with more horsepower and just drop a couple 20TB drives in it as a mirrored pair then in the future add more drives as needed. Dropping to 2 drives cuts your power consumption a bit, and staying with a 4 bay instead of something bigger will also keep the power down.
You can absolutely build your own, but synology comes with all the “home cloud” apps preconfigured and your time and effort is worth something too. I build enterprise cloud environments for a living and I don’t want to have to do that at home on my free time- synology is so plug’n’play.
What are my best options? Build something myself with current hardware?
Build something based on a MicroATX board with a CPU > i5-6500, this will probably cost around 80€ second hand with RAM. Server hardware is overkill, wastes power and older CPUs are always worse in all possible ways. Even if the board only has 4 SATA ports, a PCI to 5 SATA port card and it will be enough. Use BTRFS as your filesystem and its RAID if needed.
You may be thinking something like “I want a faster CPU in order to have fast SMB”, don’t. Your gigabit network will saturate before your CPU or even mechanical drives and when this happen you’ll get something like 10-20% CPU usage. Just don’t waste your money.
From a power consumption standpoint are SSD’s better than HDD’s? It depends on the hardware, either way is isn’t feasible to have around 20TB of SSD storage. Even considering you can now buy a 4TB SSD for 195€ that would be a lot of money.
Syncthing is a very good piece of software, to sincronize devices with a NAS - except if you use iOS. Currently I’m running Syncthing on my NAS and all my devices sync to it (no cross-device sync to avoid issues). Then I’ve an SMB share to allow access to the files on iOS devices and FileBrowser for a cloud-like web browser access experience. Works flawlessly uses very little RAM and its solid, private, secure and manageable open-source - not something like Nextcloud that calls home, breaks everything on upgrades, wastes ram and runs slowly to only deliver an inferior experience in all possible ways.
Running stuff in your NAS: if you’re really into a low power solution you might want to stay away from Docker. Simply install a clean Debian system and manually get what you need instead of tons of containers and dependencies. This will allow you to run more stuff on less hardware. Linux isn’t that hard, you can do it.
Does FileBrowser support creating public links for sharing? I use Nextcloud as a way to deliver large amounts of photos and videos to my clients.
My issue with Syncthing is that doing partial sync is sort of a pain in the ass. My Nextcloud currently has 290GB of data that I’d rather not completely sync to all of my devices and AFAICT with Syncthing, you still need to fiddle around with config files to do that, and even then its clunky and doesn’t work sometimes.
Yeah I get that Nextcloud is a bit slow but it’s definitely more capable as a drop-in cloud storage replacement than other software I’ve seen.
My issue with Syncthing is that doing partial sync is sort of a pain in the ass.
That’s like the ONLY feature everything is missing from Syncthing. Frankly I don’t get it, I just don’t understand why they don’t implement it. They’ve their ignore patterns that can essentially be leveraged to implement partial sync but they won’t just do it. https://forum.syncthing.net/t/selective-sync-feature-most-important/16394
Nice that share feature looks pretty slick. I might check this out.
Yeah I frankly don’t get why syncthing doesn’t implement it either. It’s like the only feature that really holds me back from using it, otherwise it’s pretty damn slick and has much faster sync than Nextcloud.
I agree with you that Server Hardware is overkill. That’s why I am asking for options here. For your suggested architecture: It is quite ancient and eol per last year and I have one of these already in use as a “more than my NAS thingy”. What I am not comfortable with is that that generation is already end of life and I don’t want to invest time and money into hardware that I would have to replace a year or two from now. I’m looking for a solution (self built or not) that will drag me through the next decade.
Does BTRFS include Raid support? I don’t have much experience with it. The most I did once was recover a snapshot.
CPU is the least of my concerns. I am currently looking at a low end current gen Intel CPU for my purpose. And yes, samba is slow and I will never saturate a somewhat recent CPU with it, but I have from time to time other things running on that machine.
For storage: You are overestimating a bit with the prices. For 200€ I can get 4TB SSDs. So 20TB + one for raid 5 ~1200€. That would be quite doable but on the expensive side I agree. The question was whether SSDs were considerable cause of power usage, not from a price standpoint.
As fro Syncthing… I’ll have a look at that.
Docker is just nice and simple. I remember times when deploying software on a single server was hell on earth. Conflicting libraries etc. And yes Linux isn’t hard (been using it for like 2 decades).
Does BTRFS include Raid support? I don’t have much experience with it. The most I did once was recover a snapshot.
It does have RAID support but its RAID 5 and 6 are BROKEN! The devs themselves do not recommend using these. If you need RAID 5 and 6 and you absolutely want to use BTRFS, you’ll have to go with mdraid and then put BTRFS on top, but then you lose a lot of the BTRFS self-healing capabilities. Personally for RAID 5 and 6, I still recommend ZFS’s RAIDz. It’s quite easy to setup. I have a DIY NAS with an OS drive running BTRFS and a storage pool consisting of 4x 4TB SSDs running in RAIDz1.
What I am not comfortable with is that that generation is already end of life
The thing is that those are very cheap as people want faster things for Windows desktops and as we both know Samba won’t ever saturate that CPU on a 1GbE to 2.5GbE link. If you can get a last-gen low power solution like an i3 in the same price ranges go for it and ignore my previous advice. But you know an i5-6500 or i5-7400 + motherboard + RAM for 70-80€ is a good deal and enough for the use case.
For 200€ I can get 4TB SSDs.
Sorry, my mistake. I meant 4TB.
Does BTRFS include Raid support?
Yes many possible configurations and snapshots. BTRFS also tends to be way more reliable than Ext4 and others when the hardware fails, you experience sudden power losses etc. More: https://linuxhint.com/set-up-btrfs-raid/
As fro Syncthing… I’ll have a look at that.
Syncthing + FileBrowser is a “killer” setup for a personal cloud. I’m even amazed you’ve never heard about / used Syncthing as it is very popular in self-hosting. You can also use it to sync your main NAS with the remote backup. Very reliable and easy to setup.
Docker is just nice and simple. I remember times when deploying software on a single server was hell on earth.
Never had that experience… if the software is properly done and you aren’t using a weird distro things should work out well. Eventually you can use LXC/LXD or even systemd containers to isolate problematic applications without having to deal with all the Docker overhead and mess.
Except RAID 5 and 6! Those are still broken on BTRFS and not recommended for use by the devs. It’s unfortunate because I just setup a DIY NAS and I had to go with ZFS because of this.
Eh, RAID 5 and 6 are still viable for home deployments. Not a lot of people want to be running massive drive arrays or expensive disks at home just to get decent storage. I ran a 4x 4TB RAID 5 for close to a decade and it’s survived 4 drive rebuilds. The Intel chip on the QNAP machine I was using to maintain that array died before the array itself did. Now I have an NVMe SSD-based array, so drive rebuilds are even less of a concern.
The other reason why I brought it up is that the article you linked doesn’t even mention BTRFS RAID 5 and 6 issues until all the way down at the bottom of the article in a small paragraph, when really it should be in bright red letters at the beginning.
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !selfhosted@lemmy.world
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don’t control.
Rules:
Be civil: we’re here to support and learn from one another. Insults won’t be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it’s not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don’t duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
No trolling.
Resources:
selfh.st Newsletter and index of selfhosted software and apps
I’m still a fan of synology because it does a lot of what you want out of the box without you needing to constantly manage and setup all these services from scratch. I’ve upgraded through several synology units over the years, currently using a 6TBx8 unit for much of what you mention. Since drives are so big these days, you could get a newer 4bay with more horsepower and just drop a couple 20TB drives in it as a mirrored pair then in the future add more drives as needed. Dropping to 2 drives cuts your power consumption a bit, and staying with a 4 bay instead of something bigger will also keep the power down.
You can absolutely build your own, but synology comes with all the “home cloud” apps preconfigured and your time and effort is worth something too. I build enterprise cloud environments for a living and I don’t want to have to do that at home on my free time- synology is so plug’n’play.
Build something based on a MicroATX board with a CPU > i5-6500, this will probably cost around 80€ second hand with RAM. Server hardware is overkill, wastes power and older CPUs are always worse in all possible ways. Even if the board only has 4 SATA ports, a PCI to 5 SATA port card and it will be enough. Use BTRFS as your filesystem and its RAID if needed.
You may be thinking something like “I want a faster CPU in order to have fast SMB”, don’t. Your gigabit network will saturate before your CPU or even mechanical drives and when this happen you’ll get something like 10-20% CPU usage. Just don’t waste your money.
From a power consumption standpoint are SSD’s better than HDD’s? It depends on the hardware, either way is isn’t feasible to have around 20TB of SSD storage. Even considering you can now buy a 4TB SSD for 195€ that would be a lot of money.
Nextcloud is garbage, yes very bad usability, more reasons and issues listed here: https://lemmy.world/comment/1571886 and https://lemmy.world/comment/346174
Syncthing is a very good piece of software, to sincronize devices with a NAS - except if you use iOS. Currently I’m running Syncthing on my NAS and all my devices sync to it (no cross-device sync to avoid issues). Then I’ve an SMB share to allow access to the files on iOS devices and FileBrowser for a cloud-like web browser access experience. Works flawlessly uses very little RAM and its solid, private, secure and manageable open-source - not something like Nextcloud that calls home, breaks everything on upgrades, wastes ram and runs slowly to only deliver an inferior experience in all possible ways.
Running stuff in your NAS: if you’re really into a low power solution you might want to stay away from Docker. Simply install a clean Debian system and manually get what you need instead of tons of containers and dependencies. This will allow you to run more stuff on less hardware. Linux isn’t that hard, you can do it.
Does FileBrowser support creating public links for sharing? I use Nextcloud as a way to deliver large amounts of photos and videos to my clients.
My issue with Syncthing is that doing partial sync is sort of a pain in the ass. My Nextcloud currently has 290GB of data that I’d rather not completely sync to all of my devices and AFAICT with Syncthing, you still need to fiddle around with config files to do that, and even then its clunky and doesn’t work sometimes.
Yeah I get that Nextcloud is a bit slow but it’s definitely more capable as a drop-in cloud storage replacement than other software I’ve seen.
https://noted.lol/content/images/2023/01/filebrowser-self-hosted-public-shares-noted.gif
That’s like the ONLY feature everything is missing from Syncthing. Frankly I don’t get it, I just don’t understand why they don’t implement it. They’ve their ignore patterns that can essentially be leveraged to implement partial sync but they won’t just do it. https://forum.syncthing.net/t/selective-sync-feature-most-important/16394
Nice that share feature looks pretty slick. I might check this out.
Yeah I frankly don’t get why syncthing doesn’t implement it either. It’s like the only feature that really holds me back from using it, otherwise it’s pretty damn slick and has much faster sync than Nextcloud.
Thank you for your answer.
I agree with you that Server Hardware is overkill. That’s why I am asking for options here. For your suggested architecture: It is quite ancient and eol per last year and I have one of these already in use as a “more than my NAS thingy”. What I am not comfortable with is that that generation is already end of life and I don’t want to invest time and money into hardware that I would have to replace a year or two from now. I’m looking for a solution (self built or not) that will drag me through the next decade. Does BTRFS include Raid support? I don’t have much experience with it. The most I did once was recover a snapshot.
CPU is the least of my concerns. I am currently looking at a low end current gen Intel CPU for my purpose. And yes, samba is slow and I will never saturate a somewhat recent CPU with it, but I have from time to time other things running on that machine.
For storage: You are overestimating a bit with the prices. For 200€ I can get 4TB SSDs. So 20TB + one for raid 5 ~1200€. That would be quite doable but on the expensive side I agree. The question was whether SSDs were considerable cause of power usage, not from a price standpoint.
As fro Syncthing… I’ll have a look at that.
Docker is just nice and simple. I remember times when deploying software on a single server was hell on earth. Conflicting libraries etc. And yes Linux isn’t hard (been using it for like 2 decades).
It does have RAID support but its RAID 5 and 6 are BROKEN! The devs themselves do not recommend using these. If you need RAID 5 and 6 and you absolutely want to use BTRFS, you’ll have to go with mdraid and then put BTRFS on top, but then you lose a lot of the BTRFS self-healing capabilities. Personally for RAID 5 and 6, I still recommend ZFS’s RAIDz. It’s quite easy to setup. I have a DIY NAS with an OS drive running BTRFS and a storage pool consisting of 4x 4TB SSDs running in RAIDz1.
The thing is that those are very cheap as people want faster things for Windows desktops and as we both know Samba won’t ever saturate that CPU on a 1GbE to 2.5GbE link. If you can get a last-gen low power solution like an i3 in the same price ranges go for it and ignore my previous advice. But you know an i5-6500 or i5-7400 + motherboard + RAM for 70-80€ is a good deal and enough for the use case.
Sorry, my mistake. I meant 4TB.
Yes many possible configurations and snapshots. BTRFS also tends to be way more reliable than Ext4 and others when the hardware fails, you experience sudden power losses etc. More: https://linuxhint.com/set-up-btrfs-raid/
Syncthing + FileBrowser is a “killer” setup for a personal cloud. I’m even amazed you’ve never heard about / used Syncthing as it is very popular in self-hosting. You can also use it to sync your main NAS with the remote backup. Very reliable and easy to setup.
Never had that experience… if the software is properly done and you aren’t using a weird distro things should work out well. Eventually you can use LXC/LXD or even systemd containers to isolate problematic applications without having to deal with all the Docker overhead and mess.
Except RAID 5 and 6! Those are still broken on BTRFS and not recommended for use by the devs. It’s unfortunate because I just setup a DIY NAS and I had to go with ZFS because of this.
As if anyone should be using RAID 5 or RAID 6: https://www.zdnet.com/article/why-raid-6-stops-working-in-2019/ and https://www.linkedin.com/pulse/avoid-raid5-raid6-live-happy-life-tech-tuesdays-6-jv-roig/?articleId=6568798728638038016
Those setups are highly discouraged.
Eh, RAID 5 and 6 are still viable for home deployments. Not a lot of people want to be running massive drive arrays or expensive disks at home just to get decent storage. I ran a 4x 4TB RAID 5 for close to a decade and it’s survived 4 drive rebuilds. The Intel chip on the QNAP machine I was using to maintain that array died before the array itself did. Now I have an NVMe SSD-based array, so drive rebuilds are even less of a concern.
The other reason why I brought it up is that the article you linked doesn’t even mention BTRFS RAID 5 and 6 issues until all the way down at the bottom of the article in a small paragraph, when really it should be in bright red letters at the beginning.