I run my containers in an LCX on Proxmox (yes I heard I should use a VM, but it works…)
For data storage (syncthing, jellyfin …) I make volumes in the LXC. But I was wondering if this is the best way?
I started thinking about restoring backups. The docker backups can get quite large with all the user data. I was wondering if a separate “NAS” VM and NFS shares makes more sense. Then restoring/cloning docker lxc would be faster, for troubleshooting. And the user data I could restore separately.
It’s not always possible but it’s generally good practice to configure your applications to use external storage rather than file systems - MySQL/PostgreSQL for indexable data, and S3-clones like MinIO for blob storage.
One major reason for this is that these systems generally have data replication and fall over redundancy built-in. So you can have two or more physical servers, have an instance of each type of server on each, and have these stay synchronized. If one server goes down, the disks crash, or you need to upgrade, you can easily rebuild a set of redundant servers without downtime, and all you need to do is save the configurations (and take notes!)
Like I said, not always possible, but in general the more an application needs to store “user data”, the more likely it is it has the ability to use one of the above as a backend storage system. That will reduce, significantly, the amount of application servers that need to be backed up, and may reduce your need to consider using NFS etc to separate the data.
I’m not directly familiar with either, but syncthing seems to be about backing up, so I’m not entirely surprised it’s file oriented, and jellyfin doesn’t look like it’s about user maintained content so much as being a server of content. So I’m not entirely surprised neither would support S3/Minio.
Yeah it took me a while to realize what S3 is intended to be too. But you’ll find “Blob storage” now a major part of most cloud providers, whether they support the S3 protocol (which is Amazon’s) or their own, and it’s to be used precisely the way we’re talking about: user data. Things clicked for me when I was reading the DoveCot manuals and found S3 was supported as a first class back-end storage system like maildir.
I’m old though, I’m used to this kind of thing being done (badly) by NFS et al…
Huh. I recently set up local dovecot for archiving old emails, but not S3.
I’m curious, when you work on a document, how does that work; Is it a file on your hard drive, have you mounted a bucket somehow, do you sync using restful api somehow?
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !selfhosted@lemmy.world
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don’t control.
Rules:
Be civil: we’re here to support and learn from one another. Insults won’t be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it’s not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don’t duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
It’s not always possible but it’s generally good practice to configure your applications to use external storage rather than file systems - MySQL/PostgreSQL for indexable data, and S3-clones like MinIO for blob storage.
One major reason for this is that these systems generally have data replication and fall over redundancy built-in. So you can have two or more physical servers, have an instance of each type of server on each, and have these stay synchronized. If one server goes down, the disks crash, or you need to upgrade, you can easily rebuild a set of redundant servers without downtime, and all you need to do is save the configurations (and take notes!)
Like I said, not always possible, but in general the more an application needs to store “user data”, the more likely it is it has the ability to use one of the above as a backend storage system. That will reduce, significantly, the amount of application servers that need to be backed up, and may reduce your need to consider using NFS etc to separate the data.
Interesting! I felt S3 was more a business cloud storage api.
I did a quick search, and it seems neither syncthing or jellyfin is compatible with S3. What do you do in these cases?
I’m not directly familiar with either, but syncthing seems to be about backing up, so I’m not entirely surprised it’s file oriented, and jellyfin doesn’t look like it’s about user maintained content so much as being a server of content. So I’m not entirely surprised neither would support S3/Minio.
Yeah it took me a while to realize what S3 is intended to be too. But you’ll find “Blob storage” now a major part of most cloud providers, whether they support the S3 protocol (which is Amazon’s) or their own, and it’s to be used precisely the way we’re talking about: user data. Things clicked for me when I was reading the DoveCot manuals and found S3 was supported as a first class back-end storage system like maildir.
I’m old though, I’m used to this kind of thing being done (badly) by NFS et al…
Huh. I recently set up local dovecot for archiving old emails, but not S3.
I’m curious, when you work on a document, how does that work; Is it a file on your hard drive, have you mounted a bucket somehow, do you sync using restful api somehow?
Rclone can do this for you.