My use-case: streaming video to a Linux virtual mount and want compression of said video files on the fly.
Rclone has an experimental remote for compression but this stuff is important to me so that’s no good. I know rsync can do it but will it work for video files, and how I get rsync to warch the virtual mount-point and automatically compress and move over each individual file to rclone for upload to the Cloud? This is mostly to save on upload bandwidth and storage costs.
Thanks!
Edit: I’m stupid for not mentioning this, but the problem I’m facing is that I don’t have much local storage, which is why I wanted a transparent compression layer and directly push everything to the Cloud. This might not be worth it though since video files are already compressed. I will take a look at handbrake though, thanks!
This seemed to be read-only tho, so not sure if it covers the use case you described. If you can program a little (AI help?) find a simple fuse filesystem in a language you know, fiddle with it and call ffmpeg or similar on receiving files.
What is the format of these videos? Im afraid you wont get much compression out of conventional file compressors, as video files are usually already compressed to the point where you would have to reencode them to get a smaller file.
You’re right, I don’t know why I didn’t consider that. This is going to be a mix of security cameras and live streaming video that I’ll store on the cloud, and the problem is that I have horrible upload speeds along with no local storage for caching
Stash the file in a staging directory that tdarr watches, have tdarr convert the file to something small like h265. Output the converted file to a folder rsync watches.
Unless you’ve got raw uncompressed video, any kind of transparent compression like you describe is only going to cost you in energy bills for no benefit. Most video is already compressed with specialised video compression as part of the file format, you can’t keep compressing stuff and getting smaller files.
The alternative is a lossy compression, which you could automate with some scripts or a transcoding tool like tdarr. This would reduce the quality of the video in order to reduce the file size
The problem is that I don’t have the local storage to maintain a watch folder for continually streaming video. I want to write semi-directly to the Cloud, which is why I’m looking for a transparent reencoding layer. Can handbrake do this?
I’m not sure about transparently, that’s more in the tdarr wheelhouse I’d say. You’d dump the files into a monitored folder and it will replace it with a version transcoded to your specification.
Transcoding video takes a fair bit of time and energy too FWIW, so you’re going to need enough local storage to handle both the full size and smaller one.
I have to question the idea though, cloud storage is always more expensive than local for anything remotely non-temporary, and transcoding a load of video all the time is going to increase your energy bills. If you have any kind of internet bandwidth restrictions that’s gonna factor in too.
I’d say it would be better to save up for a cheap external hard drive to store your video on. For a year’s subscription to a cloud storage service that would provide enough space for a media library, you could probably get twice the amount of storage forever.
I’m going to be using an SBC for this, which doesn’t have the capacity for an extra storage drive. Also, I’m planning to move in a couple of months, and I wouldn’t want to deal with storage in the middle of all of this. The cloud isn’t as insanely expensive as I initially thought; B2 is $6/TB, and I hope that with reencoded streams at an OK resolution I wouldn’t go beyond 1.5TB a month (I’ll be deleting stuff with bucket policies, of course).
I’ll take a look at tdarr alongside ffmpeg, thanks!
Okay fair play, if you’re doing this super short term it could make sense. Though I question what SBC you’re using that’s capable of transcoding video but not the ability to plug in an external drive.
$12/m for your 2TB of usage would make sense for maybe 5 months before it would be cheaper to buy an external disk—and of course that storage is gone once that time is up, Vs a hard disk which will probably last you a decade or so
I was considering the usual BananaPi/OrangePi/Raxda/Pine64 SBCs, are those not enough horsepower? I’d like to stay under $80 for my SBC purchase, and it will be doing double duty with managing some services like DNS and music scrobbling alongside uploading to the cloud
Hardware transcoding on SBCs is generally not fantastic, you’re gonna want to look for one that has VAAPI/VDPAU support or you’re gonna be looking at 100% CPU for half a day to transcode a film, which will make your other services effectively unavailable at the time.
I used to run my Plex server on a Pi4 with 4GB of ram and it basically crashed any time transcoding kicked in, I swapped to an intel NUC so I could get QuickSync for transcoding.
I’ll point out though, every SBC you’ve listed has usb, which is all you need for an external disk. If you’re worried about size, I’ve got a 5tb external drive that’s about 5cm², which is basically the footprint of any SBC you could use in this scenario
As everyone has said, lossless compression might not have great ratios, but if it’s still worth it I recommend dwarfs as it creates read only mountable filesystems with minimal setup
https://github.com/mhx/dwarfs
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !selfhosted@lemmy.world
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don’t control.
Rules:
Be civil: we’re here to support and learn from one another. Insults won’t be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it’s not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don’t duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
Not much help, but a quick search revealed this: https://github.com/nschlia/ffmpegfs
This seemed to be read-only tho, so not sure if it covers the use case you described. If you can program a little (AI help?) find a simple fuse filesystem in a language you know, fiddle with it and call ffmpeg or similar on receiving files.
Already compressed files like video don’t meaningfully compress more. Compression isn’t magic.
What is the format of these videos? Im afraid you wont get much compression out of conventional file compressors, as video files are usually already compressed to the point where you would have to reencode them to get a smaller file.
You’re right, I don’t know why I didn’t consider that. This is going to be a mix of security cameras and live streaming video that I’ll store on the cloud, and the problem is that I have horrible upload speeds along with no local storage for caching
This sounds stupid but what about tdarr?
Stash the file in a staging directory that tdarr watches, have tdarr convert the file to something small like h265. Output the converted file to a folder rsync watches.
Thank you, but there’s another problem: I don’t have local storage to write the files to and then upload. I need to write semi-directly to the Cloud.
Unless you’ve got raw uncompressed video, any kind of transparent compression like you describe is only going to cost you in energy bills for no benefit. Most video is already compressed with specialised video compression as part of the file format, you can’t keep compressing stuff and getting smaller files.
The alternative is a lossy compression, which you could automate with some scripts or a transcoding tool like tdarr. This would reduce the quality of the video in order to reduce the file size
The problem is that I don’t have the local storage to maintain a watch folder for continually streaming video. I want to write semi-directly to the Cloud, which is why I’m looking for a transparent reencoding layer. Can handbrake do this?
I’m not sure about transparently, that’s more in the tdarr wheelhouse I’d say. You’d dump the files into a monitored folder and it will replace it with a version transcoded to your specification.
Transcoding video takes a fair bit of time and energy too FWIW, so you’re going to need enough local storage to handle both the full size and smaller one.
I have to question the idea though, cloud storage is always more expensive than local for anything remotely non-temporary, and transcoding a load of video all the time is going to increase your energy bills. If you have any kind of internet bandwidth restrictions that’s gonna factor in too.
I’d say it would be better to save up for a cheap external hard drive to store your video on. For a year’s subscription to a cloud storage service that would provide enough space for a media library, you could probably get twice the amount of storage forever.
I’m going to be using an SBC for this, which doesn’t have the capacity for an extra storage drive. Also, I’m planning to move in a couple of months, and I wouldn’t want to deal with storage in the middle of all of this. The cloud isn’t as insanely expensive as I initially thought; B2 is $6/TB, and I hope that with reencoded streams at an OK resolution I wouldn’t go beyond 1.5TB a month (I’ll be deleting stuff with bucket policies, of course).
I’ll take a look at
tdarralongsideffmpeg, thanks!Okay fair play, if you’re doing this super short term it could make sense. Though I question what SBC you’re using that’s capable of transcoding video but not the ability to plug in an external drive.
$12/m for your 2TB of usage would make sense for maybe 5 months before it would be cheaper to buy an external disk—and of course that storage is gone once that time is up, Vs a hard disk which will probably last you a decade or so
I was considering the usual BananaPi/OrangePi/Raxda/Pine64 SBCs, are those not enough horsepower? I’d like to stay under $80 for my SBC purchase, and it will be doing double duty with managing some services like DNS and music scrobbling alongside uploading to the cloud
Hardware transcoding on SBCs is generally not fantastic, you’re gonna want to look for one that has VAAPI/VDPAU support or you’re gonna be looking at 100% CPU for half a day to transcode a film, which will make your other services effectively unavailable at the time.
I used to run my Plex server on a Pi4 with 4GB of ram and it basically crashed any time transcoding kicked in, I swapped to an intel NUC so I could get QuickSync for transcoding.
I’ll point out though, every SBC you’ve listed has usb, which is all you need for an external disk. If you’re worried about size, I’ve got a 5tb external drive that’s about 5cm², which is basically the footprint of any SBC you could use in this scenario
Thank you for the tip, I’ll keep a look out for VAAPI/VDPAU support. Thanks!
As everyone has said, lossless compression might not have great ratios, but if it’s still worth it I recommend dwarfs as it creates read only mountable filesystems with minimal setup https://github.com/mhx/dwarfs