Logcheck. It took ages to make sure innocent logs are ignored, but now I get an email as soon as anything non-routine happens on my servers. I get emails with logs from every update, every time I log in, etc. This has given me the most confidence that nothing unexpected is happening on my servers. Of course, one needs to make sure that the firewall is configured well, and that you use ssh keys etc., but logcheck is how I know I’m doing enough.

As others have said, with an incremental filesystem level mechanism, the backup process won’t be too taxing for the CPU. I have ZFS set up which makes this easy and I make hourly snapshots using sanoid which also get sent to another mirrored pair of connected drives using syncoid. Then, once a day, I upload encrypted daily snapshots to a bucket in the cloud using restic. Sounds complicated, but actually sanoid/syncoid and restic do all the heavy lifting. All I did is automate their schedules using systemd timers and some scripts to backup the right directories.

Very interesting project! However, I can’t help shake the feeling that whilst you pitch it as a platform for sharing DRM-free games, it will get used for sharing games against the licenses and wishes of publishers. I don’t really care about the publishers, but do you not think there is a great risk that once your app gets enough attention, it will draw their ire and force you to shut down? Perhaps not directly, but e.g., removing you from the windows store etc.

My configuration and deployment is managed entirely via an Ansible playbook repository. In case of absolute disaster, I just have to redeploy the playbook. I do run all my stuff on top of mirrored drives so a single failure isn’t disastrous if I replace the drive quickly enough.

For when that’s not enough, the data itself is backed up hourly (via ZFS snapshots) to a spare pair of drives and nightly to S3 buckets in the cloud (via restic). Everything automated with systemd timers and some scripts. The configuration for these backups is part of the playbooks of course. I test the backups every 6 months by trying to reproduce all the services in a test VM. This has identified issues with my restoration procedure (mostly due to potential UID mismatches).

And yes, I have once been forced to reinstall from scratch and I managed to do that rather quickly through a combination of playbooks and well tested backups.

Thanks for this useful reply! I think I’ll just need to closely examine my setup and figure out if I really need the ability to up/down interfaces like I described or whether the more persistent approach of networkd is actually more suitable for me. Sometimes I just want to reproduce behaviour that I’ve used before, but may not actually need.

Thanks for your reply! One thing I’m struggling with networkd is hysteresis. That is, toggling the interface down and then back up does not do what I expect it to. That is, setting the interface down does not clear up the configuration, and setting the interface up does not reconfigure the interface. I have to run reconfigure for that. I was hoping that the declarative approach of networkd would make it easy to predict interface state and configuration.

This does make sense because configuration is not the same as operational state. However, what would the equivalent of ifdown (set interface down and remove configuration) and ifup (set interface up and reconfigure) be using networkd and networkctl? This kind of feature would be useful for me to test config changes, debug networking issues, disconnect part of the network while I’m making some changes, etc.

Thanks a lot for these tips! Especially about using the upstream deb.

I already posted that I recommend fastmail elsewhere in this thread, but you raised so many good points that it reminded me of some extra points :)

Fastmail offers granular, per-app passwords – I have a single password which has read-only access to IMAP in order to back up all the data on a timer. This feature is missing from many (many) other email providers - using the 80/20 rule, if they even offer it it’s a single password with full access (Mailfence, for example)

Since this community is about selfhosting I think it’s worth pointing out that this is AMAZING for selfhosting. I have all me selfhosted services sending e-mail via fastmail’s SMTP. With per-app passwords I don’t need to store my normal e-mail password and the apps can be limited to SMTP only (so no read access). And in case of compromise you can revoke permissions on a per-app granularity.

Fastmail offers full CardDAV (contacts) and CalDAV (calendar) access, which makes plugging it into any other app that supports this very easy - their DNS wizard helps you set up the service records. I use “DavX5” on my Android to sync all Contacts and Calendar outside of using the Fastmail app (which is a self contained app on Android, it’s not too bad)

Fastmail has become my contacts app now - it’s really great to have all your e-mail and contacts in the same place. The contacts don’t even need to have an e-mail address - I have a lot of contacts stored for whom I only have a phone number. I sync to android using the same DavX5 app and then immediately have these contacts in whatsapp and signal.

I recommend fastmail.com though they do have done shortcomings that you need to consider such as the fact that they’re based in Australia (five eyes country) and have servers in the USA. Their advantage is a slick interface, fantastic app based on JMAP, and just generally being super convenient. They allow catch all addresses, masked emails, custom domain etc. I find them super convenient.

Apparently yes! Based on another comment in this thread: https://certificate.transparency.dev/monitors/.

If it was just storage/RAM scraping then that could be solved with SSL pass-through though. That way the reverse proxy would not decrypt the traffic and would forward the encrypted traffic further to the home server. I was actually setting that up a few hours ago. However, since the VPS provider owns the IP address of the VPS, they can simply obtain their own certificate for the domain. After all, Let’s Encrypt verifies your ownership of the domain by your ability to control the DNS entries. Therefore, even if the certificates weren’t on the VPS, the fact that I am redirecting traffic via their IP address makes me vulnerable to a malicious provider.

The “hobby exercise” was just to indicate that this is not for work and that I’m interested in an answer beyond “you need to trust your provider” which I do :) I agree, these are important questions! And they’re also interesting!

I originally used this too, but in the end had to write my own python script that basically does the same thing and is also triggered by systemd. The problem I had was that for some reason podman sometimes thinks there is a new image, but when it pulls it just gets the old image. This would then trigger restarts of the containers because auto-update doesn’t check if it actually downloaded anything new. I didn’t want those restarts so had to write my own script.

Edit: but I lock the version manually though e.g. nextcloud 27 and check once a month if I need to bump it. I do this manually in case the upgrade needs an intervention.