Primary account is now @Dark_Arc@social.packetloss.gg.

  • 1 Post
  • 7 Comments
Joined 1Y ago
cake
Cake day: Jun 13, 2023

help-circle
rss
Anyone having issues with Lemmy self hosting?
My main account is dark_arc@social.packetloss.gg. However, as of roughly ~~24-hours ago~~ (it seems this has been going on since March 10th and [gotten worse since](https://grafana.lem.rocks/d/edf3gjrxxnocgd/federation-health-activities-behind?var-instance=All&var-remote_instance=social.packetloss.gg&var-remote_software=All&from=1710013804000&to=now&orgId=1)) it seems like the server has stopped properly retrieving content from lemmy.world. It's been running smoothly for well over 9 months, and (I think) working fine for content coming in from other instances. So I'm curious if anyone else experienced anything strange with lemmy.world federation recently? # Setup Description The server flow in my case is as follows: [Public Internet] <-> [Digital Ocean Droplet] <-> [ZeroTier] <-> [Physical Machine in my Basement ([HW Info](https://lemmy.world/comment/8738511))] The Digital Ocean droplet is a virtual host machine that forwards requests via nginx to the physical machine where a second nginx server (running the standard lemmy nginx config) then forwards the request to the lemmy server software itself. # Current Status ## Lemmy Internal Error I've found this is my lemmy logs: ``` 2024-03-24T00:42:10.062274Z WARN lemmy_utils: error in spawn: Unknown: Request limit was reached during fetch 0: lemmy_apub::objects::community::from_json at crates/apub/src/objects/community.rs:126 1: lemmy_apub::fetcher::user_or_community::from_json at crates/apub/src/fetcher/user_or_community.rs:87 2: lemmy_server::root_span_builder::HTTP request with http.method=POST http.scheme="http" http.host=social.packetloss.gg http.target=/inbox otel.kind="server" request_id=688ad030-f892-4925-9ce9-fc4f3070a967 at src/root_span_builder.rs:16 ``` I'm thinking this could be the cause ... though I'm not sure how to raise the limit (it seems to be hard coded). I opened [an issue](https://github.com/LemmyNet/lemmy/issues/4563 ) with the Lemmy devs but I've since closed it while gathering more information/making sure this is truly an issue with the Lemmy server software. ## Nginx 408 and 499s I'm seeing the digital ocean nginx server reporting 499 on various "/inbox" route requests and I'm seeing the nginx running on the physical machine that talks directly to lemmy reporting 408 on various "/inbox" route requests. There are some examples in this comment: https://lemmy.world/comment/8728858
fedilink

I don’t know anything about Netbird, but I’ll link you to my ZeroTier pitch the last time I noticed someone talking about Tailscale: https://lemmy.world/comment/1058287.


If you send these files over SSH, you’re good as that’s encrypted by ZeroTier and then encrypted again inside the SSH connection (and SSH does have perfect forward security).

See their cryptography section of their docs for more info.

You can read more here about what they’re working on:

It’s been to long since I’ve read that to give anything more than a condensed “they’re improving their crypto significantly with ZeroTier 2” (not to mention memory safety via Rust).


I think it’s pretty secure and it will be getting better soon. In reality, I think it’s much more secure than what most people will end up with otherwise.

ZeroTier is open source, long running without incident, and the traffic is encrypted between peers.

The threat model is basically two fold though, in theory someone who has control of the ZeroTier roots (if you’re not using your own controller, if you’re using your own, then s/their roots/your roots/) could add routes to your devices, and add/remove devices that are part of your confirmation.

The encryption also doesn’t currently have perfect forward security, so if there’s a compromise in one of your connections, in theory some past state of that connection could be decrypted. In practice, I’m not sure this matters as traffic at a higher level for most sensitive things uses its own encryption and perfect forward security (but hey maybe you have some software that doesn’t).

The other thing I will note about that last point is that they’re working on a rust rewrite that will have updated crypto, including perfect forward security.


FOSS just means the software is open source. As I said, you can self host ZeroTier and not involve their servers (if you’re not doing things commercially, you pay for the license but still run your own controllers, or you use an older version which has been automatically relicensed by the change date to Apache 2.0).

That said, the traffic is peer-to-peer, in the majority of use cases there servers are acting as a bit more than syncthing’s servers (acting to facilitate the connection between two devices that want to talk together). See the other comment for some more thoughts here.


I’ll pitch ZeroTier instead, it’s the same concept, but it’s more FOSS friendly, older, doesn’t have the non-networking “feature bloat” of Tailscale, and can handle some really niche cases like Ethernet bridging (should you ever care).

Just:

  1. Go to their website, create an account, and create a network
  2. Add ZeroTier to the devices you need to connect
  3. Enter your network ID on those devices
  4. Approve the devices in the web control panel

If you want to go full self hosting, you can do that too but you will need something with a static IP to control everything (https://docs.zerotier.com/self-hosting/network-controllers/?utm_source=ztp) this would replace the web panel parts.

You can also do a LAN routing based solution pretty easily using something like a Raspberry Pi (or really any Linux computer).


I prefer ZeroTier, I’m not sure why Tailscale has taken off so much in recent years (perhaps just the cleaner UI and better marketing).


I’ve tried to switch in the past, but tripped over the differences in Podman vs Docker networking. IIRC Docker is better for creating an isolated network.

I have noticed that Docker doesn’t do the best job at graceful shutdowns (say for automatic installation of updates). I suspect Podman with systemd integration could do much much beter.