I dont know if this qualifies as a “toaster” but Ive used this docking bay in the past for a NAS and it served my purposes decently well. One thing to keep in mind is that random IO will be lacking with a usb interface. Also, this particular chipset does powercycle all the drives when one is removed so drive swaps end up requiring you to power the entire system off to perform. Also no integrated cooling may be a deal breaker as you illuded to.
If I was basing a nas build off of a PI, I would look to use the PCIe 1x2.0 interface on the pi 5 as a HBA.
Very interesting idea for a self hosted service! I will definitely take a stab at hosting it! I have a decent collection of DRM-free games from humblebundle and GOG that I always wanted in one place. Question, I know you dont currently have a native linux client. That being said, do you have a native linux client on the roadmap?
BTW you CAN do DNS in a unifi gateway. It just requires making dnsmasq entries through shell. Perfect solution? No. But it gets you there with no additional hardware.