There is no double nat. Passthrough mode has worked as expected for me. The one issue I have is that the RG will maintain firewall states, so it limits you to the RG hardware for those states. I have a pretty large home network though, tons of devices, IoT, etc, and it has been stable.

Latency seems decent. I have an AT&T fiber 2gb symmetrical connection and a ping to google from my Netgate pfSense machine is around 10-15ms.

I have the same Residental Gateway. Using pfSense+ on my end. The BGW320-500 is fiber capable. I assume you’re using fiber? If so you cannot hook it into ONT because the RG is the ONT. In my case I get raw fiber into a PON module that hooks into the RG. Best you can do in this case is set the RG to “passthrough mode” via web UI (192.168.1.254).

If you have a different setup that is not fiber maybe you’ll have more luck with a bypass, but I think you will need the RG regardless for auth: https://docs.netgate.com/pfsense/en/latest/recipes/authbridge.html