I did the other way round. An old nuc from 2013 turned into a gui-less debian selfhosted server (yunohost) for 2.5 years, until my old laptop (2008) died. Then I installed xfce on top, plugged a wireless keyboard/touchpad combo + hdmi to my TV, and use it as a desktop mainly for web browsing, open office and some gimp processing. But just once a week or so.

Yes I do have fail2ban. Do you mean I could have just (example) a yubikey and no ssh password? As safe as they can be, why remove the other factor?

Well fail2ban went from very active to very quiet. It is definitely worth not leaving 22 (when opening ssh is a must for different reasons)

I am not a expert in Linux, and I mostly rely on very strong passwords. I also discovered recently basic stuff like changing the default SSH port. Anyone knows of implementation of 2FA on Linux?