Hi, I was looking at private CAs since I don’t want to pay for a domain to use in my homelab.

What is everyone using for their private CA? I’ve been looking at plain OpenSSL with some automation scripts but would like more ideas. Also, if you have multiple reverse-proxy instances, how do you distribute domain-specific signed certificates to them? I’m not planning to use a wildcard, and would like to rotate certificates often.

Thanks!


Edit: thank you for everyone who commented! I would like to say that I recognise the technical difficulty in getting such a setup working compared to a simple certbot setup to Let’s Encrypt, but it’s a personal choice that I have made.

@vegetaaaaaaa@lemmy.world
link
fedilink
English
5
edit-2
9M

I’m not using a private CA for my internal services, just plain self-signed certs. But if I had to, I would probably go as simple as possible in the first time: generate the CA cert using ansible, use ansible to automate signing of all my certs by the CA cert. The openssl_* modules make this easy enough. This is not very different from my current self-signed setup, the benefit is that I’d only have to trust a single CA certificate/bypass a single certificate warning, instead of getting a warning for every single certificate/domain.

If I wanted to rotate certificates frequently, I’d look into setting up an ACME server like [1], and point mod_md or certbot to it, instead of the default letsencrypt endpoint.

This still does not solve the problem of how to get your clients to trust your private CA. There are dozens of different mechanisms to get a certificate into the trust store. On Linux machines this is easy enough (add the CA cert to /usr/local/share/ca-certificates/*.crt, run update-ca-certificates), but other operating systems use different methods (ever tried adding a custom CA cert on Android? it’s painful. Do other OS even allow it?). Then some apps (Web browsers for example) use their own CA cert store, which is different from the OS… What about clients you don’t have admin access to? etc.

So for simplicity’s sake, if I really wanted valid certs for my internal services, I’d use subdomains of an actual, purchased (more like renting…) domain name (e.g. service-name.internal.example.org), and get the certs from Let’s Encrypt (using DNS challenge, or HTTP challenge on a public-facing server and sync the certificates to the actual servers that needs them). It’s not ideal, but still better than the certificate racket system we had before Let’s Encrypt.

@MigratingtoLemmy@lemmy.world
creator
link
fedilink
English
19M

Finally got around to replying to the comments I got haha!

Thanks for the explanation. I’m curious why you’re not running your own CA since that seems to be a more seamless process than having to deal with ugly SSL errors for every website, every time you rotate the certificate.

I’m wondering about different the process is between running an ACME server and another daemon/process like certbot to pull certificates from it, vs writing an ansible playbook/simple shell script to automate the rotation of server certificates.

About my clients: I am likely never going to purchase Apple products since I recognise how much they lock down their device. Unfortunately, there are not that many android devices in the US with custom ROM support. With that said, I do plan to root all of my Android devices when KernelSU matures (in about a year, I think) - I’m currently reading up on how to insert a root and client certificate into Android’s certificate store, but I think it’s definitely possible. Other than that, I might have a throwaway Windows VM sometimes, which is doable, alongside a Void linux box with a Debian chroot. All in all, fairly doable, just a bit of work to automate.

Thanks!

@vegetaaaaaaa@lemmy.world
link
fedilink
English
1
edit-2
9M

I’m curious why you’re not running your own CA since that seems to be a more seamless process than having to deal with ugly SSL errors for every website

It’s not, it’s another service to deploy, maintain, monitor, backup and troubleshoot. The ugly SSL warning only appears once, I check the certificate fingerprint and bypass the warning, from there it’s smooth sailing. The certificate is pinned, so if it ever changes I would get a new warning and would know something shady is going on.

every time you rotate the certificate.

I don’t really rotate these certs, they have a validity of several years.

I’m wondering about different the process is between running an ACME server and another daemon/process like certbot to pull certificates from it, vs writing an ansible playbook/simple shell script to automate the rotation of server certificates.

  • Generating self-signed certs is ~40 lines of clean ansible [1], 2 lines of apache config, and one click to get through the self-signed cert warning, once.
  • Obtaining Let’s Encrypt certs is 2 lines of apache config with mod_md and the HTTP-01 challenge. But it requires a domain name in the public DNS, and port forwarding.
  • Obtaining certs from a custom ACME CA is 3 lines of apache config (the extra line is to change the ACME endpoint) and a 100k LOC ACME server daemon running somewhere with its own bugs, documentation, deployment and upgrade management tooling, config quirks… and you still have to manage certs for this service. It may be worth it if you have a lot of clients who don’t want to see the self-signed cert warning and/or worry about their private keys being compromised and thus needing to rotate the certs frequently (you still need to protect the CA key…)

likely never going to purchase Apple products since I recognise how much they lock down their device

hear hear

there are not that many android devices in the US with custom ROM support. With that said, I do plan to root all of my Android devices when KernelSU mature

I bought a cheap refurbished Samsung, installed LineageOS on it (Europe, but I don’t see why it wouldn’t work in the US?), without root - I don’t really need root, it’s a security liability, and I think the last time I tried Magisk it didn’t work. The only downside is that I have to manually tap Update for F-Droid updates to run (fully unattended requires root).

I’m currently reading up on how to insert a root and client certificate into Android’s certificate store, but I think it’s definitely possible.

I did it on that LineageOS phone, using adb push, can’t remember how exactly (did it require root? I don’t know). It works but you get a permanent warning in your notifications telling you that The network might be monitored or something. But some apps would still ignore it.

@MigratingtoLemmy@lemmy.world
creator
link
fedilink
English
29M

Europe

Samsung devices in the US have their bootloaders locked, regardless of whether you bought it from a carrier or not. I’ll be looking at other devices, and even then, custom ROM support has all but stopped for everything but the pixel. Living in Europe is great for this, for you have the FP5 available.

But some apps would still ignore it.

Apps with their own certificate store like Firefox? Yeah, I’m thinking about how I can deal with that. Is there a FOSS Android MDM solution that I can use?

I run a crude automation on top of OpenSSL CA. It checks for certain labels attached to kubernetes services. Based on that it creates kubernetes secrets containing the generated certificates.

@MigratingtoLemmy@lemmy.world
creator
link
fedilink
English
19M

I would be very interested in your setup. Could you give me a high-level overview of how your setup works?

I’d also like a link to your automation/configuration if you would. Thanks!

I just uploaded to Github: https://github.com/akashrawal/nsd4k

I only made it for myself, so expect very rough edges in there.

@MigratingtoLemmy@lemmy.world
creator
link
fedilink
English
19M

Thank you!

@TCB13@lemmy.world
link
fedilink
English
59M

Written in go, very small and portable: https://github.com/FiloSottile/mkcert. There’s also step-ca, bigger and uses ACME to deploy certificates, never used it tho.

Just be awake of the risks involved with running your own CA.

@MigratingtoLemmy@lemmy.world
creator
link
fedilink
English
29M

Thanks, could you tell me why one would run this over plain OpenSSL with automation? Also, what risks would I run running a private CA? I’d love to know!

@TCB13@lemmy.world
link
fedilink
English
2
edit-2
9M

could you tell me why one would run this over plain OpenSSL with automation?

Those projects essentially are the automation…

what risks would I run running a private CA? I’d love to know!

https://security.stackexchange.com/questions/205174/what-are-the-risks-of-installing-a-ca-on-the-same-machine-as-openvpn-server

More or less you’re adding a root certificate to your systems that will effectively accept any certificate issues with your CA’s key. If your PK gets stolen somehow and you don’t notice it, someone might be issuing certificates that are valid for those machines. Also real CA’s also have ways to revoke certificates that are checked by browsers (OCSP and CRLs), they may employ other techniques such as cross signing and chains of trust. All those make it so a compromised certificate is revoked and not trusted by anyone after the fact.

@MigratingtoLemmy@lemmy.world
creator
link
fedilink
English
19M

I do realise the security problem in keeping the private key safe. I plan to use a VM with encrypted storage underneath. Do you think that’s OK for a homelab, or should I invest time into integrating HSM modules from Nitrokey?

@TCB13@lemmy.world
link
fedilink
English
19M

Why are you pushing for your own CA in the first place?

@MigratingtoLemmy@lemmy.world
creator
link
fedilink
English
19M

I would not like to use a public domain for my internal network. By extension, I do not want any public CA to know the domains and subdomains I use in my lab and home network

@TCB13@lemmy.world
link
fedilink
English
19M

Okay that’s fair but if your only concern is about “I do not want any public CA to know the domains and subdomains I use” you get around that.

Let’s Encrypt now allows for wildcard so you can probably do something like *.network.example.org and have an SSL certificate that will cover any subdomain under network.example.org (eg. host1.network.example.org). Or even better, get a wildcard like *.example.org and you’ll be done for everything.

I’m just suggesting this alternative because it would make your life way easier and potentially more secure without actually revealing internal subdomains to the CA.

Another option is to just issue certificates without a CA and accept them one at the time on each device. This won’t expose you to a possibly stolen CA PK and you’ll get notified if previously the accepted certificate of some host changes.

openssl req -x509 -nodes -newkey rsa:2048 \
-subj "/CN=$DOMAIN_BASE/O=$ORG_NAME/OU=$ORG_UNIT_NAME/C=$COUNTRY" \
-keyout $DOMAIN_BASE.key -out $DOMAIN_BASE.crt -days $OPT_days "${ALT_NAMES[@]}"
@MigratingtoLemmy@lemmy.world
creator
link
fedilink
English
29M

My apologies, I didn’t word my concerns properly in the moment. I would like to run a private CA simply because I do not want to use a public domain for my internal network. It makes me deeply uncomfortable to use a public domain and get public certificates for something inherently so private (it’s more philosophical than technical, although I suppose that’s where a lot of opinionated technical decisions come from anyway). Your solution is elegant and simple, but I really do want to do it completely internally, and move towards zero trust security practices as I do. Basically, I want to start training myself on the security side of SRE in my lab, running services which matter to me and working like the more paranoid SRE teams in corporations work…I know this sounds like fantasy, and my needs might change going forward, but I’m also hoping to learn a lot from this and hopefully make this as robust as I can. Having a good security posture makes me feel warm inside :)

@deepdive@lemmy.world
link
fedilink
English
6
edit-2
9M

If you want to run your own pki with self-signed certificate in your homelab I really encourage you to read through this tutorial. There is a lot to process and read and it will take you some time to set everything up and understand every terminology but after that:

  • Own self-signed certificate with SAN wildcards (https://*.home.lab)
  • Certificate chain of trust
  • CSR with your own configuration
  • CRL and certificate revocation
  • X509 extensions

After everything is in place, you can write your own script that revoks, write and generates your certificate, but that is another story !

Put everything behind your reverse proxy of choice (traefik in my case) and serve all your docker services with your own self-signed wildcard certificates ! It’s complex but if you have spare time and are willing to learn something new, it’s worth the effort !

Keep in mind to never expose such certificates on the wild wild west ! Keep those certificate in a closed homelab you access through a secure tunnel on your LAN !

edit

Always take notes, to keep track of what you did and how you solved some issues and always make some visuals to have a better understanding on how things work !

never expose such certificates on the wild wild west ! Keep those certificate in a closed homelab you access through a secure tunnel on your LAN !

I’m curious, what’s the reason?

@jdrch@lemmy.world
link
fedilink
English
09M

In many architectures in which certificates are used, a client with a valid certificate is a trusted client, so a certificate falling into the wrong hands is problematic.

@MigratingtoLemmy@lemmy.world
creator
link
fedilink
English
19M

Thank you. Could you explain a bit more about your setup and the aspects I should be looking at? Specifically:

  • Certificate chain of trust: I assume you’re talking about PKI infrastructure and using root CAs + Derivative CAs? If yes, then I must note that I’m not planning to run derivative CAs because it’s just for my lab and I don’t need that much of infrastructure.
  • I have yet to figure out the CRL part with OpenSSL, I’ll have to read more about it. Thanks.
  • I do not know what X.509 extensions are and why I need them. Could you tell me more?
  • I’m also considering client certificates as an alternative to SSO, am I right in considering them this way? I will also have to think about what I could do what clients without a client certificate or my root CA’s certificate in their certificate store (maybe run another instance which doesn’t need all of the encryption and setup I’m doing and somehow redirect such clients there whilst logging their traffic?).

Thanks for the mention, I was looking at a script to automate certificate generation and revocation too.

Since we’re talking about reverse-proxies, I’ll mention that I plan to run an instance of HAProxy per podman pod so that I terminate my encrypted traffic inside the pod and exclusively route unencrypted traffic through local host inside the pod. I’m doing this because I do not want to see any unencrypted traffic in my network. Of course, this is some more overhead but I think this is doable. I got this idea from another post I made a while back. Of course, that means that every pod on my network (hosting an HAProxy instance) will be given a distinct subdomain, and I will be producing certificates for specific subdomains, instead of using a wildcard.

Thanks, I’ll be sure to document my progress as I go.

@deepdive@lemmy.world
link
fedilink
English
2
edit-2
9M

Certificate chain of trust: I assume you’re talking about PKI infrastructure and using root CAs + Derivative CAs? If yes, then I must note that I’m not planning to run derivative CAs because it’s just for my lab and I don’t need that much of infrastructure.

An intermediate CA could potentially be useful, but isn’t really needed in self-signed CA. But in case you have to revoke your rootCA, you have to replace that certificate on all your devices, which can become a lot of hassle if you share that trusted root CA with family/friends. By having a intermediate CA and hiding your root CAs private key somewhere offline, you could take away that overheat by just revoking the intermediate CA and updating the server certificate with the newly signed Intermediate bundle and serving that new certificate through the proxy. (Hope that makes sense? :|)

I do not know what X.509 extensions are and why I need them. Could you tell me more?

This will probably give you some better explanation than I could :| I have everything written in a markdown file, and reading through my notes I remember I had to put some basic constraints TRUE in my certificates to make them work on my android root store ! Some are necessary to make your root CA work properly (like CA:True). Also if you want SAN certificates (multidomaine) you have to put them in your x509 extensions.

’m also considering client certificates as an alternative to SSO, am I right in considering them this way?

Ohhh, I don’t know… I haven’t installed or used any SSO service and thinking of MFA/SSO with authelia in the future ! My guess would be that those are 2 different technologies and could work together? Having self-signed CA with a 2FA could possible work in a homelab but I have no idea how because I haven’t tested it out. But thinks to consider if you want clients certificates for your family/friends is to have a intermediate CA in case of revocation, you don’t have to replace the certificate in their root store every time you sign a new Intermediate CA.

I’ll mention that I plan to run an instance of HAProxy per podman pod so that I terminate my encrypted traffic inside the pod and exclusively route unencrypted traffic through local host inside the pod.

I have no idea about HAProxy and podman and how they work to encrypt traffic. All my traffic passes through a wireguard tunnel to my docker containers/proxy which I consider safe enough? Listening to all my traffic with wireshark seamed to do exactly what I’m expecting but I’m not an expert :L So I cannot help you further on that topic. But I will keep your idea in my notes to see If there could be further improvement in my setup with HAProxy and podman compared to docker and traefik through wireguard tunnel.

Of course, that means that every pod on my network (hosting an HAProxy instance) will be given a distinct subdomain, and I will be producing certificates for specific subdomains, instead of using a wildcard.

Openssl SAN certificates are going to be a life/time saver in your setup ! One certificat with multidomian !


I’m just a hobby homelaber/tinkerer so take everything with caution and always double check with other sources ! :) Hope it helps !


Edit

Thinking of your use case I would personally create a rootCA and an intermediateCA + certificate bundle. Put the rootCA in the trusted store on all your devices and serve the intermediateCA/certificate bundle with your proxy of choice. Signing the certificate with SAN X.509 extension for all your domains. Save your rootCA’s key somwhere offline to keep it save !

The links I gave you are very useful but every bit of information is a bit dispatched and you have to combine them by yourself, but it’s a gold mine of information !

@MigratingtoLemmy@lemmy.world
creator
link
fedilink
English
29M

Thank you for your comment. My apologies in replying so late.

After reading a bit more and thinking about my setup, I think I will run intermediate CAs. Specifically, because I want to set up an ad-hoc mTLS setup, I might keep intermediate CAs for different classes of devices/different purposes. I will need to delve deeper into it, but for now, I think I have a grasp on the idea I need to implement, in which case, intermediate CAs will likely be a better idea. Thank you.

Thanks for the material, it would seem that I have a lot of reading left to do :)

@deepdive@lemmy.world
link
fedilink
English
19M

Hey don’t worry :)

Yeah, this could be a time saver in case you should/need to revoke certificates in your homelab setup ! Imagine changing the rootCA store on 20 devices … Ugh !

Happy reading/tweaking ! Have fun !

@MigratingtoLemmy@lemmy.world
creator
link
fedilink
English
19M

Hmm, I think I’m a bit confused now.

Let’s say I have 2 intermediary CAs: one to create certificates for my servers (going to be reverse-proxies + a couple of VMs), and one for my clients (Android devices, maybe a linux machine).

I’m planning to rotate both CAs on a bi-weekly schedule, and rotate the root CA every 6 months. In which case, wouldn’t I have to insert new certificates into my servers every time I rotate the intermediary “server” CA, and the same for my clients when I rotate the “client” CA? If I don’t do that, won’t I get SSL errors every time I try to access something because the certificate expired?

@MTK@lemmy.world
link
fedilink
English
3
edit-2
9M

Use something like no-ip, you can get a domain for free and renewing it every 30 days with a few clicks is much easier then managing a CA.

The only downside is the TLD but if you don’t care to much about how your domain name looks it really is the best option.

I use no-ip with letsencrypt, the LE bot does the certificate stuff for me, I use a single domain with different ports for each service and no-ip sends an email every 30 days to reconfirm the domain. Simple and easy.

@MigratingtoLemmy@lemmy.world
creator
link
fedilink
English
29M

I would not like to use a public domain for my internal traffic, even if the traffic is not routed outside. This is more of a personal choice than me looking for the easiest technical solution. I do own a domain, but I’ll keep that specifically for elements facing the internet.

Thanks!

@TechAdmin@lemmy.world
link
fedilink
English
39M

Self-host your own ACME server. Then you can use certbot pointed there.

These instructions are old so not sure if newer/better ways, https://blog.sean-wright.com/self-host-acme-server/

@MigratingtoLemmy@lemmy.world
creator
link
fedilink
English
19M

Thank you, I was looking to host Step-CA, whilst OpenSSL is another option. I’m also planning to combine it with a vault for secrets like Conjur and encrypt the volumes underneath. I want to reach the best security posture possible in this kind of setup

Create a post

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don’t control.

Rules:

  1. Be civil: we’re here to support and learn from one another. Insults won’t be tolerated. Flame wars are frowned upon.

  2. No spam posting.

  3. Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it’s not obvious why your post topic revolves around selfhosting, please include details to make it clear.

  4. Don’t duplicate the full text of your blog or github here. Just post the link for folks to click.

  5. Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).

  6. No trolling.

Resources:

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

  • 1 user online
  • 31 users / day
  • 84 users / week
  • 216 users / month
  • 846 users / 6 months
  • 1 subscriber
  • 1.42K Posts
  • 8.09K Comments
  • Modlog