Renewing Puppet Certificates: Difference between revisions
Jgutierrez6 (talk | contribs) Tag: Manual revert |
|||
| Line 288: | Line 288: | ||
# Delete old certificates | # Delete old certificates | ||
[root@mk-auth ~]# certutil -D d /etc/dirsrv/slapd-mk-auth n "Cluster PuppetCA Certificate" | [root@mk-auth ~]# certutil -D -d /etc/dirsrv/slapd-mk-auth -n "Cluster PuppetCA Certificate" | ||
[root@mk-auth ~]# certutil -D -d /etc/dirsrv/slapd-mk-auth/ -n "mk-auth.c.keiserlab.org" | [root@mk-auth ~]# certutil -D -d /etc/dirsrv/slapd-mk-auth/ -n "mk-auth.c.keiserlab.org" | ||
Latest revision as of 20:48, 10 May 2024
On March 7th of 2019, our SSL certificates for Puppet/Foreman and TLS LDAP all reached their five year expiration period. This has caused Puppet and Foreman to stop working and our authentication server cannot be accessed. I cannot create new users that are recognized by the cluster. And users cannot login to machines that they have previously never logged into. You can imagine the trouble this has caused. As of March 22, I have not solved this yet but I will be documenting my attempts to fix this.
Notes
I got a great deal of clues based on the configuration files located in the following areas:
/etc/foreman /etc/foreman/ssl (check the README file here. In our cluster, our Puppet certs and Foreman certs are different. You need to generate Foreman certs separately) /etc/puppet /etc/puppet/puppet.conf (tells you the puppet master and puppet client FQDNs for the puppetmaster) /etc/httpd/conf.d /etc/httpd/conf.d/05-foreman-ssl.conf (tells you what SSL certificates files foreman looks at) /etc/httpd/conf.d/25-puppet.conf (tells you what certificate name that the puppetmaster uses)
Procedure to Renew SSL Certificates on Puppet Master
First of all, stop httpd. Puppet master and Foreman run via httpd
[root@alpha ~]# service httpd stop
Get subject of original certificate
[root@alpha ~]# openssl x509 -in /var/lib/puppet/ssl/ca/ca_crt.pem -noout -subject subject= /CN=Puppet CA: alpha.ucsf.bkslab.org
Get serial of original certificate
[root@alpha ~]# openssl x509 -in /var/lib/puppet/ssl/ca/ca_crt.pem -noout -serial serial=01
Extract info from puppetmaster cert
[root@alpha ~]# openssl x509 -in /var/lib/puppet/ssl/certs/puppetmaster.cluster.ucsf.bkslab.org.pem -text -noout \ -certopt no_subject,no_header,no_version,no_serial,no_signame,no_validity,no_subject,no_issuer,no_pubkey,no_sigdump,no_aux
Create directories to stage new valid periods (I determined puppetmaster.cluster.ucsf.bkslab.org.pem was the .pem file we need because certname in [master] section of /etc/puppet/puppet.conf has this name)
[root@alpha ~]# mkdir /root/puppet_renewal [root@alpha ~]# cd /root/puppet_renewal [root@alpha ~]# mkdir /root/puppet_renewal/ca [root@alpha ~]# mkdir /root/puppet_renewal/puppetmaster [root@alpha ~]# mkdir /root/puppet_renewal/puppetmaster/private_keys [root@alpha ~]# mkdir /root/puppet_renewal/puppetmaster/certs
Copy the existing certificate authority's key and the puppetmaster's private key (Only certificates expire. Private keys do not so they can be reused).
[root@alpha ~]# cp /var/lib/puppet/ssl/ca/ca_key.pem /root/puppet_renewal/ca [root@alpha ~]# cp /var/lib/puppet/ssl/private_keys/mypuppetmaster.cluster.ucsf.bkslab.org.pem /root/puppet_renewal/puppetmaster/private_keys
Create an openssl configuration file
[root@alpha ~]# vi puppet_renewal/renewpuppet.cnf renewpuppet.cnf [ v3_ca ] basicConstraints= CA:TRUE subjectKeyIdentifier= hash # authorityKeyIdentifier= keyid:always,issuer:always keyUsage = critical, cRLSign, keyCertSign nsComment = 'Puppet Ruby/OpenSSL Internal Certificate' [ v3 ] basicConstraints= CA:FALSE subjectKeyIdentifier= hash nsComment = 'Puppet Ruby/OpenSSL Internal Certificate' keyUsage = critical, digitalSignature, keyEncipherment extendedKeyUsage = critical, serverAuth, clientAuth subjectAltName = @alt_names [ alt_names ] DNS.1 = puppet DNS.2 = puppet.ucsf.bkslab.org DNS.3 = puppet.cluster.ucsf.bkslab.org DNS.4 = puppetmaster DNS.5 = puppetmaster.ucsf.bkslab.org DNS.6 = puppetmaster.cluster.ucsf.bkslab.org DNS.7 = alpha DNS.8 = alpha.ucsf.bkslab.org DNS.9 = alpha.cluster.ucsf.bkslab.org DNS.10 = alpha.compbio.ucsf.edu
Create certificate signing request with existing files
[root@alpha ~]# openssl req -key /root/puppet_renewal/ca/ca_key.pem -new -batch -subj "/CN=Puppet CA: alpha.ucsf.bkslab.org" -out /root/puppet_renewal/ca/ca_new.csr
Create a new CA certificate
[root@alpha ~]#openssl x509 -req -days 3650 -in /root/puppet_renewal/ca/ca_new.csr -signkey /root/puppet_renewal/ca/ca_key.pem \ -out /root/puppet_renewal/ca/ca_crt.pem -extfile /root/puppet_renewal/renewpuppet.cnf -extensions v3_ca -set_serial 1 Signature ok subject=/CN=Puppet CA: alpha.ucsf.bkslab.org Getting Private key
Get serial number for your existing CA
[root@alpha puppet_renewal]# echo $((0x`cat /var/lib/puppet/ssl/ca/serial`)) 451
Create new certificate signing request with puppet server's key
[root@alpha ~]# openssl req -key /root/puppet_renewal/puppetmaster/private_keys/puppetmaster.cluster.ucsf.bkslab.org.pem -new \ -batch -subj "/CN=alpha.ucsf.bkslab.org" -out /root/puppet_renewal/mypuppetmaster.csr
Create new puppet master's certificate
[root@alpha ~]# openssl x509 -extfile /root/puppet_renewal/renewpuppet.cnf -extensions v3 -req -days 1825 -in /root/puppet_renewal/mypuppetmaster.csr \ -CA /root/puppet_renewal/ca/ca_crt.pem -CAkey /root/puppet_renewal/ca/ca_key.pem -CAcreateserial \ -out /root/puppet_renewal/puppetmaster/certs/puppetmaster.cluster.ucsf.bkslab.org.pem -sha256 -set_serial 451
Replace Puppet's ca_crt.pem, ca.pem, and puppetmaster.pem
[root@alpha ~]# cp puppet_renewal/ca/ca_crt.pem /var/lib/puppet/ssl/ca/ca_crt.pem [root@alpha ~]# puppet_renewal/ca/ca_crt.pem /var/lib/puppet/ssl/certs/ca.pem [root@alpha ~]# cp puppet_renewal/puppetmaster/certs/puppetmaster.cluster.ucsf.bkslab.org.pem /var/lib/puppet/ssl/certs/puppetmaster.cluster.ucsf.bkslab.org.pem
Restart httpd
# If this fails, a mistake was made. Check /var/log/httpd.
[root@alpha ~]# service httpd start
Starting httpd: [Wed Mar 27 12:28:24 2019] [warn] module passenger_module is already loaded, skipping
[ OK ]
Renewing Foreman's certificates (in progress)
Investigate /etc/foreman/ssl/README cluster2 on foreman
Follow the commands in the README file while in the /etc/foreman/ssl directory. Only do this step after you've done the Puppet certificates in the previous section. The foreman related commands here rely on the ca.pem generated by the previous puppet commands.
#openssl genrsa -aes128 -out foreman.key 2048 openssl genrsa -out foreman.key 2048 openssl req -new -out foreman.csr -key foreman.key -config foreman.cnf openssl req -text -in foreman.csr -noout openssl x509 -req -in foreman.csr -CA /var/lib/puppet/ssl/ca/ca_crt.pem -CAkey /var/lib/puppet/ssl/ca/ca_key.pem -CAserial /var/lib/puppet/ssl/ca/serial -out foreman.crt -days 1730 openssl rsa -in foreman.key -out foreman.key-unlocked
# this section doesn't need to be done if they symlinks already exist ln -s foreman.key-unlocked key.pem ln -s foreman.crt cert.pem
Restart httpd when this is done.
Clearing old certificates
When both the Puppet & Foreman certificates are renewed, you will have an entire cluster of machines that have certificates that reference the old ca.pem. You have to regenerate them all. Follow the instructions on: http://wiki.docking.org/index.php/PuppetTricks in the Regenerating a Certificate page. Track which ones you do as it'd be annoying to do any machine more than once. Make sure to restart sssd after a puppet client run.
Libvirt, Hypervisors, and TLS
alpha/puppetmaster is used to manage the VMs. To add to the hassle of renewing Puppet certificates, we also have to copy the renewed certificates to the libvirt-relevant sections on both the puppetmaster AND the hypervisors. Otherwise, foreman cannot interact with the hypervisors/VMs.
Only do this once on the puppetmaster (hostname will depend on which name you used while regenerating SSL certs. I used alpha.ucsf.bkslab.org.pem):
copy /var/lib/puppet/ssl/certs/<hostname of puppetmaster>.pem to /etc/pki/libvirt/clientcert.pem copy /var/lib/puppet/ssl/private_keys/alpha.ucsf.bkslab.org.pem to /etc/pki/libvirt/private/clientkey.pem
We have several hypervisors, so this may need to be repeated on each one. Do this on the hypervisors:
copy /var/lib/puppet/ssl/certs/ca.pem to /etc/pki/CA/cacert.pem copy /var/lib/puppet/ssl/certs/<hostname>.pem to /etc/pki/libvirt/servercert.pem copy /var/lib/puppet/ssl/private_keys/<hostname>.pem to /etc/pki/libvirt/private/serverkey.pem service libvirtd restart (if you did anything wrong, this will fail. Be careful)
Possible Other Errors
I was getting this when trying to remove a host: "Remove Reverse DNS record for n-5-35.cluster.ucsf.bkslab.org task failed with the following error: ERF12-1261 [ProxyAPI::ProxyException]: Unable to delete DNS entry ([OpenSSL::SSL::SSLError]: SSL_connect returned=1 errno=0 state=SSLv3 read server session ticket A: sslv3 alert ce...) for proxy https://foreman.cluster.ucsf.bkslab.org:8443/dns"
This was a foreman SSL error. This happened because foreman was still using the old SSL certs that existed in /var/lib/puppet/ssl/certs. It was referencing a certificate file called alpha.cluster.ucsf.bkslab.org. The certificate I renewed was called alpha.ucsf.bkslab.org. I had to go to foreman.ucsf.bkslab.org then click Administer -> Settings -> Auth. I had to adjust the lines for ssl_certificate and ssl_priv_key to point to the new certificates I made. Then, foreman-proxy was working again.
Further Reading
A big thanks to these two blogs for pointing me in the right direction:
Sean the Sysadmin: http://www.scrosby.com/2017/06/renewing-puppet-ca-and-puppet-master.html Flying Circus: https://blog.flyingcircus.io/2017/09/01/how-to-renew-puppet-ca-and-server-certificates-in-place/
2024 Reattempt to Renew Certificates
Ben Wong emailed us these set of guides.
- Khanh and I followed these set of instructions first
- Then we followed the Edit Certificate Databases on LDAPS server mk-auth
This page is in regards to SSL certs that are created by Puppet which are then used by Foreman and LDAPS. SSL Certs expire every five years. On April 16, 2019, our SSL certificates will reach their five year expiration date. You can see the SSL certificate expiration date by logging into mk-manage and doing the following command: # This location is where puppet will generate a certificate authority file [root@mk-manage ~]# openssl x509 -enddate -noout -in /var/lib/puppet/ssl/certs/ca.pem notAfter=Apr 16 04:34:02 2019 GMT Prior to this expiration date, you must act! Generating New Puppet CAs and Certificates on Puppet Master Puppet certificates and CAs are stored in mk-manage, the Puppet/Foreman server. Puppet stores the certificates in /var/lib/puppet/ssl. I backup the Puppet SSL directory first. If I screw up anywhere, I can simply rsync the directory back and things should be fine. [root@mk-manage puppet]# rsync -Phav /var/lib/puppet/ssl /var/lib/puppet/ssl-old-20190411 . . . ssl/public_keys/mk-nas-1.compbio.ucsf.edu.pem 775 100% 5.69kB/s 0:00:00 (xfer#34, to-check=0/45) sent 68.67K bytes received 702 bytes 138.75K bytes/sec total size is 65.68K speedup is 0.95 # Stop Puppet agent and stop httpd: [root@mk-manage puppet]# service puppet stop [root@mk-manage puppet]# service httpd stop # Remove the old Puppet's SSL directory [root@mk-manage puppet]# mv /var/lib/puppet/ssl /tmp # Start a new puppet master. This generates a new certificate authority. [root@mk-manage puppet]# puppet master --no-daemonize --verbose (Issue a ctrl+C when the output has stopped) From here, you should have a new /var/lib/puppet/ssl directory complete with new certificates and a new ca.pem. Check /var/lib/puppet/ssl/certs. This directory will have a ca.pem and a certificate local to mk-manage. # The new ca.pem will have a new expiration date [root@mk-manage ~]# openssl x509 -enddate -noout -in /var/lib/puppet/ssl/certs/ca.pem notAfter=Apr 12 23:40:00 2024 GMT Syncing Puppet Clients to new Puppet Master After we've created a new SSL directory for mk-manage, we'll need to sync our puppet clients to the new Puppet Master. The puppet clients will likely have a similar /var/lib/puppet/ssl/certs directory but the ca.pem within it will be the old one while the host certificate would have been created with the old ca.pem. We'll have to create new SSL certificates that can work with the Puppet Master's new certs. On the Puppet Client: [bwong1@mk-1-a ~]$ sudo rsync -Phav /var/lib/puppet/ssl /var/lib/puppet/ssl-old-20190415 [bwong1@mk-1-a ~]$ sudo mv /var/lib/puppet/ssl /tmp [bwong1@mk-1-a ~]$ sudo puppet agent --test --waitforcert=30 The above command will generate a new puppet certificate request to the puppet master. The puppet master is defined in /etc/puppet/puppet.conf as server and ca_server. We add the --waitforcert option so that we have enough time for the puppetmaster to recognize there is a certificate request. On the Puppet Master: # View the pending certificate requests that need to be signed then sign them. [bwong1@mk-manage ~]$ sudo puppet cert list [bwong1@mk-manage ~]$ sudo puppet cert sign mk-1-a.compbio.ucsf.edu Once the certificate request is signed, mk-1-a's puppet agent will reauthenticate with mk-manage's master and will begin syncing. One of the things that should change is that /etc/openldap/cacerts/ca.pem will change to the new ca.pem that we created earlier. This file is important as it allows a host to connect to the LDAP Server. After this is done, you should restart two services: [bwong1@mk-1-a ~]$ sudo service sssd restart [bwong1@mk-1-a ~]$ sudo service puppet restart This will ensure that the host will reauthenticate with the LDAP server with the SSL/TLS and also ensure Puppet is running with the latest certificates recognized. Edit Certificate Databases on LDAPS server mk-auth Before doing this, make sure you run the Puppet Client instructions on the LDAP server mk-auth first. This way, mk-auth will have the latest Puppet certificates that it can use to authenticate with. There are two certificate databases on mk-auth. /etc/dirsrv/admin-serv - directory service administration database /etc/dirsrv/slapd-mk-auth - directory service database You can look at the certificates that exist in this database with the following command: [root@mk-auth ~]# certutil -L -d /etc/dirsrv/admin-serv/ Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI Cluster PuppetCA Certificate CT,, mk-auth.c.keiserlab.org u,u,u [root@mk-auth ~]# certutil -L -d /etc/dirsrv/slapd-mk-auth/ Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI mk-auth.c.keiserlab.org u,u,u Cluster PuppetCA Certificate CT,, If you get messages about these certificates expiring, you'll have to delete them first. Otherwise, if you try to add a new certificate while the old one is there, you'll get a message with the following: 'certutil: could not decode certificate: SEC_ERROR_REUSED_ISSUER_AND_SERIAL: You are attempting to import a cert with the same issuer/serial as an existing cert, but that is not the same cert.' So firstly, I'll delete the existing Cluster PuppetCA Certificate and the mk-auth.c.keiserlab.org certificate from the cert databases. Remember to rsync the /etc/dirsrv directory as a backup prior to doing any of this. You have to do these steps for both /etc/dirsrv/admin-serv and /etc/dirsrv/slapd-mk-auth. For slapd-mk-auth: # Delete old certificates [root@mk-auth ~]# certutil -D -d /etc/dirsrv/slapd-mk-auth -n "Cluster PuppetCA Certificate" [root@mk-auth ~]# certutil -D -d /etc/dirsrv/slapd-mk-auth/ -n "mk-auth.c.keiserlab.org" # import new puppet's ca.pem file [root@mk-auth ~]# certutil -d /etc/dirsrv/slapd-mk-auth -A -n "Cluster PuppetCA Certificate" -t CT,, -a -i /var/lib/puppet/ssl/certs/ca.pem # Creates a p12 file certificate. This command will ask you for a password that encrypts the private key. # the input is mk-auth's .pem file. Puppet should've made this already if you followed earlier steps. [root@mk-auth ~]# openssl pkcs12 -export -in /var/lib/puppet/ssl/certs/$( hostname ).pem \ -inkey /var/lib/puppet/ssl/private_keys/$( hostname ).pem \ -out /etc/pki/tls/private/$( hostname ).p12 # imports .p12 file into the certificate. This file will show up as mk-auth.c.keiserlab.org [root@mk-auth ~] pk12util -i /etc/pki/tls/private/$( hostname ).p12 -d /etc/dirsrv/slapd-mk-auth # Check that you have both a new certificate authority file and a new host certitificate [root@mk-auth SETUP]# certutil -L -d /etc/dirsrv/slapd-mk-auth/ Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI mk-auth.c.keiserlab.org u,u,u Cluster PuppetCA Certificate CT,, For admin-srv: # Deletes the old certificates in directory admin-serv. Use -n to specifiy the 'nickname' of a certificate. Nicknames are found in certutil -L -d output. [root@mk-auth ~]# certutil -D -d /etc/dirsrv/admin-serv/ -n "Cluster PuppetCA Certificate" [root@mk-auth ~]# certutil -D -d /etc/dirsrv/admin-serv/ -n "mk-auth.c.keiserlab.org" # imports a certificate authority that Puppet generated [root@mk-auth ~]# certutil -d /etc/dirsrv/admin-serv -A -n "Cluster PuppetCA Certificate" -t CT,, -a -i /var/lib/puppet/ssl/certs/ca.pem # -d directory: specify database directory containing the certificate # -A: Add an existing certificate to a certificate database. Cert database should already exist. # -n nickname: specify nickname of a certificate or key to list, create, add to a database or modify or validate # -t trustargs: specify trust attributes to apply to certificate when creating # -C: Trusted & Valid CA # -T: trusted CA for client authentication # -a: use ASCII format or allow the use of ASCII format for input/output # -i input_file: pass input file to the command. Input file can be specific cert, cert request file, or batch file of commands # Import .p12 certificate into admin-serv. It will ask you for the password you used when you created the private key file [root@mk-auth ~]# pk12util -i /etc/pki/tls/private/$( hostname ).p12 -d /etc/dirsrv/admin-serv/ Enter password for PKCS12 file: pk12util: no nickname for cert in PKCS12 file. pk12util: using nickname: mk-auth.c.keiserlab.org pk12util: PKCS12 IMPORT SUCCESSFUL # -i p12file: import keys/certs from a pkcs12 file into security database # -d directory: specify the database directory into which to import or export from certs and keys Once these certificates are set, restart the directory service daemon and the ds-admin daemon. [root@mk-auth ~]# service dirsrv restart Shutting down dirsrv: Starting dirsrv: [ OK ] [root@mk-auth ~]# service dirsrv-admin restart Shutting down dirsrv-admin: Starting dirsrv-admin: [ OK ]