Renewing Puppet Certificates
On March 7th of 2019, our SSL certificates for Puppet/Foreman and TLS LDAP all reached their five year expiration period. This has caused Puppet and Foreman to stop working and our authentication server cannot be accessed. I cannot create new users that are recognized by the cluster. And users cannot login to machines that they have previously never logged into. You can imagine the trouble this has caused. As of March 22, I have not solved this yet but I will be documenting my attempts to fix this.
Notes
I got a great deal of clues based on the configuration files located in the following areas:
/etc/foreman /etc/foreman/ssl (check the README file here. In our cluster, our Puppet certs and Foreman certs are different. You need to generate Foreman certs separately) /etc/puppet /etc/puppet/puppet.conf (tells you the puppet master and puppet client FQDNs for the puppetmaster) /etc/httpd/conf.d /etc/httpd/conf.d/05-foreman-ssl.conf (tells you what SSL certificates files foreman looks at) /etc/httpd/conf.d/25-puppet.conf (tells you what certificate name that the puppetmaster uses)
Procedure to Renew SSL Certificates on Puppet Master
First of all, stop httpd. Puppet master and Foreman run via httpd
[root@alpha ~]# service httpd stop
Get subject of original certificate
[root@alpha ~]# openssl x509 -in /var/lib/puppet/ssl/ca/ca_crt.pem -noout -subject subject= /CN=Puppet CA: alpha.ucsf.bkslab.org
Get serial of original certificate
[root@alpha ~]# openssl x509 -in /var/lib/puppet/ssl/ca/ca_crt.pem -noout -serial serial=01
Extract info from puppetmaster cert
[root@alpha ~]# openssl x509 -in /var/lib/puppet/ssl/certs/puppetmaster.cluster.ucsf.bkslab.org.pem -text -noout \ -certopt no_subject,no_header,no_version,no_serial,no_signame,no_validity,no_subject,no_issuer,no_pubkey,no_sigdump,no_aux
Create directories to stage new valid periods (I determined puppetmaster.cluster.ucsf.bkslab.org.pem was the .pem file we need because certname in [master] section of /etc/puppet/puppet.conf has this name)
[root@alpha ~]# mkdir /root/puppet_renewal [root@alpha ~]# cd /root/puppet_renewal [root@alpha ~]# mkdir /root/puppet_renewal/ca [root@alpha ~]# mkdir /root/puppet_renewal/puppetmaster [root@alpha ~]# mkdir /root/puppet_renewal/puppetmaster/private_keys [root@alpha ~]# mkdir /root/puppet_renewal/puppetmaster/certs
Copy the existing certificate authority's key and the puppetmaster's private key (Only certificates expire. Private keys do not so they can be reused).
[root@alpha ~]# cp /var/lib/puppet/ssl/ca/ca_key.pem /root/puppet_renewal/ca [root@alpha ~]# cp /var/lib/puppet/ssl/private_keys/mypuppetmaster.cluster.ucsf.bkslab.org.pem /root/puppet_renewal/puppetmaster/private_keys
Create an openssl configuration file
[root@alpha ~]# vi puppet_renewal/renewpuppet.cnf renewpuppet.cnf [ v3_ca ] basicConstraints= CA:TRUE subjectKeyIdentifier= hash # authorityKeyIdentifier= keyid:always,issuer:always keyUsage = critical, cRLSign, keyCertSign nsComment = 'Puppet Ruby/OpenSSL Internal Certificate' [ v3 ] basicConstraints= CA:FALSE subjectKeyIdentifier= hash nsComment = 'Puppet Ruby/OpenSSL Internal Certificate' keyUsage = critical, digitalSignature, keyEncipherment extendedKeyUsage = critical, serverAuth, clientAuth subjectAltName = @alt_names [ alt_names ] DNS.1 = puppet DNS.2 = puppet.ucsf.bkslab.org DNS.3 = puppet.cluster.ucsf.bkslab.org DNS.4 = puppetmaster DNS.5 = puppetmaster.ucsf.bkslab.org DNS.6 = puppetmaster.cluster.ucsf.bkslab.org DNS.7 = alpha DNS.8 = alpha.ucsf.bkslab.org DNS.9 = alpha.cluster.ucsf.bkslab.org DNS.10 = alpha.compbio.ucsf.edu
Create certificate signing request with existing files
[root@alpha ~]# openssl req -key /root/puppet_renewal/ca/ca_key.pem -new -batch -subj "/CN=Puppet CA: alpha.ucsf.bkslab.org" -out /root/puppet_renewal/ca/ca_new.csr
Create a new CA certificate
[root@alpha ~]#openssl x509 -req -days 3650 -in /root/puppet_renewal/ca/ca_new.csr -signkey /root/puppet_renewal/ca/ca_key.pem \ -out /root/puppet_renewal/ca/ca_crt.pem -extfile /root/puppet_renewal/renewpuppet.cnf -extensions v3_ca -set_serial 1 Signature ok subject=/CN=Puppet CA: alpha.ucsf.bkslab.org Getting Private key
Get serial number for your existing CA
[root@alpha puppet_renewal]# echo $((0x`cat /var/lib/puppet/ssl/ca/serial`)) 451
Create new certificate signing request with puppet server's key
[root@alpha ~]# openssl req -key /root/puppet_renewal/puppetmaster/private_keys/puppetmaster.cluster.ucsf.bkslab.org.pem -new \ -batch -subj "/CN=alpha.ucsf.bkslab.org" -out /root/puppet_renewal/mypuppetmaster.csr
Create new puppet master's certificate
[root@alpha ~]# openssl x509 -extfile /root/puppet_renewal/renewpuppet.cnf -extensions v3 -req -days 1825 -in /root/puppet_renewal/mypuppetmaster.csr \ -CA /root/puppet_renewal/ca/ca_crt.pem -CAkey /root/puppet_renewal/ca/ca_key.pem -CAcreateserial \ -out /root/puppet_renewal/puppetmaster/certs/puppetmaster.cluster.ucsf.bkslab.org.pem -sha256 -set_serial 451
Replace Puppet's ca_crt.pem, ca.pem, and puppetmaster.pem
[root@alpha ~]# cp puppet_renewal/ca/ca_crt.pem /var/lib/puppet/ssl/ca/ca_crt.pem [root@alpha ~]# puppet_renewal/ca/ca_crt.pem /var/lib/puppet/ssl/certs/ca.pem [root@alpha ~]# cp puppet_renewal/puppetmaster/certs/puppetmaster.cluster.ucsf.bkslab.org.pem /var/lib/puppet/ssl/certs/puppetmaster.cluster.ucsf.bkslab.org.pem
Restart httpd
# If this fails, a mistake was made. Check /var/log/httpd. [root@alpha ~]# service httpd start Starting httpd: [Wed Mar 27 12:28:24 2019] [warn] module passenger_module is already loaded, skipping [ OK ]
Renewing Foreman's certificates (in progress)
Investigate /etc/foreman/ssl/README cluster2 on foreman
Follow the commands in the README file while in the /etc/foreman/ssl directory. Only do this step after you've done the Puppet certificates in the previous section. The foreman related commands here rely on the ca.pem generated by the previous puppet commands.
#openssl genrsa -aes128 -out foreman.key 2048 openssl genrsa -out foreman.key 2048 openssl req -new -out foreman.csr -key foreman.key -config foreman.cnf openssl req -text -in foreman.csr -noout openssl x509 -req -in foreman.csr -CA /var/lib/puppet/ssl/ca/ca_crt.pem -CAkey /var/lib/puppet/ssl/ca/ca_key.pem -CAserial /var/lib/puppet/ssl/ca/serial -out foreman.crt -days 1730 openssl rsa -in foreman.key -out foreman.key-unlocked ln -s foreman.key-unlocked key.pem ln -s foreman.crt cert.pem
Restart httpd when this is done.
Further Reading
A big thanks to these two blogs for pointing me in the right direction:
Sean the Sysadmin: http://www.scrosby.com/2017/06/renewing-puppet-ca-and-puppet-master.html Flying Circus: https://blog.flyingcircus.io/2017/09/01/how-to-renew-puppet-ca-and-server-certificates-in-place/