Renewing Puppet Certificates: Difference between revisions

From DISI
Jump to navigation Jump to search
m (→‎Libvirt, Hypervisors, and TLS: correction of paths)
 
(13 intermediate revisions by the same user not shown)
Line 1: Line 1:
On March 7th of 2019, our SSL certificates for Puppet/Foreman and TLS LDAP all reached their five year expiration period.  This has caused Puppet and Foreman to stop working and our authentication server cannot be accessed.  I cannot create new users that are recognized by the cluster.  And users cannot login to machines that they have previously never logged into.  You can imagine the trouble this has caused.  As of March 22, I have not solved this yet but I will be documenting my attempts to fix this.   
On March 7th of 2019, our SSL certificates for Puppet/Foreman and TLS LDAP all reached their five year expiration period.  This has caused Puppet and Foreman to stop working and our authentication server cannot be accessed.  I cannot create new users that are recognized by the cluster.  And users cannot login to machines that they have previously never logged into.  You can imagine the trouble this has caused.  As of March 22, I have not solved this yet but I will be documenting my attempts to fix this.   
===Notes===
I got a great deal of clues based on the configuration files located in the following areas:
/etc/foreman
/etc/foreman/ssl                      (check the README file here.  In our cluster, our Puppet certs and Foreman certs are different.  You need to generate Foreman certs separately)
/etc/puppet
/etc/puppet/puppet.conf              (tells you the puppet master and puppet client FQDNs for the puppetmaster)
/etc/httpd/conf.d
/etc/httpd/conf.d/05-foreman-ssl.conf (tells you what SSL certificates files foreman looks at)
/etc/httpd/conf.d/25-puppet.conf      (tells you what certificate name that the puppetmaster uses)


===Procedure to Renew SSL Certificates on Puppet Master===
===Procedure to Renew SSL Certificates on Puppet Master===
Line 17: Line 27:
  -certopt no_subject,no_header,no_version,no_serial,no_signame,no_validity,no_subject,no_issuer,no_pubkey,no_sigdump,no_aux
  -certopt no_subject,no_header,no_version,no_serial,no_signame,no_validity,no_subject,no_issuer,no_pubkey,no_sigdump,no_aux


Create directories to stage new valid periods (I determined puppetmaster.cluster.ucsf.bkslab.org.pem was the .pem file we need because it had the most DNS alt names)
Create directories to stage new valid periods (I determined puppetmaster.cluster.ucsf.bkslab.org.pem was the .pem file we need because certname in [master] section of /etc/puppet/puppet.conf has this name)
  [root@alpha ~]# mkdir /root/puppet_renewal
  [root@alpha ~]# mkdir /root/puppet_renewal
  [root@alpha ~]# cd /root/puppet_renewal
  [root@alpha ~]# cd /root/puppet_renewal
Line 78: Line 88:


Create new puppet master's certificate
Create new puppet master's certificate
  [root@alpha ~]# # openssl x509 -extfile /root/puppet_renewal/renewpuppet.cnf -extensions v3 -req -days 1825 -in /root/puppet_renewal/mypuppetmaster.csr \
  [root@alpha ~]# openssl x509 -extfile /root/puppet_renewal/renewpuppet.cnf -extensions v3 -req -days 1825 -in /root/puppet_renewal/mypuppetmaster.csr \
  -CA /root/puppet_renewal/ca/ca_crt.pem -CAkey /root/puppet_renewal/ca/ca_key.pem -CAcreateserial \
  -CA /root/puppet_renewal/ca/ca_crt.pem -CAkey /root/puppet_renewal/ca/ca_key.pem -CAcreateserial \
  -out /root/puppet_renewal/puppetmaster/certs/puppetmaster.cluster.ucsf.bkslab.org.pem -sha256 -set_serial 451
  -out /root/puppet_renewal/puppetmaster/certs/puppetmaster.cluster.ucsf.bkslab.org.pem -sha256 -set_serial 451
Line 88: Line 98:


Restart httpd
Restart httpd
# If this fails, a mistake was made.  Check /var/log/httpd. 
  [root@alpha ~]# service httpd start
  [root@alpha ~]# service httpd start
Starting httpd: [Wed Mar 27 12:28:24 2019] [warn] module passenger_module is already loaded, skipping
                                                          [  OK  ]


===Renewing Foreman's certificates (in progress)===
===Renewing Foreman's certificates (in progress)===
Investigate /etc/foreman/ssl/README
Investigate /etc/foreman/ssl/README
cluster2 on foreman
cluster2 on foreman
Follow the commands in the README file while in the /etc/foreman/ssl directory.  Only do this step after you've done the Puppet certificates in the previous section. 
The foreman related commands here rely on the ca.pem generated by the previous puppet commands. 


  #openssl genrsa -aes128 -out foreman.key 2048
  #openssl genrsa -aes128 -out foreman.key 2048
Line 101: Line 117:
   
   
  openssl rsa -in foreman.key -out foreman.key-unlocked
  openssl rsa -in foreman.key -out foreman.key-unlocked
   
 
  # this section doesn't need to be done if they symlinks already exist
  ln -s foreman.key-unlocked key.pem
  ln -s foreman.key-unlocked key.pem
  ln -s foreman.crt cert.pem
  ln -s foreman.crt cert.pem


Restart httpd when this is done.
===Clearing old certificates===
When both the Puppet & Foreman certificates are renewed, you will have an entire cluster of machines that have certificates that reference the old ca.pem.  You have to regenerate them all.  Follow the instructions on: http://wiki.docking.org/index.php/PuppetTricks in the Regenerating a Certificate page.  Track which ones you do as it'd be annoying to do any machine more than once.  Make sure to restart sssd after a puppet client run. 
===Libvirt, Hypervisors, and TLS===
alpha/puppetmaster is used to manage the VMs.  To add to the hassle of renewing Puppet certificates, we also have to copy the renewed certificates to the libvirt-relevant sections on both the puppetmaster AND the hypervisors.  Otherwise, foreman cannot interact with the hypervisors/VMs. 
Only do this once on the puppetmaster (hostname will depend on which name you used while regenerating SSL certs.  I used alpha.ucsf.bkslab.org.pem):
  copy /var/lib/puppet/ssl/certs/<hostname of puppetmaster>.pem to /etc/pki/libvirt/clientcert.pem
  copy /var/lib/puppet/ssl/private_keys/alpha.ucsf.bkslab.org.pem to /etc/pki/libvirt/private/clientkey.pem
We have several hypervisors, so this may need to be repeated on each one.  Do this on the hypervisors:
  copy /var/lib/puppet/ssl/certs/ca.pem to /etc/pki/CA/cacert.pem
  copy /var/lib/puppet/ssl/certs/<hostname>.pem to /etc/pki/libvirt/servercert.pem
  copy /var/lib/puppet/ssl/private_keys/<hostname>.pem to /etc/pki/libvirt/private/serverkey.pem
  service libvirtd restart (if you did anything wrong, this will fail.  Be careful)
===Possible Other Errors===
I was getting this when trying to remove a host: "Remove Reverse DNS record for n-5-35.cluster.ucsf.bkslab.org task failed with the following error: ERF12-1261 [ProxyAPI::ProxyException]: Unable to delete DNS entry ([OpenSSL::SSL::SSLError]: SSL_connect returned=1 errno=0 state=SSLv3 read server session ticket A: sslv3 alert ce...) for proxy https://foreman.cluster.ucsf.bkslab.org:8443/dns"
This was a foreman SSL error.  This happened because foreman was still using the old SSL certs that existed in /var/lib/puppet/ssl/certs.  It was referencing a certificate file called alpha.cluster.ucsf.bkslab.org.  The certificate I renewed was called alpha.ucsf.bkslab.org.  I had to go to foreman.ucsf.bkslab.org then click Administer -> Settings -> Auth.  I had to adjust the lines for ssl_certificate and ssl_priv_key to point to the new certificates I made.  Then, foreman-proxy was working again.
===Further Reading===
A big thanks to these two blogs for pointing me in the right direction:
Sean the Sysadmin: http://www.scrosby.com/2017/06/renewing-puppet-ca-and-puppet-master.html
Flying Circus: https://blog.flyingcircus.io/2017/09/01/how-to-renew-puppet-ca-and-server-certificates-in-place/
[[Category:Sysadmin]]
[[Category:Sysadmin]]

Latest revision as of 23:22, 10 May 2019

On March 7th of 2019, our SSL certificates for Puppet/Foreman and TLS LDAP all reached their five year expiration period. This has caused Puppet and Foreman to stop working and our authentication server cannot be accessed. I cannot create new users that are recognized by the cluster. And users cannot login to machines that they have previously never logged into. You can imagine the trouble this has caused. As of March 22, I have not solved this yet but I will be documenting my attempts to fix this.

Notes

I got a great deal of clues based on the configuration files located in the following areas:

/etc/foreman
/etc/foreman/ssl                      (check the README file here.  In our cluster, our Puppet certs and Foreman certs are different.  You need to generate Foreman certs separately)
/etc/puppet
/etc/puppet/puppet.conf               (tells you the puppet master and puppet client FQDNs for the puppetmaster)
/etc/httpd/conf.d
/etc/httpd/conf.d/05-foreman-ssl.conf (tells you what SSL certificates files foreman looks at)
/etc/httpd/conf.d/25-puppet.conf      (tells you what certificate name that the puppetmaster uses)

Procedure to Renew SSL Certificates on Puppet Master

First of all, stop httpd. Puppet master and Foreman run via httpd

[root@alpha ~]# service httpd stop

Get subject of original certificate

[root@alpha ~]# openssl x509 -in /var/lib/puppet/ssl/ca/ca_crt.pem -noout -subject
subject= /CN=Puppet CA: alpha.ucsf.bkslab.org

Get serial of original certificate

[root@alpha ~]# openssl x509 -in /var/lib/puppet/ssl/ca/ca_crt.pem -noout -serial
serial=01

Extract info from puppetmaster cert

[root@alpha ~]# openssl x509 -in /var/lib/puppet/ssl/certs/puppetmaster.cluster.ucsf.bkslab.org.pem -text -noout \
-certopt no_subject,no_header,no_version,no_serial,no_signame,no_validity,no_subject,no_issuer,no_pubkey,no_sigdump,no_aux

Create directories to stage new valid periods (I determined puppetmaster.cluster.ucsf.bkslab.org.pem was the .pem file we need because certname in [master] section of /etc/puppet/puppet.conf has this name)

[root@alpha ~]# mkdir /root/puppet_renewal
[root@alpha ~]# cd /root/puppet_renewal
[root@alpha ~]# mkdir /root/puppet_renewal/ca
[root@alpha ~]# mkdir /root/puppet_renewal/puppetmaster
[root@alpha ~]# mkdir /root/puppet_renewal/puppetmaster/private_keys
[root@alpha ~]# mkdir /root/puppet_renewal/puppetmaster/certs

Copy the existing certificate authority's key and the puppetmaster's private key (Only certificates expire. Private keys do not so they can be reused).

[root@alpha ~]# cp /var/lib/puppet/ssl/ca/ca_key.pem /root/puppet_renewal/ca
[root@alpha ~]# cp /var/lib/puppet/ssl/private_keys/mypuppetmaster.cluster.ucsf.bkslab.org.pem /root/puppet_renewal/puppetmaster/private_keys

Create an openssl configuration file

[root@alpha ~]# vi puppet_renewal/renewpuppet.cnf
renewpuppet.cnf
[ v3_ca ]
basicConstraints= CA:TRUE
subjectKeyIdentifier= hash
# authorityKeyIdentifier= keyid:always,issuer:always
keyUsage = critical, cRLSign, keyCertSign
nsComment = 'Puppet Ruby/OpenSSL Internal Certificate' 

[ v3 ]
basicConstraints= CA:FALSE
subjectKeyIdentifier= hash
nsComment = 'Puppet Ruby/OpenSSL Internal Certificate'
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = critical, serverAuth, clientAuth
subjectAltName = @alt_names 

[ alt_names ]
DNS.1 = puppet
DNS.2 = puppet.ucsf.bkslab.org
DNS.3 = puppet.cluster.ucsf.bkslab.org
DNS.4 = puppetmaster
DNS.5 = puppetmaster.ucsf.bkslab.org
DNS.6 = puppetmaster.cluster.ucsf.bkslab.org
DNS.7 = alpha
DNS.8 = alpha.ucsf.bkslab.org
DNS.9 = alpha.cluster.ucsf.bkslab.org
DNS.10 = alpha.compbio.ucsf.edu

Create certificate signing request with existing files

[root@alpha ~]# openssl req -key /root/puppet_renewal/ca/ca_key.pem -new -batch -subj "/CN=Puppet CA: alpha.ucsf.bkslab.org" -out /root/puppet_renewal/ca/ca_new.csr

Create a new CA certificate

[root@alpha ~]#openssl x509 -req -days 3650 -in /root/puppet_renewal/ca/ca_new.csr -signkey /root/puppet_renewal/ca/ca_key.pem \
-out /root/puppet_renewal/ca/ca_crt.pem -extfile /root/puppet_renewal/renewpuppet.cnf -extensions v3_ca -set_serial 1
Signature ok
subject=/CN=Puppet CA: alpha.ucsf.bkslab.org
Getting Private key

Get serial number for your existing CA

[root@alpha puppet_renewal]# echo $((0x`cat /var/lib/puppet/ssl/ca/serial`))
451

Create new certificate signing request with puppet server's key

[root@alpha ~]# openssl req -key /root/puppet_renewal/puppetmaster/private_keys/puppetmaster.cluster.ucsf.bkslab.org.pem -new \
-batch -subj "/CN=alpha.ucsf.bkslab.org" -out /root/puppet_renewal/mypuppetmaster.csr

Create new puppet master's certificate

[root@alpha ~]# openssl x509 -extfile /root/puppet_renewal/renewpuppet.cnf -extensions v3 -req -days 1825 -in /root/puppet_renewal/mypuppetmaster.csr \
-CA /root/puppet_renewal/ca/ca_crt.pem -CAkey /root/puppet_renewal/ca/ca_key.pem -CAcreateserial \
-out /root/puppet_renewal/puppetmaster/certs/puppetmaster.cluster.ucsf.bkslab.org.pem -sha256 -set_serial 451

Replace Puppet's ca_crt.pem, ca.pem, and puppetmaster.pem

 [root@alpha ~]# cp puppet_renewal/ca/ca_crt.pem /var/lib/puppet/ssl/ca/ca_crt.pem
 [root@alpha ~]# puppet_renewal/ca/ca_crt.pem /var/lib/puppet/ssl/certs/ca.pem
 [root@alpha ~]# cp puppet_renewal/puppetmaster/certs/puppetmaster.cluster.ucsf.bkslab.org.pem /var/lib/puppet/ssl/certs/puppetmaster.cluster.ucsf.bkslab.org.pem

Restart httpd

# If this fails, a mistake was made.  Check /var/log/httpd.  
[root@alpha ~]# service httpd start
Starting httpd: [Wed Mar 27 12:28:24 2019] [warn] module passenger_module is already loaded, skipping
                                                          [  OK  ]

Renewing Foreman's certificates (in progress)

Investigate /etc/foreman/ssl/README cluster2 on foreman

Follow the commands in the README file while in the /etc/foreman/ssl directory. Only do this step after you've done the Puppet certificates in the previous section. The foreman related commands here rely on the ca.pem generated by the previous puppet commands.

#openssl genrsa -aes128 -out foreman.key 2048
openssl genrsa -out foreman.key 2048
openssl req -new -out foreman.csr -key foreman.key -config foreman.cnf 
openssl req -text -in foreman.csr -noout
openssl x509 -req -in foreman.csr -CA /var/lib/puppet/ssl/ca/ca_crt.pem -CAkey /var/lib/puppet/ssl/ca/ca_key.pem -CAserial /var/lib/puppet/ssl/ca/serial -out foreman.crt -days 1730 

openssl rsa -in foreman.key -out foreman.key-unlocked
# this section doesn't need to be done if they symlinks already exist 
ln -s foreman.key-unlocked key.pem
ln -s foreman.crt cert.pem

Restart httpd when this is done.

Clearing old certificates

When both the Puppet & Foreman certificates are renewed, you will have an entire cluster of machines that have certificates that reference the old ca.pem. You have to regenerate them all. Follow the instructions on: http://wiki.docking.org/index.php/PuppetTricks in the Regenerating a Certificate page. Track which ones you do as it'd be annoying to do any machine more than once. Make sure to restart sssd after a puppet client run.

Libvirt, Hypervisors, and TLS

alpha/puppetmaster is used to manage the VMs. To add to the hassle of renewing Puppet certificates, we also have to copy the renewed certificates to the libvirt-relevant sections on both the puppetmaster AND the hypervisors. Otherwise, foreman cannot interact with the hypervisors/VMs.

Only do this once on the puppetmaster (hostname will depend on which name you used while regenerating SSL certs. I used alpha.ucsf.bkslab.org.pem):

 copy /var/lib/puppet/ssl/certs/<hostname of puppetmaster>.pem to /etc/pki/libvirt/clientcert.pem 
 copy /var/lib/puppet/ssl/private_keys/alpha.ucsf.bkslab.org.pem to /etc/pki/libvirt/private/clientkey.pem

We have several hypervisors, so this may need to be repeated on each one. Do this on the hypervisors:

 copy /var/lib/puppet/ssl/certs/ca.pem to /etc/pki/CA/cacert.pem
 copy /var/lib/puppet/ssl/certs/<hostname>.pem to /etc/pki/libvirt/servercert.pem
 copy /var/lib/puppet/ssl/private_keys/<hostname>.pem to /etc/pki/libvirt/private/serverkey.pem
 service libvirtd restart (if you did anything wrong, this will fail.  Be careful)

Possible Other Errors

I was getting this when trying to remove a host: "Remove Reverse DNS record for n-5-35.cluster.ucsf.bkslab.org task failed with the following error: ERF12-1261 [ProxyAPI::ProxyException]: Unable to delete DNS entry ([OpenSSL::SSL::SSLError]: SSL_connect returned=1 errno=0 state=SSLv3 read server session ticket A: sslv3 alert ce...) for proxy https://foreman.cluster.ucsf.bkslab.org:8443/dns"

This was a foreman SSL error. This happened because foreman was still using the old SSL certs that existed in /var/lib/puppet/ssl/certs. It was referencing a certificate file called alpha.cluster.ucsf.bkslab.org. The certificate I renewed was called alpha.ucsf.bkslab.org. I had to go to foreman.ucsf.bkslab.org then click Administer -> Settings -> Auth. I had to adjust the lines for ssl_certificate and ssl_priv_key to point to the new certificates I made. Then, foreman-proxy was working again.

Further Reading

A big thanks to these two blogs for pointing me in the right direction:

Sean the Sysadmin: http://www.scrosby.com/2017/06/renewing-puppet-ca-and-puppet-master.html
Flying Circus: https://blog.flyingcircus.io/2017/09/01/how-to-renew-puppet-ca-and-server-certificates-in-place/