Troubleshooting - Puppet Failed to generate additional resources using 'eval generate: SSL connect returned=1'

Error Details

This error occurs after initiating a puppet agent run with:

[root@aleph2 /]# puppet agent --test 
info: Retrieving plugin
err: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [CRL is not yet valid for /CN=puppetmaster.cluster.ucsf.bkslab.org]
err: /File[/var/lib/puppet/lib]: Could not evaluate: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [CRL is not yet valid for /CN=puppetmaster.cluster.ucsf.bkslab.org] Could not retrieve file metadata for puppet://puppetmaster.cluster.ucsf.bkslab.org/plugins: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [CRL is not yet valid for /CN=puppetmaster.cluster.ucsf.bkslab.org]
err: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [CRL is not yet valid for /CN=puppetmaster.cluster.ucsf.bkslab.org]
warning: Not using cache on failed catalog
err: Could not retrieve catalog; skipping run
err: Could not send report: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [CRL is not yet valid for /CN=puppetmaster.cluster.ucsf.bkslab.org]

Causes: The key hint in the error messages is that "CRL is not yet valid for /CN=puppetmaster.cluster.ucsf.bkslab.org". This means that the time between the puppet agent and the puppetmaster is too different; they are out of sync. To address this, we need to ensure that time is the same.

Solution: The key resolution is to ensure that time synced up on both puppet master and puppet agent. Make sure ntp is turned on on both hosts and that the ntpd server they point to is working fine. If they do not have access to NTP, set the time manually on the puppet agent to a time that is similar to the time/date on the puppet master.

Example: I did this example on aleph2. Commands also shown from alpha (puppetmaster) to show the time drift

# Today is June 6, 2017, around 10:00 AM
[root@aleph2 network-scripts]# date
Wed May 31 19:27:21 PDT 2017
# That's not the right time at all!

# Let's look at alpha
-bash-4.1$ hostname
alpha.cluster.ucsf.bkslab.org
-bash-4.1$ date
Tue Jun  6 10:04:45 PDT 2017

# The time has to be corrected on aleph2
[root@aleph2 network-scripts]# service ntpd status
ntpd (pid  27743) is running...
# NTPD is working so what gives? I checked ntp.conf and saw that it was pointing to a public time server but I had no public network yet!  I had to resort to setting time manually.  
[root@aleph2 network-scripts]# date -s '2017-06-06 09:58'
Tue Jun  6 09:58:00 PDT 2017
# And now puppet works!
[root@aleph2 network-scripts]# puppet agent --test
notice: Run of Puppet configuration client already in progress; skipping