PuppetTricks

This page is a collection of tricks and tips for using Puppet to administer systems.

The names master, puppetmaster, and foreman all refer to (at the time of writing this) alpha. The name client refers to any machine that is maintained by puppet.

Creating a new Puppet Module

Puppet code consists of Puppet modules. Modules are self contained bundles of code and data that exist to manage a particular technology. Modules consist of classes which consist of resource types.
Our puppet modules exist in two places: /opt/bks/src/cluster/puppet/modules & /etc/puppet/modules
We write new Puppet module configs in the /opt directory but when we build the packages, they go to /etc/puppet/modules. Any changes to Puppet code should occur in the /opt directory then built later.

To generate a new Puppet module, we first go to /opt/bks/src/cluster/puppet/modules. Decide on what you want to call it. Notice in this directory, everything is called bks-<module name>. This is because Puppet wants you to put the author's name first prior to the module name. You should name your module the same way with "bks-" in front of the module name.

Here's an example of me creating a new module: bks-selinux

1) Generate a new Puppet module. The generate command produces a basic skeleton of a typical Puppet module structure. We'll need to edit it to our liking.

[root@alpha ~]# cd /opt/bks/src/cluster/puppet/modules/
[root@alpha modules]# puppet module generate bks-selinux
Generating module at /share/utils/bks/src/cluster/puppet/modules/bks-selinux
bks-selinux
bks-selinux/spec
bks-selinux/spec/spec_helper.rb
bks-selinux/manifests
bks-selinux/manifests/init.pp
bks-selinux/tests
bks-selinux/tests/init.pp
bks-selinux/Modulefile
bks-selinux/README

2) We want to copy a SELinux configuration file to the destination nodes. I made this configuration file, sgehald.pp, and it must be copied to other nodes. First, it should be copied to the module's directory. I make a directory in the bks-selinux directory named files then copy the SELinux config file there.

[root@alpha bks-selinux]# mkdir files
[root@alpha bks-selinux]# cp /tmp/sgehald.pp files

3) Edit the module's manifests/init.pp with desired configuration information. Here I am telling Puppet to copy a SELinux .pp file from the Puppetmaster to destination nodes with proper ownership and permissions. Afterwards, I tell Puppet to ensure that the sgehald selmodule is loaded. The require line means that the selmodule command will not run unless the sgehald.pp file exists on the destination machine already at location /usr/share/selinux/targeted/sgehald.pp. Note that puppet:///selinux/sgehald.pp refers to sgehald.pp which exists inside the file directory of puppet module, bks-selinux,.

class selinux {
 file {'/usr/share/selinux/targeted/sgehald.pp':
   ensure => present,
   owner  => 'root',
   group  => 'root',
   mode   => 644,
   source => 'puppet:///selinux/sgehald.pp',
 }
 selmodule {'sgehald':
   ensure      => present,
   syncversion => true,
   require     => File ['/usr/share/selinux/targeted/sgehald.pp'],
 }
}

4) After we're finished editing the init.pp file, we should verify the contents are syntactically correct. Use puppet parser validate against the init.pp file to verify this. If you get no output, Puppet thinks your init.pp file is fine

[root@alpha manifests]# puppet parser validate init.pp

5) Now that init.pp file is fine, it's time to build the module. While in the module directory, issue the following command to create a new Puppet package:

[root@alpha bks-selinux]# puppet module build . 
Building /share/utils/bks/src/cluster/puppet/modules/bks-selinux for release
pkg/bks-selinux-0.0.1.tar.gz

6) The previous command created a package but it is not yet usable in production. To make it usable, do a puppet module install commmand with the -f flag. (why do we use an -f flag? I tried the same command without and with the -f flag and you should see. It looks for this module on forge.puppetlabs.com first but since we are building locally, we force the build)

[root@alpha bks-selinux]# puppet module install pkg/bks-selinux-0.0.1.tar.gz 
Preparing to install into /etc/puppet/modules ...
Downloading from http://forge.puppetlabs.com ...
Error: Could not find release information for this module (bks/selinux) (HTTP 410)
Error: Try 'puppet help module install' for usage
[root@alpha bks-selinux]# puppet module install -f pkg/bks-selinux-0.0.1.tar.gz 
Preparing to install into /etc/puppet/modules ...
Installing -- do not interrupt ...
/etc/puppet/modules
└── bks-selinux (v0.0.1)

7) After running a puppet module install, check /etc/puppet/modules. There should be an selinux module directory now.

[root@alpha manifests]# ls -lhd /etc/puppet/modules/selinux/
drwxr-xr-x. 6 root root 4.0K Jul  1 13:50 /etc/puppet/modules/selinux/

8) To put this Puppet module into production, we must go to foreman.ucsf.bkslab.org. Then, Configure -> Puppet Classes -> Import from puppetmaster.cluster.ucsf.bkslab.org. Select both Production and Development environments and say yes. From there, you have to figure which nodes need this particular Puppet module. Since this SELinux fix existed to avoid GPU job failures, I activated the Puppet Module on cluster/noautogpu host group. Then, all nodes now have the proper SELinux permission to run GPU jobs!

Regenerating a Certificate

On Client

   $ sudo service puppet stop
   $ sudo mv /var/lib/puppet/ssl /var/lib/puppet/ssl~
   $ puppet agent --no-daemonize --onetime --verbose --waitforcert=60

On Server (within 60 seconds)

   $ sudo puppet cert clean <client hostname>
   $ sudo puppet cert sign <client hostname>
      OR if you wish to allow DNS aliases
   $ sudo puppet cert --allow-dns-alt-names sign <client hostname>

Note this can also be done through Foreman by going to the Infrastructure -> Smart Proxies -> Puppetmaster -> Certificates page

On Client

The previous run should finish without errors (errors are in purple). It should then be possible to run `sudo puppet agent -t` without any waiting or errors.

changing mounts, dbraw

cd /opt/bks/src/cluster/puppet/modules/bks-nfs_mounts/files/dbraw

or edit

manifests/init.pp

Then from nfs_mounts directory

• 1) Bump version in Modulefile
• 2) puppet module build .
• 3) puppet module install -f pkg/bks-nfs_mounts-0.0.X.tar.gz
• 4) proceed to standard goat sacrifice procedure

Adding aliases for server on Alpha

As root on Alpha

$ cd /var/named/include
# Save a copy before edit db.aliases.inc 
$ vim db.aliases.inc
$ cd /var/named/dynamic
# Save a copy before edit db.cluster.ucsf.bkslab.org
$ vim db.cluster.ucsf.bkslab.org

# Restart named service
$ service named restart

# Test in another machine
$ ping <alias> // should return the machine you set alias for

Adding new mount point to Puppet

$ cd /opt/bks/src/cluster/puppet/modules/bks-nfs_mounts/

1. Save a copy and edit manifests/init.pp
$ vim manifests/init.pp
2. Validate and check syntax
$ puppet parser validate manifests/init.pp
3. Create and install a new module package
 a. Bump up version in Modulefile
 b. Create a new module package
    $ puppet module build .
 c. Install the new module package
    $ puppet module install -f pkg/bks-nfs_mounts-<version>.tar.gz

Mount new disk to machine

Run it manually

$ sudo puppet agent --test