User Acceptance Testing - Puppet

Acceptance Testing

Acceptance testing for Puppet is done using Beaker. This product was developed in-house at Puppet labs and grew out of the old rspec-system tool. Documentation for Beaker is a little patchy but getting rapidly better.

Beaker can use loads of different cloud and virtualization infrastructures to create SUTs (System Under Test).

Puppet Acceptance Testing With Beaker

Test-driven development can bring a number of benefits. An approach where you write tests first makes you think about what you need to develop. Subsequent refactoring generally leads to designs that emerge in the code with high levels of test coverage.

Acceptance Testing with Beaker

Beaker can manage the temporary creation of a SUT via a hypervisor. The Puppet module under test is installed and then the tests are executed.

Install Beaker

Beaker can be installed via a Gemfile

group :development, :test do
...
  gem 'beaker', :require => false
  gem 'beaker-rspec', :require => false
...
end

Use bundler to install the gems

gem install bundler
bundle install

Beaker Configuration

Typically Beaker acceptance tests are held in a spec/acceptance subdirectory.

A spec/acceptance/node sets subdirectory defines the machines that we want to test against. Node sets are yaml files that define the machine’s name, location, and platform. The hypervisor type is also specified. Examples for Vagrant can be found here.

module
  spec      
      acceptance
          _spec.rb
          nodesets
              default.yml
              ubuntu-server-1404-x64.yml
              ...
      defines
      fixtures
      spec_helper.rb
      spec_helper_acceptance.rb
  tests

The spec_helper_acceptance.rb file provides the basic configuration. The module under test is copied onto each host. UNSUPPORTED_PLATFORMS provides an easy way to de-scope tests for a particular platform.

require 'beaker-rspec/spec_helper'
require 'beaker-rspec/helpers/serverspec'

...
hosts.each do |host|
  install_puppet
end
...

UNSUPPORTED_PLATFORMS = ['windows','AIX','Solaris']
RSpec.configure do |c|
  proj_root = File.expand_path(File.join(File.dirname(__FILE__), '..'))
  c.formatter = :documentation
  c.before :suite do
    hosts.each do |host|
      copy_module_to(host, :source => proj_root, :module_name => 'module')
      
      ...
      
      on host, puppet('module','install','puppetlabs-stdlib'), { :acceptable_exit_codes => [0,1] }
      on host, puppet('module','install','puppetlabs-concat'), { :acceptable_exit_codes => [0,1] }
    end
  end

Acceptance Tests

The acceptance tests describe the expected behavior and state of the SUT and are written in RSpec. In this example, a test context is defined for user (and group) creation.

require ‘spec_helper_acceptance'

describe 'module defintion', :unless => UNSUPPORTED_PLATFORMS.include?(fact('osfamily')) do

  context 'user is created with a valid input parameter’ do
    it 'should work with without errors’ do
      pp = <<-EOS
        $theuser = {
        'auser' => { 'shell' => '/bin/bash' }
      }
      module { 'auser':
        user => $theuser
      }
      EOS
      apply_manifest(pp, :catch_failures => true)
      expect(apply_manifest(pp, :catch_failures => true).exit_code).to be_zero
   end
   
   describe user('auser') do
    it { is_expected.to exist }
    it { is_expected.to belong_to_group 'auser' }
   end
   
   describe group('auser') do
      it { is_expected.to exist }
   end
    
 end
end

This test context applies the manifest, expects it to execute without errors and then validates that a user is created in the correct group.

Tests can be run with

bundle exec rspec spec/acceptance

Test Execution

Beaker will initially use the specified hypervisor to bring up a SUT.

Hypervisor for ubuntu-server-1404-x64 is vagrant
Beaker::Hypervisor, found some vagrant boxes to create
created Vagrantfile for VagrantHost ubuntu-server-1404-x64
...

Puppet will then be installed and the acceptance tests within spec/acceptance will be performed.

Generally, each test applies manifests within the module for a particular context.

ubuntu-server-1404-x64 $  mktemp -t apply_manifest.pp.XXXXXX  
/tmp/apply_manifest.pp.ae1dqd

ubuntu-server-1404-x64 executed in 0.01 seconds
localhost $ scp /var/folders/b4/whdj5tqn1_n_2783hl5cnvch0001bn/T/beaker20141123-1076-b25en7 
ubuntu-server-1404-x64:/tmp/apply_manifest.pp.ae1dqd {:ignore => }

ubuntu-server-1404-x64 $  env PATH="/usr/bin:/opt/puppet-git-repos/hiera/bin:${PATH}" 
RUBYLIB="/opt/puppet-git-repos/hiera/lib:/opt/puppet-git-repos/hiera-puppet/lib:${RUBYLIB}" 
puppet apply --verbose --detailed-exitcodes /tmp/apply_manifest.pp.ae1dqd  

[...Puppet Logging...] ubuntu-server-1404-x64 executed in 6.22 seconds Exited: 2

The tests are then performed. Beaker resolves them to an underlying operating system command in order to validate the assertion

User "auser"

ubuntu-server-1404-x64 $  id auser  
uid=1001(auser) gid=1001(auser) groups=1001(auser)

ubuntu-server-1404-x64 executed in 0.01 seconds
      should exist []
      
ubuntu-server-1404-x64 $  id auser | awk '{print $3}' | grep -- auser  
groups=1001(auser)

ubuntu-server-1404-x64 executed in 0.01 seconds
      should belong to group "auser"
    Group "auser"

ubuntu-server-1404-x64 $  getent group auser  
auser:x:1001:

ubuntu-server-1404-x64 executed in 0.01 seconds
      should exist []   

        The hypervisor will then shutdown and destroy the SUT. Beaker reports on the results when all the tests have been completed.

Destroying vagrant boxes

==> ubuntu-server-1404-x64: Forcing shutdown of VM...
==> ubuntu-server-1404-x64: Destroying VM and associated drives...

Finished in 3 minutes 8.1 seconds
20 examples, 0 failures