Monitoring Compute services with Munin – OpenStack

Munin the monitoring tool surveys all your computers and remembers what it saw.

Munin Monitoring System is one of the popular computer systems, network, and infrastructure monitoring tools. With Munin, real-time information about the servers, applications, services, SANs, network, and many more resources is available in graphs through a web interface. And Munin is easy to set up and deploy with its plug-and-play capabilities, and coupled with the fact that it’s free and open-source, making it a popular monitoring tool for web servers and hosts.

Munin actually consists of two parts, namely Munin Master and Munin node. Munin Server is the data gatherer and grapher which also hosts the software website. The Munin Server will periodically poll all the nodes in your network, it’s aware of for data, which is, in turn, will use to create graphs and HTML pages for viewing via a web browser. Munin Node is the agent that monitors and extracts data from the node it runs on and then passes the data to the server for further processing.

Monitoring Compute services and nodes with Munin Configuration

We will be configuring Munin on a server that has access to the OpenStack Compute environment hosts. Ensure this server has enough RAM, disk, and CPU capacity for the environment you are running. As a bare minimum in a test environment, it is possible to run this on a VM with 1vCPU, 1.5 GB of RAM, and 8 GB of disk space.

To set up Munin with OpenStack, carry out the following steps:

• Install Munin.
• Configure the Munin nodes.
• Configure OpenStack plugins for Munin.

The Munin master is responsible for gathering data from Munin nodes. It stores this data in RRD, files, and graphs them on request. It also checks whether the fetched values fell below or go over specific thresholds (warning, critical) and will send alerts if this happens and the administrator configured it to do so.
It must be installed first, as follows:

• Configure a server with Ubuntu 12.04, 64 bit Version with access to the servers in our OpenStack environment.
• Install Munin from the Ubuntu repositories:

Related Article: OpenStack Tutorial

 

sudo apt-get update
sudo apt-get install apache2
sudo apt-get install munin munin-plugins-extra sudo service apache2 restart

• By default, the Apache configuration for Munin only allows access from 127.0.0.1. To allow access from our network, we edit /etc/apache2/conf.d/munin and allow the server(s) or network(s) that can access Munin. For example, to allow access from 172.16.0.0/16, we add the following access line in:

Allow from 172.16.

• We reload the Apache service to pick up this change. We do this as follows:

sudo service apache2 reload

• At this stage, we have a basic installation of Munin that is gathering statistics for the running machine where we have just installed Munin. This can be seen if you load up a web browser and browse to https://server/munin:

• Configuration of Munin Master is done in the /etc/munin/munin.conf. Here, we tell Munin where our OpenStack hosts, which are specified as FQDNs, are. Munin groups these hosts under the same domain. For example, to add in two OpenStack hosts that have addresses 172.16.0.200 (openstack1) and 172.16.0.201 (openstack2), we add the following section into the /etc/munin/munin.conf file:

[controller.cloud.test] address 172.16.0.200 use_node_name yes
[openstack2.cloud.test] address 172.16.0.201 use_node_name yes

We can now proceed to configure the nodes that we want to monitor, for example openstack1 and openstack2.

Munin Nodes

• A munin node is a data collector for munin that runs on a slice.
  With the Munin master server, we can now configure the Munin nodes. These have an agent on them, called munin-node  that the master uses to gather the information and present to the user.
• We first need to install the munin-node package on our OpenStack hosts. So, for each one we execute the following:

sudo apt-get update
sudo apt-get -y install munin-node munin-plugins-extra

• Once installed, we need to configure this so that our Munin master host is allowed to get information from the node. To do this, we edit the /etc/munin/munin-node.conf file and add in an allow line. To allow our master on IP address 172.16.0.253, we add the following entry:

allow ^172.16.0.253$

• Once that the line is in, we can restart the munin-node service to pick up the change:

sudo restart munin-node

 

With Munin master installed, and having a couple of nodes with graphs showing up on the Master, we can add the plugins to pick up the OpenStack services and graph them. To do this, we check out some plugins from GitHub.

• We first ensure that we have the git client available to us on our OpenStack nodes:

sudo apt-get update
sudo apt-get -y install git

• We can now check out the OpenStack plugins for Munin as they’re not yet available in the munin-plugins-extra package:

git clone https://github.com/munin-monitoring/contrib.git

• This checks out contributed code and plugins to a directory named contrib. We copy the relevant plugins for the OpenStack services into the Munin plugins directory, as follows:

cd contrib/plugins
sudo cp nova/* /usr/share/munin/plugins/ 
sudo cp keystone/* /usr/share/munin/plugins 
sudo cp glance/* /usr/share/munin/plugins

• Munin-node comes with a utility that allows us to enable appropriate plugins on our hosts, automatically. We run the following commands to do this:

sudo munin-node-configure --suggest 
sudo -i # get root shell
munin-node-configure --shell 2>&1 | egrep -v "^#" | sh

• The Keystone and Glance plugins don’t get picked up automatically, so we add these to the plugins’ directory, manually, with symlinks:

cd /etc/munin/plugins
sudo ln -s /usr/share/munin/plugins/keystone_stats 
sudo ln -s /usr/share/munin/plugins/glance_size sudo 
ln -s /usr/share/munin/plugins/glance_status

 

Related Article: Installing Openstack Dashboard

• We also need to add an extra configuration file to sit alongside the OpenStack plugins called /etc/munin/plugin-conf.d/openstack:

[nova_*] user nova
[keystone_*] user keystone
[glance_*] user glance

• With the appropriate plugins configured, we restart the munin-node service, as follows, to pick up the change:

sudo restart munin-node

• When the master server refreshes, we see OpenStack services as options and graphs, which we can click on:

Related Article: OpenStack Interview Questions

Compute Services with Munin

Munin is an excellent, open-source networked, resource-monitoring tool that can help analyze resource trends and identify problems with our OpenStack environment. Configuration is very straightforward, without of the box configuration providing lots of very useful graphs from RRD (Round Robin Database) files. By adding in a few extra configuration options and plugins, we can extend this to monitoring our OpenStack environment.
Once Munin has been installed, we have to do a few things to configure it to produce graphed statistics for our environment:

• Configure the master Munin server with the nodes we wish to get graphs from. This is done in the /etc/munin/munin.conf file by using the tree-like structure domain/host address sections.
• We then configure each node with the munin-node. This is a service that has its own configuration file where we have to explicitly set what Munin server that can pull graphs from it. This is set in with the allow line in the /etc/munin/munin.conf file.
• Finally, we configure appropriate plugins for the services that we want to monitor. With the OpenStack plugins installed, we can monitor the Compute, Keystone, and Glance services and obtain statistics on the number of instances running, the number of floating IPs assigned, allocated, and used, and so on.