OpenStack – Monitoring the storage service using StatsD/Graphite

When monitoring the OPENSTACK STORAGE SERVICE, Swift, we are looking at gathering key metrics from within the storage cluster in order to make decisions on its health. For this, we can use a small piece of middleware named swift-informant, together with StatsD and Graphite, to produce near real-time stats of our cluster.

Configuring Storage Device using StatsD/ Graphite and Installation

We will be configuring StatsD and Graphite on a server that has access to the OpenStack Storage proxy server. Ensure that, this server has enough RAM (at least 1 GB), disk (at least 10 GB), and CPU (1 CPU for small test environments) capacity for the environment you are running.

To install StatsD and Graphite, carry out the following steps:

Prerequisites

For this, we will be configuring a new Ubuntu 12.04 server. Once Ubuntu has been installed, we need to install some prerequisite packages.

apt-get -y install git python-pip gcc python2.7-dev apache2 libapache2-mod-python python-cairo python-django libapache2-mod-wsgi python-django-tagging

Graphite

• Installation of Graphite is achieved using the Python Package Index tool, pip:

sudo pip install carbon sudo pip install whisper
sudo pip install graphite-web

• Once installed, we can configure the installation. Example of configuration files for Graphite are found at /opt/graphite/conf. We rename these to their respective conf files:

cd /opt/graphite/conf
sudo mv carbon.conf.example carbon.conf
sudo mv storage-schemas.conf.example storage-schemas.conf

• We now create the vhost file for Apache that will load the Graphite frontend. Create /etc/apache2/sites-available/graphite with the following contents:

ServerName 172.16.0.253
DocumentRoot "/opt/graphite/webapp"
ErrorLog /opt/graphite/storage/log/webapp/error.log CustomLog /opt/graphite/storage/log/webapp/access.log common

• I’ve found that an equal number of processes & threads tends to show the best performance for Graphite (ymmv). WSGIDaemonProcess graphite processes=5 threads=5 display-

name='%{GROUP}' inactivity-timeout=120 WSGIProcessGroup graphite WSGIApplicationGroup %{GLOBAL}
WSGIImportScript /opt/graphite/conf/graphite.wsgi process-group=graphite application-group=%{GLOBAL}
WSGIScriptAlias / /opt/graphite/conf/graphite.wsgi
Alias /content/ /opt/graphite/webapp/content/  SetHandler None

 

[Related Article: TroubleShooting OpenStack Object Storage services and Dashboard]

• The graphite.wsgi file has to be accessible by apache. It won’t be visible to clients because of the DocumentRoot though. 

Order deny,allow Allow from all

• We enable this website using the a2ensite utility:

sudo a2ensite graphite

• We now need to enable the WSGI file for Graphite:

sudo mv graphite.wsgi.example graphite.wsgi

• Various areas need to change their ownership to that of the process running the Apache web server:

sudo chown -R www-data:www-data /opt/graphite/storage/log/ sudo touch /opt/graphite/storage/index
sudo chown www-data:www-data /opt/graphite/storage/index

• We can now restart Apache to pick up these changes:

sudo service apache2 restart

• The Graphite service runs with a SQLite database backend, so we need to initialize this.

[Related Article: Setting up a Chef Environment For OpenStack]

cd /opt/graphite/webapp/graphite sudo python manage.py syncdb

• This will ask for some information, as displayed next:

You just installed Django's auth system, which means you don't have any superusers defined.
Would you like to create one now? (yes/no): yes Username (Leave blank to use 'root'):
E-mail address: user@somedomain.com Password:
Password (again):
Superuser created successfully. Installing custom SQL … Installing indexes …
No fixtures found.

• We also need to ensure that Apache can write to this too:

sudo chown -R www-data:www-data /opt/graphite/storage

• Finally, we start the services, thus:

cd /opt/graphite
sudo bin/carbon-cache.py start

StatsD

• StatsD runs using js, so we have to install it first, using packages from Ubuntu’s repositories:

sudo apt-get update
sudo apt-get -y install nodejs

• We then check out the StatsD code from Git:

git clone https://github.com/etsy/statsd.git

• Configuring StatsD is done by modifying an example configuration file:

cd statsd
cp exampleConfig.js Config.js

• We need to modify the js file to change the graphiteHost: parameter to localhost, as we’re running Graphite on the same host as StatsD:

{
graphitePort: 2003
graphiteHost: "localhost"
port: 8125
}

• To start the service, we issue the following command:

nohup node stats.js Config.js &

[Related Article: Collecting Usage Statistics - OpenStack Cloud]

swift-informant

We are now ready to configure the OpenStack Swift proxy server to include the swift-informant middleware in the pipeline. This is done with configuring the /etc/swift/proxy-server.conf file.

• We first download and install the middleware by running the following commands:

git clone https://github.com/pandemicsyn/swift-informant.git cd swift-informant
sudo python setup.py install

• Once installed, we modify the pipeline in /etc/swift/proxy-server.conf to specify a filter named informant:

[pipeline:main]
pipeline = informant healthcheck cache swift3 s3token tokenauth keystone proxy-server

• We then add in the informant filter section, specifying the address of our StatsD server, in the statsd_host section, as follows:

[filter:informant]
use = egg:informant#informant statsd_host = 172.16.0.9
statsd_port = 8125
standard statsd sample rate 0.0 <= 1
statsd_sample_rate = 0.5
list of allowed methods, all others will generate a "BAD_METHOD" event
valid_http_methods = GET,HEAD,POST,PUT,DELETE,COPY
send multiple statsd events per packet as supported by statsdpy
combined_events = no
prepends name to metric collection output for easier recognition, e.g. company.swift.
metric_name_prepend =

• Once done, we simply restart our OpenStack proxy service:

sudo swift-init proxy-server restart

• Load up your web browser and point it to your Graphite web installation, to see the graphs get populated in real time.

 

Gaining Insights through StatsD/Graphite

Gaining insight into what our OpenStack Storage cluster is doing can be achieved by including a piece of middleware in the pipeline of our OpenStack Storage proxy server named swift- informant, along with StatsD and Graphite. StatsD is a node.js service that listens for statistics sent to it in UDP packets. Graphite takes this data and gives us a real-time graph view of our running services.
Installation and configuration is done in stages. We first install and configure a server that will be used for StatsD and Graphite. Graphite can be installed using Python’s Package Index (using the pip tool), and for this, we install three pieces of software: carbon (the collector), whisper (fixed-size RRD service), and the Django Web Interface, graphite-web. Using the pip tool installs these services to the /opt directory of our server.

• Once the server for running Graphite and StatsD has been set up, we can configure the OpenStack Storage proxy service, so that statistics are then sent to the Graphite and StatsD server. With the appropriate configuration in place, the OpenStack Storage service will happily send events, via UDP, to the StatsD service.
  Configuration of the Graphite interface is done in an Apache vhost file that we place in Ubuntu’s Apache’s sites-available installation.
• Note that vhost needs to be configured appropriately for our environment—specifically the path to the DJANGO_ROOT area—as part of our Python installation. For Ubuntu 12.04, this is

        /usr/lib/python2.7/dist-packages/django. Alias /media/ "/usr/lib/python2.7/dist-packages/django/contrib/admin/media/"

We then ensure that the Graphite WSGI (Web Service Gateway Interface) file is in place at the appropriate path, as specified by the WSGIScriptAlias directive at /opt/graphite/conf/graphite.wsgi.

• Once in place, we ensure that our filesystem has the appropriate permissions to allow Graphite to write various logs and information as it’s running.
• When this has been done, we simply restart Apache to pick up the changes.
• With the Graphite web interface configured, we initialize the database; for this installation, we will make use of a SQLite database resource. This is achieved by running the syncdb option in the Graphite manage.py script in the /opt/graphite/webapp/graphite directory. This asks us to create a superuser called user for the system, to manage it later.
• Once this has been done, we can start the collector service, carbon, which starts the appropriate services that will listen for data being sent to it.
• With all that in place, we simply move our efforts to the OpenStack Storage proxy service, where we check out the     swift-informant pipeline of our proxy service.