Checking cluster health – OpenStack
We are able to measure the health of our cluster by using the swift-dispersion-report tool. This is done by checking the set of our distributed containers, to ensure that the objects are in their proper places within the cluster. This tool checks if a set of deliberately distributed containers and objects are currently in their proper places within the cluster. For instance, a common deployment has three replicas of each object. The health of that object can be measured by checking if each replica is in its proper place.
Log in to the OpenStack Object Storage Proxy Server. To log on to our OpenStack Object Storage Proxy host, that was created using Vagrant, issue the following command:
vagrant ssh swift.
How to achieve it…
Carry out the following steps to set up the swift-dispersion tools to report on cluster health:
- We first create the configuration file (/etc/swift/dispersion.conf) required by the swift-dispersion tools, as follows:
auth_url = http://172.16.0.200:5000/auth/v2.0
auth_user = cookbook:admin
auth_key = openstack
- Now we need to create containers and objects throughout our cluster, so that they are in distinct places, by using the swift-dispersion-populate. We do this as follows:
- Once these containers and objects have been set up, we can then run swift- dispersion-report, as follows:
This produces the following result:
- We then set up a cron job that repeatedly checks the health of these containers and objects. We do this as follows:
echo "/usr/bin/swift-dispersion-report" | sudo tee -a /etc/cron.hourly/swift-dispersion-report
How it works…
The health of objects can be measured by checking whether the replicas are correct. If our OpenStack Object Storage cluster replicates an object 3 times and 2 of the 3 are in the correct place, the object would be 66.66% healthy.
To ensure we have enough replicated objects in our cluster, we populate it with the swift- dispersion-populate tool, which creates 2,621 containers and objects, thereby increasing our cluster size. Once in place, we can then set up a cron job that will run hourly to ensure our cluster is consistent and therefore giving a good indication that our cluster is healthy.
By setting up a cron job on our proxy node (which has access to all our nodes), we can constantly measure the length of our entire cluster. In our example, the cron job runs hourly, executing the swift-depression-report tool.