Troubleshooting OpenStack Object Storage services and Dashboard

Troubleshooting OpenStack Object Storage services

OPENSTACK STORAGE SERVICE (SWIFT) is built for HIGHLY AVAILABLE STORAGES, but there may be times when something might go wrong, from authentication issues to failing hardware.

How to OpenStack Object Storage services

Carry out the following steps when encountering the problems presented.

Authentication issues

Authentication issues in Swift occur when a user or a system has been configured with the wrong credentials. A Swift system that has been supported by OpenStack Authentication service (Keystone) will require performing authentication steps against Keystone manually as well as viewing logs during the transactions. Check the Keystone logs for evidence of user authentication issues for Swift.

[Related Article:  OpenStack Authentication]

The user will see the following message with authentication issues:

If Swift is working correctly, but Keystone isn’t, skip to the Troubleshooting OpenStack Authentication recipe.

Swift can add complexity to authentication issues when ACLs have been applied to containers. For example, a user might not have been placed in an appropriate group that is allowed to perform that function on that container. To view a container’s ACL, issue the following command on a client that has the Swift tool installed:

swift -V 2.0 -A https://keystone_server:5000/v2.0 -U tenant:user -K password stat container

The Read ACL: and Write ACL: information will show which roles are allowed to perform those actions.

To check a user’s role, run the following set of commands on the Keystone server:

Administrator Credentialsexport OS_USERNAME=adminexport OS_PASSWORD=openstack 
export OS_AUTH_URL=https://172.16.0.200:5000/v2.0 export OS_TENANT_NAME=cookbook 
Get User ID
keystone user-list
# Get Tenant ID keystone tenant-list
# Use the user-id and tenant-id to get the roles for # that user in that tenant
keystone -I admin -K openstack -N https://172.16.0.200:5000/v2.0/ -T cookbook role-list --user user-id --tenant tenant-id

Now compare with the ACL roles assigned to the container.

Handling drive failure

When a drive fails in an OpenStack Storage environment, you must first ensure the drive is unmounted so that Swift isn’t attempting to write data to it. Replace the drive and rebalance the rings. This is covered in more detail in the Detecting and Replacing Failed Hard Drives recipe in ADMINISTERING OPENSTACK STORAGE.

Handling server failure and reboots

The OpenStack Storage service is very resilient. If a server is out of action for a couple of hours, Swift can happily work around this server being missing from the ring. Any longer than a couple of hours, though, and the server will need removing from the ring. To do this, follow the steps mentioned in the REMOVING NODES FROM A CLUSTER recipe in  ADMINISTERING OPENSTACK STORAGE.

How OpenStack Storage service work

The OpenStack Storage service, Swift, is a robust object storage environment, and as such, handles a relatively large number of failures within this environment. Troubleshooting Swift involves running client tests, viewing logs, and in the event of failure, identifying what the best course of action is.

[Related Article: OpenStack Network]

Troubleshooting OpenStack Dashboard

The OPENSTACK DASHBOARD, Horizon, provides the web UI that your end-users will use to consume your OpenStack environment, so keeping it always in the running is critical. There are a few instances, however, where Horizon may decide to go away.

How to achieve OpenStack Dashboard

When the Horizon goes awry, you can check the following.

[Related Article: Monitoring MySQL with Hyperic]

Unable to log into the OpenStack Dashboard

If you find you are unable to log into Horizon, check that you have a valid user/password.
To do this, log into a node that has the python-keystone  authenticate with the same user:

export OS_TENANT_NAME=cookbook export OS_USERNAME=admin export OS_PASSWORD=openstack
export OS_AUTH_URL=https://172.16.0.200:5000/v2.0/
keystone user-list

Next, if you are able to log in, but are presented with a Something went the wrong screen, validate that all services listed in Keystone are accessible to the server running horizon. To do this, log into the horizon server, and if you do not have the python-keystone client, install it:

sudo apt-get install -y python-keystoneclient
export OS_TENANT_NAME=cookbook export OS_USERNAME=admin export OS_PASSWORD=openstack
export OS_AUTH_URL=https://172.16.0.200:5000/v2.0/
for i in 'keystone endpoint-list | grep http | awk {'print $6'} | cut -d / -f 3,3 | cut -d : -f 1'; do ping -c 1 $i; done

[Related Article: Creating a sandbox Network server for Neutron with VirtualBox]

Additionally, you can edit the settings file for Horizon to enable more detailed logging and further troubleshooting by changing the following LOGGING lines section in /etc/openstack-dashboard/local_settings.py.

LOGGING = { 'version': 1,
When set to True this will disable all logging except 
for loggers specified in this configuration dictionary. Note
that if nothing is specified here and disable_existing_loggers 
is True, django.db.backends will still log unless it is
disabled explicitly.
'disable_existing_loggers': False, 'handlers': {
'null': {
'level': 'DEBUG',
'class': 'django.utils.log.NullHandler',
}, 'console': {
# Set the level to "DEBUG" for verbose output logging. 'level': 'INFO',
'class': 'logging.StreamHandler',
},
}, 'loggers': {
Logging from django.db.backends is VERY verbose, send to null 
by default.
'django.db.backends': { 'handlers': ['null'], 'propagate': False,
}, 'requests': {
'handlers': ['null'], 'propagate': False,
}, 'horizon': {
'handlers': ['console'], 'propagate': False,
}, 'openstack_dashboard': {
'handlers': ['console'], 'propagate': False,
}, 'novaclient': {
'handlers': ['console'], 'propagate': False,
}, 'keystoneclient': {
'handlers': ['console'], 'propagate': False,
},
'glanceclient': { 'handlers': ['console'], 'propagate': False,
}, 'nose.plugins.manager': {
'handlers': ['console'], 'propagate': False,
}
}
}

[Related Article: Configuring Ubuntu Cloud]

How OpenStack Dashboard works

With Horizon being dependent on the good health of your OpenStack environment, most horizon issues will be solved as you troubleshoot other services. That said, with the guidance in this section, you will be able to find which service is causing horizon angst and allow your users back into the system.