Troubleshooting OpenStack


OpenStack is a complex suite of software that can make tracking down issues and faults quite daunting to beginners and experienced system administrators alike. While there is no single approach to troubleshooting systems, understanding where OpenStack logs vital information or what tools are available to help track down bugs will help resolve issues we may encounter. However, OpenStack like all software will have bugs that we are not able to solve ourselves. In that case, we will show you how gathering the required information so that the OpenStack community can identify bugs and suggest fixes which is important in ensuring those bugs or issues are dealt with quickly and efficiently.

Understanding logging

Logging is important in all computer systems, but the more complex the system, the more you rely on logging to be able to spot problems and cut down on troubleshooting time. Understanding logging in OpenStack is important to ensure your environment is healthy and you are able to submit relevant log entries back to the community to help fix bugs.

Getting ready

Log in as the root user onto the appropriate servers where the OpenStack services are installed. This makes troubleshooting easier as root privileges are required to view all the logs.

How to accomplish it…

OpenStack produces a large number of logs that help troubleshoot our OpenStack installations. The following details outline where these services write their logs:

OpenStack Compute services logs

Logs for the OpenStack Compute services are written to /var/log/nova/, which is owned by the nova user, by default. To read these, log in as the root user (or use sudo privileges when accessing the files). The following is a list of services and their corresponding logs. Note that not all logs exist on all servers. For example, compute.log exists on your compute hosts only:

nova-compute: /var/log/nova/nova-compute.log

Log entries regarding the spinning up and running of the instances

nova-network: /var/log/nova/nova-network.log

Log entries regarding network state, assignment, routing, and security groups

nova-manage: /var/log/nova/nova-manage.log

Log entries produced when running the nova-manage command

nova-conductor: /var/log/nova/nova-conductor.log

Log entries regarding services making requests for database information

nova-scheduler: /var/log/nova/nova-scheduler.log

Log entries pertaining to the scheduler, its assignment of tasks to nodes, and messages from the queue

nova-api: /var/log/nova/nova-api.log

Log entries regarding user interaction with OpenStack as well as messages regarding interaction with other components of OpenStack

nova-cert: /var/log/nova/nova-cert.log

Entries regarding the nova-cert process

nova-console: /var/log/nova/nova-console.log

Details about the nova-console VNC service

nova-consoleauth: /var/log/nova/nova-consoleauth.log

Authentication details related to the nova-console service

nova-dhcpbridge: /var/log/nova/nova-dhcpbridge.log

Network information regarding the dhcpbridge service

OpenStack Dashboard logs

OpenStack Dashboard (Horizon) is a web application that runs through Apache by default, so any errors and access details will be in the Apache logs. These can be found in /var/log/apache2/*.log, which will help you understand who is accessing the service as well as the report on any errors seen with the service.

OpenStack Storage logs

OpenStack Object Storage (Swift) writes logs to syslog by default. On an Ubuntu system, these can be viewed in /var/log/syslog. On other systems, these might be available at /var/log/messages.

The OpenStack Block Storage service, Cinder, will produce logs in /var/log/cinder by default. The following list is a breakdown of the log files:

cinder-api: /var/log/cinder/cinder-api.log

Details about the cinder-api service

cinder-scheduler: /var/log/cinder-scheduler.log

Details related to the operation of the Cinder scheduling service

cinder-volume: /var/log/cinder/cinder-volume.log

Log entries related to the Cinder volume service

OpenStack Identity logs

The OpenStack Identity service, Keystone, writes its logging information to /var/log/keystone/keystone.log. Depending on how you have Keystone configured, the information in this log file can be very sparse to extremely verbose including complete plaintext requests.

OpenStack Image Service logs

The OpenStack Image Service Glance stores its logs in /var/log/glance/*.log with a separate log file for each service. The following is a list of the default log files:

api: /var/log/glance/api.log

Entries related to the glance API

registry: /var/log/glance/registry.log

Log entries related to the Glance registry service. Things like metadata updates and access will be stored here, depending on your logging configuration.

OpenStack Network Service logs

OpenStack Networking Service, formerly Quantum, now Neutron, stores its log files in /var/log/quantum/*.log with a separate log file for each service. The following is a list of the corresponding logs:

dhcp-agent: /var/log/quantum/dhcp-agent.log

Log entries pertaining to the dhcp-agent

l3-agent: /var/log/quantum/l3-agent.log

Log entries related to the l3 agent and its functionality

metadata-agent: /var/log/quantum/metadata-agent.log

This file contains log entries related to requests Quantum has proxied to the Nova metadata service.

openvswitch-agent: /var/log/quantum/openvswitch-agent.log

Entries related the the operation of Open vSwitch. When implementing OpenStack Networking, if you use a different plugin, its log file will be named accordingly.

server: /var/log/quantum/server.log

Details and entries related to the quantum API service

OpenVSwitch Server: /var/log/openvswitch/ovs-vswitchd.log

Details and entries related to the OpenVSwitch Switch Daemon

Changing log levels

By default, each OpenStack service has a sane level of logging, which is determined by the level set as a Warning. That is, it will log enough information to provide you the status of the running system as well as some basic troubleshooting information. However, there will be times that you need to adjust the logging verbosity either up or down to help diagnose an issue or reduce logging noise.

As each service can be configured similarly, we will show you how to make these changes on the OpenStack Compute service.

Log-level settings in OpenStack Compute services

To do this, log into the box where the OpenStack Compute service is running and execute the following commands:

sudo vim /etc/nova/logging.conf

Change the following log levels to either DEBUG, INFO or WARNING in any of the services listed:


Log-level settings in other OpenStack services

Other services such as Glance and Keystone currently have their log-level settings within their main configuration files such as /etc/glance/glance-api.conf. Adjust the log levels by altering the following lines to achieve INFO or DEBUG levels:



Restart the relevant service to pick up the log-level change.

How it works…

Logging is an important activity in any software, and OpenStack is no different. It allows an administrator to track down problematic activity that can be used in conjunction with the community to help provide a solution. Understanding where the services log and managing those logs to allow someone to identify problems quickly and easily are important.


0 Responses on Troubleshooting OpenStack"

Leave a Message

Your email address will not be published. Required fields are marked *

Copy Rights Reserved © Mindmajix.com All rights reserved. Disclaimer.