OpenStack is a complex suite of software that can make tracking down issues and faults quite daunting to beginners and experienced system administrators alike. While there is no single approach to troubleshooting systems, understanding where OpenStack logs vital information or what tools are available to help track down bugs will help resolve issues we may encounter. However, OpenStack like all software will have bugs that we are not able to solve ourselves. In that case, we will show you how gathering the required information so that the OpenStack community can identify bugs and suggest fixes which is important in ensuring those bugs or issues are dealt with quickly and efficiently.
Logging is important in all computer systems, but the more complex the system, the more you rely on logging to be able to spot problems and cut down on troubleshooting time. Understanding logging in OpenStack is important to ensure your environment is healthy and you are able to submit relevant log entries back to the community to help fix bugs.
Log in as the root user onto the appropriate servers where the OpenStack services are installed. This makes troubleshooting easier as root privileges are required to view all the logs.
How to accomplish it…
OpenStack produces a large number of logs that help troubleshoot our OpenStack installations. The following details outline where these services write their logs:
OpenStack Compute services logs
Logs for the OpenStack Compute services are written to /var/log/nova/, which is owned by the nova user, by default. To read these, log in as the root user (or use sudo privileges when accessing the files). The following is a list of services and their corresponding logs. Note that not all logs exist on all servers. For example, compute.log exists on your compute hosts only:
Log entries regarding the spinning up and running of the instances
Log entries regarding network state, assignment, routing, and security groups
Log entries produced when running the nova-manage command
Log entries regarding services making requests for database information
Log entries pertaining to the scheduler, its assignment of tasks to nodes, and messages from the queue
Log entries regarding user interaction with OpenStack as well as messages regarding interaction with other components of OpenStack
Entries regarding the nova-cert process
Details about the nova-console VNC service
Authentication details related to the nova-console service
Network information regarding the dhcpbridge service
OpenStack Dashboard logs
OpenStack Dashboard (Horizon) is a web application that runs through Apache by default, so any errors and access details will be in the Apache logs. These can be found in /var/log/apache2/*.log, which will help you understand who is accessing the service as well as the report on any errors seen with the service.
OpenStack Storage logs
OpenStack Object Storage (Swift) writes logs to syslog by default. On an Ubuntu system, these can be viewed in /var/log/syslog. On other systems, these might be available at /var/log/messages.
The OpenStack Block Storage service, Cinder, will produce logs in /var/log/cinder by default. The following list is a breakdown of the log files:
Details about the cinder-api service
Details related to the operation of the Cinder scheduling service
Log entries related to the Cinder volume service
OpenStack Identity logs
The OpenStack Identity service, Keystone, writes its logging information to /var/log/keystone/keystone.log. Depending on how you have Keystone configured, the information in this log file can be very sparse to extremely verbose including complete plaintext requests.
OpenStack Image Service logs
The OpenStack Image Service Glance stores its logs in /var/log/glance/*.log with a separate log file for each service. The following is a list of the default log files:
Entries related to the glance API
Log entries related to the Glance registry service. Things like metadata updates and access will be stored here, depending on your logging configuration.
OpenStack Network Service logs
OpenStack Networking Service, formerly Quantum, now Neutron, stores its log files in /var/log/quantum/*.log with a separate log file for each service. The following is a list of the corresponding logs:
Log entries pertaining to the dhcp-agent
Log entries related to the l3 agent and its functionality
This file contains log entries related to requests Quantum has proxied to the Nova metadata service.
Entries related the the operation of Open vSwitch. When implementing OpenStack Networking, if you use a different plugin, its log file will be named accordingly.
Details and entries related to the quantum API service
OpenVSwitch Server: /var/log/openvswitch/ovs-vswitchd.log
Details and entries related to the OpenVSwitch Switch Daemon
Changing log levels
By default, each OpenStack service has a sane level of logging, which is determined by the level set as a Warning. That is, it will log enough information to provide you the status of the running system as well as some basic troubleshooting information. However, there will be times that you need to adjust the logging verbosity either up or down to help diagnose an issue or reduce logging noise.
As each service can be configured similarly, we will show you how to make these changes on the OpenStack Compute service.
Log-level settings in OpenStack Compute services
To do this, log into the box where the OpenStack Compute service is running and execute the following commands:
sudo vim /etc/nova/logging.conf
Change the following log levels to either DEBUG, INFO or WARNING in any of the services listed:
Log-level settings in other OpenStack services
Other services such as Glance and Keystone currently have their log-level settings within their main configuration files such as /etc/glance/glance-api.conf. Adjust the log levels by altering the following lines to achieve INFO or DEBUG levels:
Restart the relevant service to pick up the log-level change.
How it works…
Logging is an important activity in any software, and OpenStack is no different. It allows an administrator to track down problematic activity that can be used in conjunction with the community to help provide a solution. Understanding where the services log and managing those logs to allow someone to identify problems quickly and easily are important.