OpenStack is a complex suite of software that can make tracking down issues and faults quite daunting to beginners and experienced system administrators alike. While there is no single approach to troubleshooting systems, understanding where OpenStack logs vital information or what tools are available to help track down bugs will help resolve issues we may encounter. However, OpenStack like all software will have bugs that we are not able to solve ourselves. In that case, we will show you how to gather the required information so that the OpenStack community can identify bugs and suggest fixes that are important in ensuring those bugs or issues are dealt with quickly and efficiently.
Understanding logging in Troubleshooting Openstack
- Logging is important in all computer systems, but the more complex the system, the more you rely on logging to be able to spot problems and cut down on troubleshooting time. Understanding logging in OpenStack is important to ensure your environment is healthy and you are able to submit relevant log entries back to the community to help fix bugs.
- Log in as the root user onto the appropriate servers where the OpenStack services are installed. This makes troubleshooting easier as root privileges are required to view all the logs.
[Related Article: Troubleshooting OpenStack Authentication and Networking]
- OpenStack produces a large number of logs that help troubleshoot our OpenStack installations. The following details outline where these services write their logs:
#1.OpenStack Compute Services Logs
Logs for the OpenStack Compute services are written to /var/log/nova/, which is owned by the nova user, by default. To read these, log in as the root user (or use sudo privileges when accessing the files). The following is a list of services and their corresponding logs. Note that not all logs exist on all servers. For example, compute.log exists on your compute hosts only:
Log entries regarding the spinning up and running of the instances
Log entries regarding network state, assignment, routing, and security groups
Log entries produced when running the nova-manage command
Log entries regarding services making requests for database information
Log entries pertaining to the scheduler, its assignment of tasks to nodes, and messages from the queue
Log entries regarding user interaction with OpenStack as well as messages regarding interaction with other components of OpenStack
Entries regarding the nova-cert process
Details about the nova-console VNC service
Authentication details related to the nova-console service
Subscribe to our youtube channel to get new updates..!
Network information regarding the dhcpbridge service
#2.OpenStack Dashboard logs
OpenStack Dashboard (Horizon) is a web application that runs through Apache by default, so any errors and access details will be in the Apache logs. These can be found in /var/log/apache2/*.log, which will help you understand who is accessing the service as well as the report on any errors seen with the service.
[Related Article: Troubleshooting OpenStack Object Storage services and Dashboard]
#3.OpenStack Storage logs
OpenStack Object Storage (Swift) writes logs to syslog by default. On an Ubuntu system, these can be viewed in /var/log/syslog. On other systems, these might be available at /var/log/messages.
The OpenStack Block Storage service, Cinder, will produce logs in /var/log/cinder by default. The following list is a breakdown of the log files:
Details about the cinder-api service
Details related to the operation of the Cinder scheduling service
Log entries related to the Cinder volume service
[Related Article: OpenStack Networking]
#4.OpenStack Identity logs
The OpenStack Identity service, Keystone, writes its logging information to /var/log/keystone/keystone.log. Depending on how you have Keystone configured, the information in this log file can be very sparse to extremely verbose including complete plaintext requests.
OpenStack Image Service logs
The OpenStack Image Service Glance stores its logs in /var/log/glance/*.log with a separate log file for each service. The following is a list of the default log files:
Entries related to the glance API
Log entries related to the Glance registry service. Things like metadata updates and access will be stored here, depending on your logging configuration.
#5.OpenStack Network Service logs
OpenStack Networking Service, formerly Quantum, now Neutron, stores its log files in /var/log/quantum/*.log with a separate log file for each service. The following is a list of the corresponding logs:
Log entries pertaining to the dhcp-agent
Log entries related to the l3 agent and its functionality
This file contains log entries related to requests Quantum has proxied to the Nova metadata service.
Entries related the
[DEFAULT] #show more verbose log output (sets INFO log level output) verbose=FALSE #Show debugging output in logs(sets DEBUG log level output) debug=FALSE
operation of Open vSwitch. When implementing OpenStack Networking, if you use a different plugin, its log file will be named accordingly.
Details and entries related to the quantum API service
OpenVSwitch Server: /var/log/openvswitch/ovs-vswitchd.log
Details and entries related to the OpenVSwitch Switch Daemon
Changing log levels
By default, each OpenStack service has a sane level of logging, which is determined by the level set as a Warning. That is, it will log enough information to provide you the status of the running system as well as some basic troubleshooting information. However, there will be times that you need to adjust the logging verbosity either up or down to help diagnose an issue or reduce logging noise.
As each service can be configured similarly, we will show you how to make these changes on the OpenStack Compute service.
Log-level settings in OpenStack Compute services
To do this, log into the box where the OpenStack Compute service is running and execute the following commands:
sudo vim /etc/nova/logging.conf
[Related Article: Troubleshooting OpenStack Compute services]
Change the following log levels to either DEBUG, INFO, or WARNING in any of the services listed:
[logger_root] Level=WARNING handlers=null [logger_nova] Level=INFO handlers=stderr qualname=nova
Other services such as Glance and Keystone currently have their log-level settings within their main configuration files such as /etc/glance/glance-api.conf. Adjust the log levels by altering the following lines to achieve INFO or DEBUG levels:
[DEFAULT] #show more verbose log outputs(sets INFO Log level output) verbose=False #Show debugging output in logs (sets DEBUG Log level output) debug=False
Restart the relevant service to pick up the log-level change.
Logging is an important activity in any software, and OpenStack is no different. It allows an administrator to track down problematic activity that can be used in conjunction with the community to help provide a solution. Understanding where the services log and managing those logs to allow someone to identify problems quickly and easily are important