If you have a Linux Cloud Server with Rackspace, you may notice a couple extra processes that might not exist on a stock build of that distro on physical hardware. For example, here’s what’s in /etc/rc3.d/ on a fresh Debian build:
# ls -l /etc/rc3.d/ total 4 -rw-r--r-- 1 root root 677 Jan 1 2011 README lrwxrwxrwx 1 root root 17 May 2 2011 S01rsyslog -> ../init.d/rsyslog lrwxrwxrwx 1 root root 14 May 2 2011 S01sudo -> ../init.d/sudo lrwxrwxrwx 1 root root 31 May 3 2011 S01xe-linux-distribution -> ../init.d/xe-linux-distribution lrwxrwxrwx 1 root root 15 May 2 2011 S02acpid -> ../init.d/acpid lrwxrwxrwx 1 root root 14 May 2 2011 S02cron -> ../init.d/cron lrwxrwxrwx 1 root root 20 Jul 20 2011 S02nova-agent -> ../init.d/nova-agent lrwxrwxrwx 1 root root 13 May 2 2011 S02ssh -> ../init.d/ssh lrwxrwxrwx 1 root root 18 May 2 2011 S03bootlogs -> ../init.d/bootlogs lrwxrwxrwx 1 root root 18 May 2 2011 S04rc.local -> ../init.d/rc.local lrwxrwxrwx 1 root root 19 May 2 2011 S04rmnologin -> ../init.d/rmnologin lrwxrwxrwx 1 root root 23 May 2 2011 S04stop-bootlogd -> ../init.d/stop-bootlogd
The two extra items here are xe-linux-distribution and nova-agent, which are essential for the cloud part of your Rackspace server to function correctly. Otherwise you pretty much just have a VM. The goal of this guide is to give an overview on what these two services do, common things that can go wrong with them, and how to fix it.
xe-linux-distribution
Before we can start talking about nova-agent, we need to touch on xe-linux-distribution. The xe-linux-distribution service is provided by the xe-guest-utilities package, which is installed by default on all Linux kicks on the Rackspace Cloud. This service is responsible for allowing the VM to communicate with the hypervisor via xenstore, which is a psudo file system shared between the hypervisor and the VM. Whenever you use the Rackspace control panel to reset your root password, the request is sent to nova-agent via xenstore.
Because nova-agent relies on xenstore to function, it is important to make sure that xe-linux-distrobution starts BEFORE the nova-agent service. We will touch more on this a bit later in the troubleshooting section of this guide.
While it is possible to reinstall xe-guest-utilities, this isn’t something you should do just for the sake of doing it. The specific version of the package that you need can vary depending on the version of the underlying hypervisor. So in short: DON’T REINSTALL XE-GUEST-UTILITIES UNLESS YOU KNOW WHAT YOU’RE DOING.
Citrix kind of makes it a pain to install xe-guest-utilities if you don’t have access to the hypervisor. Say the underlying hypervisor is XenServer 5.6. First you need to create an account at their website and download the XenServer 5.6 Service Pack 2 install ISO. After you download the ISO to your server (the file name should be XenServer-5.6.100-SP2-install-cd.iso), you will then need to mount it and extract a compressed tarball containing yet another ISO, which then will actually contain the packages. You can then install the package by running the install.sh script in the Linux directory:
# mkdir xs-iso # mount -o loop XenServer-5.6.100-SP2-install-cd.iso xs-iso/ # cp xs-iso/packages.main/tools-iso.tar.bz2 . # tar -xvf tools-iso.tar.bz2 ./ ./opt/ ./opt/xensource/ ./opt/xensource/packages/ ./opt/xensource/packages/iso/ ./opt/xensource/packages/iso/xs-tools-5.6.100.iso ./opt/xensource/libexec/ ./opt/xensource/libexec/rexec_client ./opt/xensource/libexec/pv_drv_up.sh # cp opt/xensource/packages/iso/xs-tools-5.6.100.iso . # mkdir xs-tools # mount -o loop xs-tools-5.6.100.iso xs-tools # ls -lah xs-tools/Linux/ total 274K dr-xr-xr-x 5 root root 4.0K May 10 2011 . dr-xr-xr-x 3 root root 2.0K Apr 18 2011 .. dr-xr-xr-x 2 root root 2.0K Apr 7 2011 debian-etch dr-xr-xr-x 2 root root 2.0K Apr 7 2011 debian-lenny -r-xr-xr-x 1 root root 16K May 10 2011 install.sh dr-xr-xr-x 2 root root 2.0K Apr 18 2011 rhel4x -r--r--r-- 1 root root 152 Apr 18 2011 versions.deb -r--r--r-- 1 root root 254 Apr 18 2011 versions.rpm -r--r--r-- 1 root root 55K Apr 18 2011 xe-guest-utilities_5.6.100-651_amd64.deb -r--r--r-- 1 root root 55K Apr 18 2011 xe-guest-utilities_5.6.100-651_i386.deb -r--r--r-- 1 root root 16K Apr 18 2011 xe-guest-utilities-5.6.100-651.i386.rpm -r--r--r-- 1 root root 16K Apr 18 2011 xe-guest-utilities-5.6.100-651.x86_64.rpm -r--r--r-- 1 root root 49K Apr 18 2011 xe-guest-utilities-xenstore-5.6.100-651.i386.rpm -r--r--r-- 1 root root 49K Apr 18 2011 xe-guest-utilities-xenstore-5.6.100-651.x86_64.rpm -r-xr-xr-x 1 root root 7.3K May 10 2011 xe-linux-distribution # bash xs-tools/Linux/install.sh Detected `Debian 6.0.1' (debian version 6). The following changes will be made to this Virtual Machine: * packages to be installed/upgraded: - xe-guest-utilities_5.6.100-651_amd64.deb Continue? [y/n] y Selecting previously deselected package xe-guest-utilities. (Reading database ... 14589 files and directories currently installed.) Unpacking xe-guest-utilities (from .../xe-guest-utilities_5.6.100-651_amd64.deb) ... Setting up xe-guest-utilities (5.6.100-651) ... insserv: warning: script 'K01nova-agent' missing LSB tags and overrides insserv: warning: script 'nova-agent' missing LSB tags and overrides Detecting Linux distribution version: OK Starting xe daemon: OK You should now reboot this Virtual Machine. # insserv xe-linux-distribution insserv: warning: script 'K01nova-agent' missing LSB tags and overrides insserv: warning: script 'nova-agent' missing LSB tags and overrides
The last bit was me installing xe-linux-distribution as a service on Debian so it starts at boot. You might need to do this for your distribution as well. You’ll notice how insserv throws errors about nova-agent missing LSB tags, more will be said about this in a bit.
One neat thing your can do with the XenStore is pull information about your VM. For example:
# xenstore-ls vm-data provider_data = "" ip_whitelist = "" 13 = "209.114.40.174" 12 = "209.114.42.108" 11 = "209.114.42.106" 10 = "209.114.42.104" 9 = "209.114.42.7" 8 = "209.114.42.95" 7 = "209.114.42.78" 6 = "209.114.42.65" 5 = "209.114.42.90" 4 = "50.56.251.251" 3 = "89.234.21.64/28" 2 = "67.192.155.96/27" 1 = "173.203.5.160/27" 0 = "173.203.32.136/29" roles = "["identity:user-admin", "admin"]" region = "dfw" provider = "Rackspace" hostname = "test" networking = "" BC764E04D936 = "{"label": "private", "broadcast": "10.181.127.255", "ips": [{"ip": "10.181.30.181", "netmask": "255.255.128.0"\..." BC764E04CFE5 = "{"ip6s": [{"ip": "2001:4800:7810:0512:e38a:a8b5:ff04:cfe5", "netmask": 64, "enabled": "1", "gateway": "fe80::d\..." # xenstore-ls data host = "" meminfo_total = "521052" meminfo_free = "63832" os_name = "Debian 6.0.1" os_majorver = "6" os_minorver = "0" os_uname = "2.6.32-5-xen-amd64" os_distro = "debian" updated = "Sun May 26 05:32:56 UTC 2013" guest = ""
As well, on Rackspace your networking configuration is available at vm-data/networking/<your interface’s mac address>:
# ip addr show dev eth0 scope host 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether bc:76:4e:04:cf:e5 brd ff:ff:ff:ff:ff:ff # xenstore-read vm-data/networking/BC764E04CFE5 | python -mjson.tool { "broadcast": "166.78.10.255", "dns": [ "72.3.128.241", "72.3.128.240" ], "gateway": "166.78.10.1", "gateway_v6": "fe80::def", "ip6s": [ { "enabled": "1", "gateway": "fe80::def", "ip": "2001:4800:7810:0512:e38a:a8b5:ff04:cfe5", "netmask": 64 } ], "ips": [ { "enabled": "1", "gateway": "166.78.10.1", "ip": "166.78.10.124", "netmask": "255.255.255.0" } ], "label": "public", "mac": "BC:76:4E:04:CF:E5" }
nova-agent
With xe-linux-distribution out of the way, we can finally start talking about nova-agent. At its most basic form, nova-agent is a set of python scripts that allow the hypervisor to send commands to the virtual machine. This allows automated configuration of the virtual machine when it’s first created (resetting networking configuration and injecting provisioning scripts), and allows for certain administrative tasks to be performed without actually logging into the system (password resets and adding IP addresses).
The *nix version of nova-agent is currently maintained at its GitHub repository. I won’t cover building and installing it because that is already covered in its GitHub’s documentation. There are quite a few commands that nova-agent is able to understand, and it’s written in a way where the commands can be expanded. The current list is as follows:
Description | Command | Command Arguments | Response Code |
Get version info from the agent | “version” | “agent” or “xentools” or “updater” | “0” |
Get a list of agent features | “features” | “” | “0” |
Do a Diffie-Hellman Key Exchange | “keyinit” | local public key | “D0” |
(Re)set root password | “password” | encrypted password | “0” |
Configure the networking | “resetnetwork” | “” | “0” |
Update the agent | “agentupdate” | “URL,MD5SUM” | “0” |
Inject a file | “injectfile” | base64 encoding of “file_path,file_contents” | “0” |
Guest is ready/available? | |||
Update XenTools (Windows only) | |||
KMS Licensing (Windows only) |
This table is from the OpenStack wiki.
The nova-agent service reads these commands from data/host/<message_uuid> in xenstore and will place its response to data/guest/<message_uuid>. You can actually impersonate the host and send commands to yourself. For instance, say you screwed up your networking configuration and need to reset it to the defaults from when you spun up your server:
# uuid=$(uuidgen) # xenstore-write data/host/$uuid '{"name":"resetnetwork","value":""}' wait a few moments... # xenstore-read data/guest/$uuid {"message": "", "returncode": "0"}
Bam, you didn’t need to edit anything. Nova-agent automatically pulled all the networking information from vm-data/networking/ in xenstore and built all the configuration files needed. You can check just what nova-agent did by looking at its logs:
# tail /var/log/nova-agent.log 2013-05-26 08:06:29,855 [INFO] Received command 'resetnetwork' with argument: '' 2013-05-26 08:06:29,901 [INFO] staging /etc/network/interfaces (10572.tmp~) 2013-05-26 08:06:29,902 [INFO] skipping /etc/hosts (no changes) 2013-05-26 08:06:29,902 [INFO] skipping /etc/hostname (no changes) 2013-05-26 08:06:29,902 [INFO] skipping /etc/resolv.conf (no changes) 2013-05-26 08:06:29,902 [INFO] backing up /etc/network/interfaces (1369555589.bak~) 2013-05-26 08:06:29,902 [INFO] updating /etc/network/interfaces 2013-05-26 08:06:30,232 [INFO] 'resetnetwork' completed with code '0', message ''
Pretty spiffy, huh?
Troubleshooting and Common Problems
Problem: insserv complains about missing LSB tags in the nova-agent service:
# insserv xe-linux-distribution insserv: warning: script 'K01nova-agent' missing LSB tags and overrides insserv: warning: script 'nova-agent' missing LSB tags and overrides
This is due to missing LSB headers in the init script of the build of nova-agent provided by Rackspace. This has been “fixed” in the GitHub repository, however the naming convention kind of goes against LSB standards. To add LSB headers to nova-agent, just drop this in towards to top of its init script (/etc/init.d/nova-agent):
### BEGIN INIT INFO # Provides: nova-agent # Required-Start: xe-linux-distribution $remote_fs $syslog # Required-Stop: xe-linux-distribution $remote_fs $syslog # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: Start nova-agent at boot time # Description: nova-agent is a guest agent for OpenStack nova. ### END INIT INFO
You will also need to open /etc/init.d/xe-linux-distribution and change the Provides line from:
# Provides: XenServer Virtual Machine Tools
to:
# Provides: xe-linux-distribution
Then, depending on your distribution, run the following commands:
On Debian run:
# insserv xe-linux-distribution # insserv nova-agent
On CentOS/RHEL run:
# chkconfig xe-linux-distribution on # chkconfig nova-agent on
On Ubuntu run (this will throw a crap-ton of errors since most of Ubuntu uses upstart jobs, but update-rc.d refuses to honor LSB headers):
# /usr/lib/insserv/insserv xe-linux-distribution # /usr/lib/insserv/insserv nova-agent
You can then verify that it set the boot priority correctly, with xe-linux-distribution starting before nova-agent:
# ls -l /etc/rc3.d/ total 4 -rw-r--r-- 1 root root 677 Jan 1 2011 README lrwxrwxrwx 1 root root 17 May 2 2011 S01rsyslog -> ../init.d/rsyslog lrwxrwxrwx 1 root root 14 May 2 2011 S01sudo -> ../init.d/sudo lrwxrwxrwx 1 root root 31 Jun 9 06:47 S01xe-linux-distribution -> ../init.d/xe-linux-distribution lrwxrwxrwx 1 root root 15 May 2 2011 S02acpid -> ../init.d/acpid lrwxrwxrwx 1 root root 14 May 2 2011 S02cron -> ../init.d/cron lrwxrwxrwx 1 root root 20 Jul 20 2011 S02nova-agent -> ../init.d/nova-agent lrwxrwxrwx 1 root root 13 May 2 2011 S02ssh -> ../init.d/ssh lrwxrwxrwx 1 root root 18 May 2 2011 S03bootlogs -> ../init.d/bootlogs lrwxrwxrwx 1 root root 18 May 2 2011 S04rc.local -> ../init.d/rc.local lrwxrwxrwx 1 root root 19 May 2 2011 S04rmnologin -> ../init.d/rmnologin lrwxrwxrwx 1 root root 23 May 2 2011 S04stop-bootlogd -> ../init.d/stop-bootlogd
Problem: nova-agent won’t start at boot and you’re getting something like this in /var/log/nova-agent.log:
2013-05-26 12:17:28,806 [ERROR] Failed to run python code: A python exception has occurred: 2013-05-26 12:17:28,817 [ERROR] [EXC] Traceback (most recent call last): 2013-05-26 12:17:28,817 [ERROR] [EXC] File "/usr/share/nova-agent/nova-agent.py", line 38, in <module> 2013-05-26 12:17:28,817 [ERROR] [EXC] xs = plugins.XSComm() 2013-05-26 12:17:28,817 [ERROR] [EXC] File "xscomm.py", line 43, in __init__ 2013-05-26 12:17:28,817 [ERROR] [EXC] PyXenStoreError: Couldn't open connection to the xenstore: No such file or directory 2013-05-26 12:17:28,817 [ERROR] failed to parse config file '/usr/share/nova-agent/nova-agent.py'
This is what happens if nova-agent tries to start when xenstore is not accessible, which means that xe-linux-distribution isn’t started. Doing the above fix for the LSB headers should solve this problem. While this issue doesn’t generally impact the day-to-day use of your server, it does impact building from any snapshots of your server since the build will fail during the post-install.
Problem: nova-agent seems to crash on a “resetnetwork” command:
There is a known bug with nova-agent’s “resetnetwork” command when a VPN tunnel exists on the server. This can cause issues if you snapshot a server when a VPN service like OpenVPN configured to start at boot. When you build a new server from this snapshot, the infrastructure will send a “resetnetwork” to nova-agent as part of the build process, and nova-agent will segfault when it enumerates the interfaces on the server and comes across the tunnel. An issue has been raised in the GitHub for nova-agent and a patch to fix this has already been merged, however it is possible that your server is running an older version of nova-agent without the patch.
There are two possible solutions for this issue:
- Grab the latest version of nova-agent from the GitHub repository and build and install it to your server manually.
- Disable your VPN service from starting at boot before starting the snapshot of your server. The service can be re-enabled once the snapshot it complete.
Happy hacking!
Thank you, Deuce. This was the most helpful article I’ve read about nova agent. You’ve saved me eons of time struggling with my builds. I can now devote that time saved to my friend-fiction. Have a great weekend!
XXOO