Synthetic Works

If you have a Linux Cloud Server with Rackspace, you may notice a couple extra processes that might not exist on a stock build of that distro on physical hardware. For example, here’s what’s in /etc/rc3.d/ on a fresh Debian build:

# ls -l /etc/rc3.d/
total 4
-rw-r--r-- 1 root root 677 Jan  1  2011 README
lrwxrwxrwx 1 root root  17 May  2  2011 S01rsyslog -> ../init.d/rsyslog
lrwxrwxrwx 1 root root  14 May  2  2011 S01sudo -> ../init.d/sudo
lrwxrwxrwx 1 root root  31 May  3  2011 S01xe-linux-distribution -> ../init.d/xe-linux-distribution
lrwxrwxrwx 1 root root  15 May  2  2011 S02acpid -> ../init.d/acpid
lrwxrwxrwx 1 root root  14 May  2  2011 S02cron -> ../init.d/cron
lrwxrwxrwx 1 root root  20 Jul 20  2011 S02nova-agent -> ../init.d/nova-agent
lrwxrwxrwx 1 root root  13 May  2  2011 S02ssh -> ../init.d/ssh
lrwxrwxrwx 1 root root  18 May  2  2011 S03bootlogs -> ../init.d/bootlogs
lrwxrwxrwx 1 root root  18 May  2  2011 S04rc.local -> ../init.d/rc.local
lrwxrwxrwx 1 root root  19 May  2  2011 S04rmnologin -> ../init.d/rmnologin
lrwxrwxrwx 1 root root  23 May  2  2011 S04stop-bootlogd -> ../init.d/stop-bootlogd

The two extra items here are xe-linux-distribution and nova-agent, which are essential for the cloud part of your Rackspace server to function correctly. Otherwise you pretty much just have a VM. The goal of this guide is to give an overview on what these two services do, common things that can go wrong with them, and how to fix it.

xe-linux-distribution

Before we can start talking about nova-agent, we need to touch on xe-linux-distribution. The xe-linux-distribution service is provided by the xe-guest-utilities package, which is installed by default on all Linux kicks on the Rackspace Cloud. This service is responsible for allowing the VM to communicate with the hypervisor via xenstore, which is a psudo file system shared between the hypervisor and the VM. Whenever you use the Rackspace control panel to reset your root password, the request is sent to nova-agent via xenstore.

Because nova-agent relies on xenstore to function, it is important to make sure that xe-linux-distrobution starts BEFORE the nova-agent service. We will touch more on this a bit later in the troubleshooting section of this guide.

While it is possible to reinstall xe-guest-utilities, this isn’t something you should do just for the sake of doing it. The specific version of the package that you need can vary depending on the version of the underlying hypervisor. So in short: DON’T REINSTALL XE-GUEST-UTILITIES UNLESS YOU KNOW WHAT YOU’RE DOING.

Citrix kind of makes it a pain to install xe-guest-utilities if you don’t have access to the hypervisor. Say the underlying hypervisor is XenServer 5.6. First you need to create an account at their website and download the XenServer 5.6 Service Pack 2 install ISO. After you download the ISO to your server (the file name should be XenServer-5.6.100-SP2-install-cd.iso), you will then need to mount it and extract a compressed tarball containing yet another ISO, which then will actually contain the packages. You can then install the package by running the install.sh script in the Linux directory:

# mkdir xs-iso
# mount -o loop XenServer-5.6.100-SP2-install-cd.iso xs-iso/
# cp xs-iso/packages.main/tools-iso.tar.bz2 .
# tar -xvf tools-iso.tar.bz2 
./
./opt/
./opt/xensource/
./opt/xensource/packages/
./opt/xensource/packages/iso/
./opt/xensource/packages/iso/xs-tools-5.6.100.iso
./opt/xensource/libexec/
./opt/xensource/libexec/rexec_client
./opt/xensource/libexec/pv_drv_up.sh
# cp opt/xensource/packages/iso/xs-tools-5.6.100.iso .
# mkdir xs-tools
# mount -o loop xs-tools-5.6.100.iso xs-tools
# ls -lah xs-tools/Linux/
total 274K
dr-xr-xr-x 5 root root 4.0K May 10  2011 .
dr-xr-xr-x 3 root root 2.0K Apr 18  2011 ..
dr-xr-xr-x 2 root root 2.0K Apr  7  2011 debian-etch
dr-xr-xr-x 2 root root 2.0K Apr  7  2011 debian-lenny
-r-xr-xr-x 1 root root  16K May 10  2011 install.sh
dr-xr-xr-x 2 root root 2.0K Apr 18  2011 rhel4x
-r--r--r-- 1 root root  152 Apr 18  2011 versions.deb
-r--r--r-- 1 root root  254 Apr 18  2011 versions.rpm
-r--r--r-- 1 root root  55K Apr 18  2011 xe-guest-utilities_5.6.100-651_amd64.deb
-r--r--r-- 1 root root  55K Apr 18  2011 xe-guest-utilities_5.6.100-651_i386.deb
-r--r--r-- 1 root root  16K Apr 18  2011 xe-guest-utilities-5.6.100-651.i386.rpm
-r--r--r-- 1 root root  16K Apr 18  2011 xe-guest-utilities-5.6.100-651.x86_64.rpm
-r--r--r-- 1 root root  49K Apr 18  2011 xe-guest-utilities-xenstore-5.6.100-651.i386.rpm
-r--r--r-- 1 root root  49K Apr 18  2011 xe-guest-utilities-xenstore-5.6.100-651.x86_64.rpm
-r-xr-xr-x 1 root root 7.3K May 10  2011 xe-linux-distribution
# bash xs-tools/Linux/install.sh 
Detected `Debian 6.0.1' (debian version 6).

The following changes will be made to this Virtual Machine:
  * packages to be installed/upgraded:
    - xe-guest-utilities_5.6.100-651_amd64.deb

Continue? [y/n] y

Selecting previously deselected package xe-guest-utilities.
(Reading database ... 14589 files and directories currently installed.)
Unpacking xe-guest-utilities (from .../xe-guest-utilities_5.6.100-651_amd64.deb) ...
Setting up xe-guest-utilities (5.6.100-651) ...
insserv: warning: script 'K01nova-agent' missing LSB tags and overrides
insserv: warning: script 'nova-agent' missing LSB tags and overrides
Detecting Linux distribution version: OK
Starting xe daemon:  OK

You should now reboot this Virtual Machine.
# insserv xe-linux-distribution
insserv: warning: script 'K01nova-agent' missing LSB tags and overrides
insserv: warning: script 'nova-agent' missing LSB tags and overrides

The last bit was me installing xe-linux-distribution as a service on Debian so it starts at boot. You might need to do this for your distribution as well. You’ll notice how insserv throws errors about nova-agent missing LSB tags, more will be said about this in a bit.

One neat thing your can do with the XenStore is pull information about your VM. For example:

# xenstore-ls vm-data
provider_data = ""
 ip_whitelist = ""
  13 = "209.114.40.174"
  12 = "209.114.42.108"
  11 = "209.114.42.106"
  10 = "209.114.42.104"
  9 = "209.114.42.7"
  8 = "209.114.42.95"
  7 = "209.114.42.78"
  6 = "209.114.42.65"
  5 = "209.114.42.90"
  4 = "50.56.251.251"
  3 = "89.234.21.64/28"
  2 = "67.192.155.96/27"
  1 = "173.203.5.160/27"
  0 = "173.203.32.136/29"
 roles = "["identity:user-admin", "admin"]"
 region = "dfw"
 provider = "Rackspace"
hostname = "test"
networking = ""
 BC764E04D936 = "{"label": "private", "broadcast": "10.181.127.255", "ips": [{"ip": "10.181.30.181", "netmask": "255.255.128.0"\..."
 BC764E04CFE5 = "{"ip6s": [{"ip": "2001:4800:7810:0512:e38a:a8b5:ff04:cfe5", "netmask": 64, "enabled": "1", "gateway": "fe80::d\..."

# xenstore-ls data
host = ""
meminfo_total = "521052"
meminfo_free = "63832"
os_name = "Debian 6.0.1"
os_majorver = "6"
os_minorver = "0"
os_uname = "2.6.32-5-xen-amd64"
os_distro = "debian"
updated = "Sun May 26 05:32:56 UTC 2013"
guest = ""

As well, on Rackspace your networking configuration is available at vm-data/networking/<your interface’s mac address>:

# ip addr show dev eth0 scope host
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether bc:76:4e:04:cf:e5 brd ff:ff:ff:ff:ff:ff
# xenstore-read vm-data/networking/BC764E04CFE5 | python -mjson.tool
{
    "broadcast": "166.78.10.255", 
    "dns": [
        "72.3.128.241", 
        "72.3.128.240"
    ], 
    "gateway": "166.78.10.1", 
    "gateway_v6": "fe80::def", 
    "ip6s": [
        {
            "enabled": "1", 
            "gateway": "fe80::def", 
            "ip": "2001:4800:7810:0512:e38a:a8b5:ff04:cfe5", 
            "netmask": 64
        }
    ], 
    "ips": [
        {
            "enabled": "1", 
            "gateway": "166.78.10.1", 
            "ip": "166.78.10.124", 
            "netmask": "255.255.255.0"
        }
    ], 
    "label": "public", 
    "mac": "BC:76:4E:04:CF:E5"
}

nova-agent

With xe-linux-distribution out of the way, we can finally start talking about nova-agent. At its most basic form, nova-agent is a set of python scripts that allow the hypervisor to send commands to the virtual machine. This allows automated configuration of the virtual machine when it’s first created (resetting networking configuration and injecting provisioning scripts), and allows for certain administrative tasks to be performed without actually logging into the system (password resets and adding IP addresses).

The *nix version of nova-agent is currently maintained at its GitHub repository. I won’t cover building and installing it because that is already covered in its GitHub’s documentation. There are quite a few commands that nova-agent is able to understand, and it’s written in a way where the commands can be expanded. The current list is as follows:

Description	Command	Command Arguments	Response Code
Get version info from the agent	“version”	“agent” or “xentools” or “updater”	“0”
Get a list of agent features	“features”	“”	“0”
Do a Diffie-Hellman Key Exchange	“keyinit”	local public key	“D0”
(Re)set root password	“password”	encrypted password	“0”
Configure the networking	“resetnetwork”	“”	“0”
Update the agent	“agentupdate”	“URL,MD5SUM”	“0”
Inject a file	“injectfile”	base64 encoding of “file_path,file_contents”	“0”
Guest is ready/available?
Update XenTools (Windows only)
KMS Licensing (Windows only)

This table is from the OpenStack wiki.

The nova-agent service reads these commands from data/host/<message_uuid> in xenstore and will place its response to data/guest/<message_uuid>. You can actually impersonate the host and send commands to yourself. For instance, say you screwed up your networking configuration and need to reset it to the defaults from when you spun up your server:

# uuid=$(uuidgen)
# xenstore-write data/host/$uuid '{"name":"resetnetwork","value":""}'
wait a few moments...
# xenstore-read data/guest/$uuid
{"message": "", "returncode": "0"}

Bam, you didn’t need to edit anything. Nova-agent automatically pulled all the networking information from vm-data/networking/ in xenstore and built all the configuration files needed. You can check just what nova-agent did by looking at its logs:

# tail /var/log/nova-agent.log 
2013-05-26 08:06:29,855 [INFO] Received command 'resetnetwork' with argument: ''
2013-05-26 08:06:29,901 [INFO] staging /etc/network/interfaces (10572.tmp~)
2013-05-26 08:06:29,902 [INFO] skipping /etc/hosts (no changes)
2013-05-26 08:06:29,902 [INFO] skipping /etc/hostname (no changes)
2013-05-26 08:06:29,902 [INFO] skipping /etc/resolv.conf (no changes)
2013-05-26 08:06:29,902 [INFO] backing up /etc/network/interfaces (1369555589.bak~)
2013-05-26 08:06:29,902 [INFO] updating /etc/network/interfaces
2013-05-26 08:06:30,232 [INFO] 'resetnetwork' completed with code '0', message ''

Pretty spiffy, huh?

Troubleshooting and Common Problems

Problem: insserv complains about missing LSB tags in the nova-agent service:

# insserv xe-linux-distribution
insserv: warning: script 'K01nova-agent' missing LSB tags and overrides
insserv: warning: script 'nova-agent' missing LSB tags and overrides

This is due to missing LSB headers in the init script of the build of nova-agent provided by Rackspace. This has been “fixed” in the GitHub repository, however the naming convention kind of goes against LSB standards. To add LSB headers to nova-agent, just drop this in towards to top of its init script (/etc/init.d/nova-agent):

### BEGIN INIT INFO
# Provides: nova-agent
# Required-Start: xe-linux-distribution $remote_fs $syslog
# Required-Stop: xe-linux-distribution $remote_fs $syslog
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Start nova-agent at boot time
# Description: nova-agent is a guest agent for OpenStack nova.
### END INIT INFO

You will also need to open /etc/init.d/xe-linux-distribution and change the Provides line from:

# Provides:          XenServer Virtual Machine Tools

to:

# Provides:          xe-linux-distribution

Then, depending on your distribution, run the following commands:

On Debian run:

# insserv xe-linux-distribution
# insserv nova-agent

On CentOS/RHEL run:

# chkconfig xe-linux-distribution on
# chkconfig nova-agent on

On Ubuntu run (this will throw a crap-ton of errors since most of Ubuntu uses upstart jobs, but update-rc.d refuses to honor LSB headers):

# /usr/lib/insserv/insserv xe-linux-distribution
# /usr/lib/insserv/insserv nova-agent

You can then verify that it set the boot priority correctly, with xe-linux-distribution starting before nova-agent:

# ls -l /etc/rc3.d/
total 4
-rw-r--r-- 1 root root 677 Jan  1  2011 README
lrwxrwxrwx 1 root root  17 May  2  2011 S01rsyslog -> ../init.d/rsyslog
lrwxrwxrwx 1 root root  14 May  2  2011 S01sudo -> ../init.d/sudo
lrwxrwxrwx 1 root root  31 Jun  9 06:47 S01xe-linux-distribution -> ../init.d/xe-linux-distribution
lrwxrwxrwx 1 root root  15 May  2  2011 S02acpid -> ../init.d/acpid
lrwxrwxrwx 1 root root  14 May  2  2011 S02cron -> ../init.d/cron
lrwxrwxrwx 1 root root  20 Jul 20  2011 S02nova-agent -> ../init.d/nova-agent
lrwxrwxrwx 1 root root  13 May  2  2011 S02ssh -> ../init.d/ssh
lrwxrwxrwx 1 root root  18 May  2  2011 S03bootlogs -> ../init.d/bootlogs
lrwxrwxrwx 1 root root  18 May  2  2011 S04rc.local -> ../init.d/rc.local
lrwxrwxrwx 1 root root  19 May  2  2011 S04rmnologin -> ../init.d/rmnologin
lrwxrwxrwx 1 root root  23 May  2  2011 S04stop-bootlogd -> ../init.d/stop-bootlogd

Problem: nova-agent won’t start at boot and you’re getting something like this in /var/log/nova-agent.log:

2013-05-26 12:17:28,806 [ERROR] Failed to run python code: A python exception has occurred:
2013-05-26 12:17:28,817 [ERROR] [EXC] Traceback (most recent call last):
2013-05-26 12:17:28,817 [ERROR] [EXC]   File "/usr/share/nova-agent/nova-agent.py", line 38, in <module>
2013-05-26 12:17:28,817 [ERROR] [EXC]     xs = plugins.XSComm()
2013-05-26 12:17:28,817 [ERROR] [EXC]   File "xscomm.py", line 43, in __init__
2013-05-26 12:17:28,817 [ERROR] [EXC] PyXenStoreError: Couldn't open connection to the xenstore: No such file or directory
2013-05-26 12:17:28,817 [ERROR] failed to parse config file '/usr/share/nova-agent/nova-agent.py'

This is what happens if nova-agent tries to start when xenstore is not accessible, which means that xe-linux-distribution isn’t started. Doing the above fix for the LSB headers should solve this problem. While this issue doesn’t generally impact the day-to-day use of your server, it does impact building from any snapshots of your server since the build will fail during the post-install.

Problem: nova-agent seems to crash on a “resetnetwork” command:

There is a known bug with nova-agent’s “resetnetwork” command when a VPN tunnel exists on the server. This can cause issues if you snapshot a server when a VPN service like OpenVPN configured to start at boot. When you build a new server from this snapshot, the infrastructure will send a “resetnetwork” to nova-agent as part of the build process, and nova-agent will segfault when it enumerates the interfaces on the server and comes across the tunnel. An issue has been raised in the GitHub for nova-agent and a patch to fix this has already been merged, however it is possible that your server is running an older version of nova-agent without the patch.

There are two possible solutions for this issue:

1. Grab the latest version of nova-agent from the GitHub repository and build and install it to your server manually.
2. Disable your VPN service from starting at boot before starting the snapshot of your server. The service can be re-enabled once the snapshot it complete.

Happy hacking!