Posted by: John Bresnahan | March 7, 2014

OpenStack and Docker


I recently tried to set up OpenStack with docker as the hypervisor on a single node and I ran into mountains of trouble.  I tried with DevStack and entirely failed using both the master branch and stable/havana.  After much work I was able to launch container but the network was not right.  Ultimately I found a path that worked.  This post explains how I did this.

Create the base image

CentOS 6.5

The first step is to have a VM that can support this.  Because I was using RDO this needed to be a Red Hat derivative.  I originally chose a stock vagrant CentOS 6.5 VM.  I got everything set up and then ran out of disk space (many bad words were said).  Thus I used packer and the templates here to create a CentOS VM with 40GB of disk space.  I had to change the “disk_size” value under “builders” to something larger than 40000. Then I ran the build.

packer build template.json

When this completed I had a centos-6.5 vagrant box ready to boot.


I wanted to manage this VM with Vagrant and because OpenStack is fairly intolerant to HOST_IP changes I had to inject in an interface with a static IP address.  Below is the Vagrant file I used:

Vagrant.configure("2") do |config| = "centos-6.5-base" :private_network, ip: ""
   ES_OS_MEM = 3072
   ES_OS_CPUS = 1
   config.vm.hostname = "rdodocker"
  config.vm.provider :virtualbox do |vb|
    vb.customize ["modifyvm", :id, "--memory", ES_OS_MEM]
    vb.customize ["modifyvm", :id, "--cpus", ES_OS_CPUS]

After running vagrant up to boot this VM I got the following error:

The following SSH command responded with a non-zero exit status.
Vagrant assumes that this means the command failed!
/sbin/ifdown eth1 2> /dev/null
Stdout from the command:

Stderr from the command:

Thankfully this was easily solved.  I sshed into the VM with vagrant ssh, and then ran the following:

cat <<EOM | sudo tee /etc/sysconfig/network-scripts/ifcfg-eth1 >/dev/null

After that I exited from the ssh session and repackaged the VM with vagrant package –output I added the new box to vagrant and altered my Vagrant file to boot it.  I now had a base image on which I could install OpenStack.


Through much trial and error I came to the conclusion that I needed the icehouse development release of RDO.  Unfortunately this alone was not enough to properly handle docker.  I also had to install nova from the master branch into a python virtualenv and reconfigure the box to use that nova code.  This section has the specifics of what I did.

RDO Install

I followed the instructions for installing RDO that are here, only instead of running packstack –allinone I used a custom answer file.  I generated a template answer file with the command packstack –gen-answer-file=~/answers.txt.  Then I opened that file and I substituted out ever IP address with the IP address that vagrant was injecting into my VM (in my case this was  I also set the following:


This is very important.  The docker driver does not work with neutron.  (I learned this the hard way).  I then installed RDO with the command packstack –answerfile answers.txt.


Once RDO was installed and working (without docker) I set up the VM such that nova would use docker.  The instructions here are basically what I followed.  Here is my command set:

sudo yum -y install docker-io
sudo service docker start
sudo chkconfig docker on
sudo yum -y install docker-registry
sudo usermod -G docker nova
sudo service redis start
sudo chkconfig redis on
sudo service docker-registry start
sudo chkconfig docker-registry on

I edited the file /etc/sysconfig/docker-registry and add the following:

export SETTINGS_FLAVOR=openstack
export REGISTRY_PORT=5042 
. /root/keystonerc_admin 

Note that some of the values in that file were already set.  I removed those entires.

OpenStack for Docker Configuration

I changed this entry in  /etc/nova/nova.conf

compute_driver = docker.DockerDriver

this value (and uncommented it) in /etc/glance/glance-api.conf:

container_formats = ami,ari,aki,bare,ovf,docker

Hand Rolled Nova

Unfortunately docker does not work with the current release of RDO icehouse.  Therefore I had to get the latest code from the master branch of nova.  Further I had to install it.  To be safe I put it in its own python virtualenv.  In order to install nova like this a lot of dependencies must be installed.  Here is the yum command I used to install what I needed (and in some cases just wanted).

yum update
 yum install -y telnet git libxslt-devel libffi-devel python-virtualenv mysql-devel

Then I installed nova and its package specific dependencies into a virtualenv that I created.  The command sequence is below:

git clone
virtualenv --no-site-packages /usr/local/OPENSTACKVE
source /usr/local/OPENSTACKVE/bin/activate
pip install -r requirements.txt
pip install qpid-python
pip install mysql-python
python install

At this point I had an updated version of the nova software but it was running against an old version of the data base.  Thus I had to run:

nova-manage db sync

The final step was to change all of the nova startup scripts to point to the code in the virtualenv instead of the code installed to the system.  I did this by opening up every file at /etc/init.d/openstack-nova-* and changing the exec=”/usr/bin/nova-$suffix” line to exec=”/usr/local/OPENSTACKVE/bin/nova-$suffix”.  I then rebooted the VM and I was FINALLY all set to launch docker containers that I could ssh into!

About these ads


  1. hello john, i followed your tuto.but i got error at the step #pip install -r requirements.txt.

    [root@server1 nova]# pip install -r requirements.txt
    Downloading/unpacking pbr>=0.6,=0.7.8,=0.6.1 (from -r requirements.txt (line 3))
    Downloading amqplib-1.0.2.tgz (58kB): 58kB downloaded
    Running egg_info for package amqplib
    Downloading/unpacking anyjson>=0.3.3 (from -r requirements.txt (line 4))
    Downloading anyjson-0.3.3.tar.gz
    Running egg_info for package anyjson
    Downloading/unpacking argparse (from -r requirements.txt (line 5))
    You are installing an externally hosted file. Future versions of pip will default to disallowing externally hosted files.
    Downloading argparse-1.2.1.tar.gz (69kB): 69kB downloaded
    Running egg_info for package argparse
    warning: no previously-included files matching ‘*.pyc’ found anywhere in distribution
    warning: no previously-included files matching ‘*.pyo’ found anywhere in distribution
    warning: no previously-included files matching ‘*.orig’ found anywhere in distribution
    warning: no previously-included files matching ‘*.rej’ found anywhere in distribution
    no previously-included directories found matching ‘doc/_build’
    no previously-included directories found matching ‘env24′
    no previously-included directories found matching ‘env25′
    no previously-included directories found matching ‘env26′
    no previously-included directories found matching ‘env27′
    Downloading/unpacking boto>=2.12.0,!=2.13.0 (from -r requirements.txt (line 6))
    Downloading boto-2.27.0.tar.gz (6.8MB): 532kB downloaded
    Cleaning up…
    Traceback (most recent call last):
    File “/usr/local/OPENSTACKVE/lib/python2.6/site-packages/pip/”, line 134, in main
    status =, args)
    File “/usr/local/OPENSTACKVE/lib/python2.6/site-packages/pip/commands/”, line 236, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
    File “/usr/local/OPENSTACKVE/lib/python2.6/site-packages/pip/”, line 1092, in prepare_files
    self.unpack_url(url, location, self.is_download)
    File “/usr/local/OPENSTACKVE/lib/python2.6/site-packages/pip/”, line 1238, in unpack_url
    retval = unpack_http_url(link, location, self.download_cache, self.download_dir)
    File “/usr/local/OPENSTACKVE/lib/python2.6/site-packages/pip/”, line 622, in unpack_http_url
    download_hash = _download_url(resp, link, temp_location)
    File “/usr/local/OPENSTACKVE/lib/python2.6/site-packages/pip/”, line 495, in _download_url
    chunk =
    File “/usr/lib64/python2.6/”, line 383, in read
    data = self._sock.recv(left)
    File “/usr/lib64/python2.6/”, line 542, in read
    s =
    File “/usr/lib64/python2.6/”, line 383, in read
    data = self._sock.recv(left)
    File “/usr/lib64/python2.6/”, line 215, in recv
    File “/usr/lib64/python2.6/”, line 136, in read
    SSLError: The read operation timed out

    Storing complete log in /root/.pip/pip.log

    Did i missed any thing out.
    Thank you

    • It looks like your network timed out while fetching boto. Can you try the install again?

      Unfortunately there is a bigger problem on the horizon here. Docker has been removed from nova so this guide will not work unless you only pull nova from a point before docker was removed.

  2. Last commit when Docker was still in Nova is f607712c60e222d1c26306c1035eda3535a0120e

    git clone
    git checkout f607712c60e222d1c26306c1035eda3535a0120e

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s



Get every new post delivered to your Inbox.

%d bloggers like this: