Installing CloudStack 4.3 on Ubuntu Server 14.04

The universe is most surprisingly finite, it seems. Technology even more so. A few posts ago, I lamented that n IP addresses always need to be split n+1 ways. If it is true with IP addresses, it's twice as true with hardware. Given three servers, there will be demand for, at minimum, five. Certainly there must be a way for us to sneak out of this problem too, right?

We could always just install multiple sets of services on the same bare metal. ¹ While this is a viable solution part of the time, the added environmental complexity can quickly out-weigh the benefits. Even if it were an option, setting up and reallocating hardware can be a slow / costly process. If only there were another alternative. Wait, I know. We could host OUR OWN CLOUD!

A Quick Plee

I suspect that most people accept that the cloud is a magical land inhabited by cat videos, music, and an ever increasing number of unicorns. While there's no denying that it's cool, before you go down the rabbit hole, it would be worth it to take a moment and stop.

Ask yourself “Do I actually need to run my own cloud infrastructure?” You might need some quick provisioning capabilities, but do you ACTUALLY need to run a cloud? Contrary to popular thought, there is no magical band of fairies to maintain it for you and those unicorns aren't sysadmins.

While you might need the infrastructure, if possible, you might be better getting it from Microsoft Azure, Amazon, DigitalOcean, or any of the other cloud computing providers on the web.

This is your last chance. Still good? Okay.

Prerequisites

Before we get started, I'm going to assume that you've already installed Ubuntu Server 14.04, that you've installed an SSH server, and that you've setup a static IP for the machine: either by static DHCP lease or static configuration. On top of this, I'm going to assume that you are running as root for this. If not, let's go ahead and sudo su now.

Networking

One of the most important steps that we can perform is get our networking configuration setup properly. There are a number of ports that have to be open for CloudStack. Before all is done, we'll be hosting a SSH, NFS, HTTP, and LOCKD server, among others. We can save our future selves some cryptic errors later on by ensuring proper communication now.

Update the following rules to reflect your network. In this case, my example network is 10.1.x.x. After we've put the rules in place, we've save them by installing iptables-persistent. ²

iptables -A INPUT -s 10.1.0.0/16 -m state --state NEW -p tcp --dport 22 -j ACCEPT
iptables -A INPUT -s 10.1.0.0/16 -m state --state NEW -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -s 10.1.0.0/16 -m state --state NEW -p tcp --dport 8080 -j ACCEPT
iptables -A INPUT -s 10.1.0.0/16 -m state --state NEW -p udp --dport 111 -j ACCEPT
iptables -A INPUT -s 10.1.0.0/16 -m state --state NEW -p tcp --dport 111 -j ACCEPT
iptables -A INPUT -s 10.1.0.0/16 -m state --state NEW -p tcp --dport 2049 -j ACCEPT
iptables -A INPUT -s 10.1.0.0/16 -m state --state NEW -p tcp --dport 32803 -j ACCEPT
iptables -A INPUT -s 10.1.0.0/16 -m state --state NEW -p udp --dport 32769 -j ACCEPT
iptables -A INPUT -s 10.1.0.0/16 -m state --state NEW -p tcp --dport 892 -j ACCEPT
iptables -A INPUT -s 10.1.0.0/16 -m state --state NEW -p udp --dport 892 -j ACCEPT
iptables -A INPUT -s 10.1.0.0/16 -m state --state NEW -p tcp --dport 875 -j ACCEPT
iptables -A INPUT -s 10.1.0.0/16 -m state --state NEW -p udp --dport 875 -j ACCEPT
iptables -A INPUT -s 10.1.0.0/16 -m state --state NEW -p tcp --dport 662 -j ACCEPT
iptables -A INPUT -s 10.1.0.0/16 -m state --state NEW -p udp --dport 662 -j ACCEPT                
apt-get install iptables-persistent

Time

Clock skew is generally annoying, but usually quite harmless. You know, except when it isn't. While it might be annoying with just one machine, when we get two systems at play, a skewed clock can cause a number of operations to unexpectedly timeout. By installing OpenNTPD, our servers will attempt to automatically sync themselves to the correct time periodically.

apt-get install openntpd

NFS

Internally, disk volumes, templates, and snapshots are all stored on a private NFS share. While this might seem to be an unusual choice at first, it actually provides considerable flexibility. Provided that two servers are located on a moderately fast network with one another, we can perform computation and storage on different nodes, for instance. Here, we setup NFS and then expose two shares at IP:/srv/images/primary and IP:/srv/images/secondary.

apt-get install nfs-kernel-server quota
mkdir /srv/images/primary -p
mkdir /srv/images/secondary -p
echo "/srv/images *(rw,async,no_root_squash,no_subtree_check)" >> /etc/exports
exportfs -a

Initially, this is all the configuration that I attempted for NFS. While it seemed to work part of the time, I'd randomly receive errors about the SSVM timing out when allocating some larger drives. I was only able to resolve the problem after following the steps from Martijn Koster.

We'll start by updating lockd.

echo "options lockd nlm_udpport=32769 nlm_tcpport=32803" > /etc/modprobe.d/lockd.conf

Next, we need to update a few configuration files. In each of the three following files, find the listed option and update it as listed here.

/etc/default/nfs-kernel-server

 RPCMOUNTDOPTS="-p 892 --manage-gids"

/etc/default/nfs-common

NEED_STATD=yes
STATDOPTS="--port 662 --outgoing-port 2020"

/etc/default/quota

RPCRQUOTADOPTS="-p 875"

Cloudstack

Now we're ready to actually install CloudStack. We'll start by adding the security certificate for the CloudStack repo, and then updating our cache.

echo "deb http://cloudstack.apt-get.eu/ubuntu precise 4.3" > /etc/apt/sources.list.d/cloudstack.list
wget -O - http://cloudstack.apt-get.eu/release.asc | sudo apt-key add -
apt-get update

At this point an apt-cache search cloudstack should show several items. While options are nice, in this case we really only need two of the packages. These two packages, however, bring in quite a number of dependencies of their own. If you were hoping for a coffee break, this would be the time.

apt-get install cloudstack-management cloudstack-agent

MySQL

Our configuration and machine information is going to be persisted using a MySQL database. Here, we install it and then secure it from unnecessary dangers by removing the test database and preventing remote root access. When prompted, answer yes to all of the questions, filling in your new password as necessary.

apt-get install mysql-server
mysql_secure_installation

The presence of a MySQL database isn't enough, however. The following script configures connections from our CloudStack management server to increase the timeout. While not strictly necessary, this is highly recommended.

echo "[mysqld]
innodb_rollback_on_timeout=1
innodb_lock_wait_timeout=600
max_connections=350
log-bin=mysql-bin
binlog-format = 'ROW'" > /etc/mysql/conf.d/cloudstack.cnf
service mysql restart
ufw allow mysql

Now that we've configured MySQL, we're ready to actually initialize our database. After replacing the two placeholders, the following command will, as root, create our database, database user, and initial data.

cloudstack-setup-databases cloud:SECURE_PASSWORD_FOR_CLOUDSTACK@localhost --deploy-as=root:MY_ROOT_DATABASE_PASSWORD

KVM Hypervisor

The actual virtualization is performed by KVM and libvirt. In order for CloudStack to control it, we need to update the default configuration so that it will listen on TCP. Find and update the listed entries in the following two files.

/etc/libvirt/libvirtd.conf

listen_tls = 0
listen_tcp = 1
tcp_port = "16509"
mdns_adv = 0
auth_tcp = "none"

/etc/default/libvirt-bin

libvirtd_ops="-d -l"

With our configuration, CloudStack will be more than happy to launch machines for us. It could be a bit useless for those of us who aren't computers though: until we configure qemu to work with VNC, any machine we were to spin up would effectively be headless. Great for horsemen, useless for most applications.

/etc/libvirt/qemu.conf

vnc_listen = "0.0.0.0"

With libvirtd configured, we can tell apparmor to relax its security precautions just a bit.

ln -s /etc/apparmor.d/usr.sbin.libvirtd /etc/apparmor.d/disable/
ln -s /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper /etc/apparmor.d/disable/
apparmor_parser -R /etc/apparmor.d/usr.sbin.libvirtd
apparmor_parser -R /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper 

Finally, we can download our first set of virtual machines: system VMs. Interestingly enough, CloudStack not only controls all of our virtual machines for us, but it is also comprised of them. There's one VM in charge of controlling storage, and another in charge of allowing web based VNC. The same goes for any router that we might want to spin up: it too will be a VM. Because CloudStack supports several hypervisors, there are several system template options. In our case, we're using the template for 64-bit system VMs running on KVM³.

/usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt \
-m /srv/images/secondary \
-u http://download.cloud.com/templates/4.3/systemvm64template-2014-01-14-master-kvm.qcow2.bz2 \
-h kvm \
-F

MySQL Connector

One of the details that I couldn't seem to find mentioned anywhere is that, while the debs installed all of the other dependencies, the one thing they didn't install was the MySQL connector for Java. It's absence would cause the web-app to fail with a rather unintuitive 404. The solution was to grab the latest connector and to place it deep inside the CloudStack management server.

cd /tmp
wget http://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.30.tar.gz
tar -zvxf mysql-connector-java-5.1.30.tar.gz
cp /tmp/mysql-connector-java-5.1.30/mysql-connector-java-5.1.30-bin.jar /usr/share/cloudstack-management/webapps/client/WEB-INF/lib

Automatic Startup

If all has gone well so far, we can finish setup for the CloudStack agent and management server.

cloudstack-setup-management
update-rc.d tomcat6 disable
update-rc.d cloudstack-management defaults
update-rc.d cloudstack-agent defaults

Permissions

Now we need to allow the root user to login to SSH, and to allow our cloud user access to a tty. Update the following two configuration files to contain these values. /etc/sudoers

Defaults:cloud !requiretty

/etc/ssh/sshd_config

PermitRootLogin yes

Finally we need to set a secure root password that CloudStack can use and to reboot the machine.

passwd
reboot

Swappiness (Added 2014-Jul-15)

After creating several machines, they would randomly appear to incur severe performance degradation. We ended up tracing this to swapping. Running cat /proc/swaps revealed that roughly seven gigabytes of swap were being used by our machines. Disabling swap returned the system performance to it's original speed. Read more here.

To disable swap, open /etc/fstab in a text editor. Find the line containing the text “swap” and comment it out.

Closing

At this point, we should be able to see the configuration page for our instance at http://ip.add.ress:8080/client. This post is long enough without trying to configure CloudStack too. For that, the manual is actually rather straight forward.

Footnotes