Update-Testbed.Html

Upgrading Emulab

Emulab Installation Documentation

Upgrading Emulab

Prev
Setting up the Console Server
Home

Updating the Testbed

This document describes the general procedure for upgrading your existing Emulab. We assume that you are running or have upgraded to FreeBSD 6.3 on your boss/ops/fs servers. If you haven't already, see the Appendix on upgrading from 4.10 to 6.3 in the Wiki. These instructions also apply if you are tracking the Emulab source code via GIT.

Step 0

"Shutdown" the testbed. Disable the web interface and prevent user logins to "users":

boss> wap webcontrol -l nologin

We also want to make sure that none of the testbed daemons are running:

boss> sudo mv -f /usr/local/etc/rc.d/3.testbed.sh /usr/local/etc/rc.d/3.testbed.sh.save

And add this to /etc/rc.conf to prevent the cron-based daemons from running:

cron_enable="NO"

Then reboot boss.

Step 1

Unpack the Emulab software. You should do this in a directory accessible to your boss and ops machines (and your fs and tipserv machines if you have them) to make the various installs easier. Typically this would be a directory in /users or /proj.

If you doing this install via a GIT checkout, then do your checkout to a location in /users or /proj.

Step 2

Look at doc/UPDATING in the source tree to find out what you have to update in the boss/ops FreeBSD environment before updating the Emulab software. Entries are timestamped, so look for things that have changed since the last time you installed. Some entries are marked as needing to be done after install. Skip those but remember them for later.

NOTE: if you have just upgraded your boss and ops nodes to FreeBSD 6.3 using the instructions in http://users.emulab.net/trac/emulab/wiki/appendix/upgrade-410-63.html and the update-ports script, you can skip some of this. Specifically, the script has updated and patched your mysql (20060831 and 20061201), updated ulsshxmlrpcpp (20070317), updated tftp (20070320), and replaced elvin with pubsub (20070511).

Step 3

BACKUP YOUR DATABASE on boss. Do:

boss> mysqldump tbdb > db.backup

Step 4

Update the DB on boss. Look at sql/database-migrate.txt, which is a list of SQL commands to perform. In some cases, it also includes instructions regarding operations you need to perform on your data to make it work in the new schema.

Note: To find out which changes to apply in database-migrate.txt, diff the version of this file from the last release with the version from this release.

Some changes in database-migrate.txt may require you to run scripts that do not exist until after running gmake boss-install. In that case just run them afterward.

If you need to upgrade the Emulab Server Packages for this release, follow the directions in Step 1 of [boss-emusoft.html Installing Emulab Software on Boss] (add the '-f' option to the pkg_add command to force an upgrade).

Step 5

Configure and build an object tree. This should be done in a different directory then your source tree:

boss> cd /your/objdir
boss> /your/srcdir/configure --with-TBDEFS=<your-defs-file> <options>
boss> gmake clean
boss> gmake

Step 6

This step applies to releases greater then 5.0, or if you are doing this install from a current checkout.

Starting with database revision 4.160, we have changed the method with which database changes are applied. Instead of storing sequences of instructions in database-migrate.txt (very error prone), changes are now stored as perl scripts that are applied using a utility script in the db directory. To apply these changes:

boss> cd /your/objdir/db
boss> sudo gmake install
boss> cd /your/srcdir/sql/updates
boss> /usr/testbed/sbin/dbupdate tbdb

Do not worry if nothing happens; your database may already be uptodate.

Step 7

Now, install the newly-built Emulab software:

boss> cd /your/objdir
boss> sudo gmake boss-install

Be aware that the "boss-install" make target will do further checks for missing rows in the DB tables. In that case, that target will fail and tell you what do do to fix your DB. It does the table check in two pieces, so it's possible that you will have to fix the DB and restart the boss-install process again. Also note that you should not run the install/boss-install perl script as part of an updating procedure. It is only meant for an initial installation, and is not setup to perform upgrades.

You must also update the rc.d scripts:

boss> cd /your/objdir/rc.d
boss> sudo gmake install

If anything goes wrong with the DB update step, contact us. You can still abort the upgrade at this point by putting back the old DB:

mysql> drop database tbdb;
mysql> create database tbdb;
quit
boss> mysql < db.backup

and bringing the testbed back up. After this step, you are committed to the upgrade.

Step 8

If you have one or more "tipserv" machines, login to them, cd to your build directory and do:

tipserv> sudo gmake tipserv-install

Step 9

If you need to upgrade the Emulab Server Packages for this release, follow the directions in Step 2 of [ops-emusoft.html Installing Emulab Software on Ops] (add the '-f' option to the pkg_add command to force an upgrade). This only needs to be done on the ops server.

If you have a combined ops/fs machine (the common case), skip to step 10. Otherwise:

ops> sudo gmake ops-install
ops> cd rc.d; sudo gmake control-install

Skip to Step 11.

Step 10

ops> sudo gmake opsfs-install

Step 11

Install a "shared" copyright to appear on the web pages. We have not yet integrated this step, so it is a little awkward:

Edit www/copyright-contrib.html and fill in values for @SITENAME@, @SITEDATES@, @SITECOPYRIGHT@. Then copy this file to copyright.html in your installed testbed tree, e.g.:

boss> cp copyright-contrib.html /usr/testbed/www/copyright.html

Step 12

Undo the changes in Step 1:

boss> wap webcontrol -l login
boss> sudo rm -f /usr/local/etc/rc.d/3.testbed.sh.save

And enable cron-based daemons again in /etc/rc.conf:

cron_enable="YES"

Step 13

Reboot tipservers, ops and boss and make sure things boot up ok. Try creating/swapping/modifying/destroying an experiment and look for errors.

Step 14

Bring up the web interface and inform users that they should reboot their existing experiment nodes if they notice odd behaviors.

Things should be backward compatible so you don't have to update the standard disk images immediately. But you will want to do it.

Prev
Setting up the Console Server
Home