Emulab Installation Documentation

Installing the Emulab Core Server

• Prev

  Installing FreeBSD on boss

• Next

  Creating the first project

• Home

Installing Emulab Software on boss

This procedure is similar to installing the testbed software on ops. Before beginning, make sure that ops is installed, running, and reachable through the network by your new boss machine.

Step 1 - Installing packages

To make sure that you're running with versions of software from the ports collection that are known to work with our software, and to save you hours of compile time, we provide pre-built binary packages of the ports required by Emulab.

If you haven't already, download the Emulab Server Package (you can use the FreeBSD 'fetch' command to download the file.)

Unpack this someplace (you need about 550MB of space, so don't use /tmp unless it is large enough). Let's say you unpacked it to /var/tmp. You would then run:

# force install of pkg
env ASSUME_ALWAYS_YES=true pkg info

# install base 'fs' applications
pkg add /var/tmp/packages/emulab-boss-6.3.txz

Of course, if you unpacked it somewhere else, put in the correct path.

We provide all of the binary packages necessary to build fs/ops/boss in our packages tarball. It is a pretty bare-bones set of ports; in particular, it does not include any text editor other than 'ed'. To get 'emacs' and 'vim' and a few other useful applications, you can install the emulab-extras package:

# install useful applications
pkg add /var/tmp/packages/emulab-extras-6.3.txz

After installing these packages from the pre-built collection, you should check for updates from the 'Emulab' packages repository. To do this, you will first need to create a description of the repository. Create /etc/pkg/Emulab.conf with the contents:

Emulab: {
  url: "https://www.emulab.net/FreeBSD/10.3/packages",
  mirror_type: NONE,
  enabled: yes
}

Now you can synchronize using:

# mark all current packages as coming from the Emulab repo
pkg annotate -ay -A repository Emulab

# check for updates to all installed packages
pkg upgrade

Please do not update any of the Emulab-installed packages other than from the Emulab repository. Otherwise you risk descending into "dependency hell"; for example, by loading some newer application package which in turn might update some shared library versions which in turn might cause other applications to stop working because they require older versions of the libraries.

In theory, you can add more packages later, either in binary form or by building from ports. Be aware that if you use binary packages loaded from the standard (presumably newer) FreeBSD repository, "dependency hell" is always just one wrong step away. Should you want to build from ports, please download the 'approved' ports collection that we have tested with and which corresponds to the binary packages we build. The package is about 66MB; it unpacks to 900MB. Unpack it to a place such as /usr/ports (if it exists, rename the old directory to ports.old, or just remove it), and install whichever ports you want to make fs feel like 'home'.

Step 2 - Unpacking and running 'configure'

At this point, you should already have made your definitions file. If not, refer to Creating the Definitions File. You will use this same file on boss, ops, and fs. You should also have the the Emulab source tree available.

Download the Emulab source tree from the emulab-devel git repository and then run its configure script, providing the path of your definitions file:

git clone git://git-public.flux.utah.edu/emulab-devel.git /usr/testbed/src/testbed
mkdir -p /usr/testbed/obj/testbed
cd /usr/testbed/obj/testbed
/usr/testbed/src/testbed/configure --with-TBDEFS=/path/to/your/defs-file

Remember, you have to use the same defs-file that you used on ops.

Step 3 - Running the boss installation script

In the object tree you've configured (say, /usr/testbed/obj/testbed), there's an 'install' subdirectory with a script called 'emulab-install'. Just run this script as root (note the same package directory argument as above):

cd install
perl emulab-install -p /var/tmp/packages boss

This will set up passwordless sudo for anyone in the wheel group. If this script fails, send the output to Utah so we can fix it. Note that emulab-install requires internet access so it can grab some files.

The source installation is performed by the emulab-install script and should be completely automatic. However, you may check it or fix up problems by running the makes yourself, as follows:

gmake
gmake boss-install

Step 4 - Verify SSH configuration

The emulab-install script should have set up root ssh from boss to ops. To confirm this, make sure this works:

boss> sudo ssh ops ls /

If this fails, you will need to set this up by hand, typing a password:

scp /root/.ssh/identity.pub ops:/root/.ssh/authorized_keys

Step 5 - Setting up named

The testbed software manipulates DNS zone files for two reasons. First, it adds your nodes to them so that you don't have to. Second, it creates CNAMEs for all the nodes in every experiment (so that you can use, for example, 'nodeA.myexp.myproj.emulab.net' to refer to your node regardless of which physical node it got mapped to).

The named_setup script does this by generating zone files. In general, it concatenates a '.head' file, written by you, with its own generated entries. The main zone file is /etc/namedb/OURDOMAIN.db, where OURDOMAIN is from your defs file. (OURDOMAIN, unless explicitly specified, is taken to be the domain portion of BOSSNODE). We also generate reverse zone files (for inverse lookups, i.e. turning IP addresses back into names) in /etc/namedb/reverse/SUBNET.db, where SUBNET is the class-C subnet in which the addresses reside (i.e. 10.0.0.db). This value is defined in the defs file, as is TESTBED_NETWORK.

emulab-install makes a reasonable attempt to create a set of named configuration files for you, placing them in /etc/namedb. If your testbed consists of a single class-C network, then these files will most likely be correct, although you'll want to look at them to make sure. Look at these files to make sure:

/etc/namedb/OURDOMAIN.db.head
/etc/namedb/reverse/SUBNET.db.head
/etc/namedb/named.conf

If you have more than one class-C subnet for your testbed, you'll need a copy of the reverse zone file for each one. You want to put boss, ops, and any 'infrastructure' equipment such as routers and switches into the zone files. These zone files do not need to include the nodes - they will be added automatically. Be sure to edit /etc/namedb/namedb.conf if you add any reverse map files (follow the format for the existing entry).

Once you think you've got things set up, run make sure the following command doesn't give you any error messages:

/usr/testbed/sbin/named_setup

This command will generate the following files:

/etc/namedb/OURDOMAIN.db
/etc/namedb/reverse/SUBNET.db

Step 5a - Named for an unroutable control network.

If you are using routable IP addresses for all of your testbed, then you can skip this step. However, if you are using unroutable private IP addresses for all or part of the testbed, you will need to use a feature of bind called 'views' so that the private addresses are only exposed to clients within the testbed. See the bind documentation for more details on this feature. You will want to make sure that loopback queries from boss itself see the internal view; i.e., you want boss to resolve its own hostname to its private address, not its public one.

In order to use multiple views, we generate multiple zone files. In addition to OURDOMAIN.db, which will be used for the 'external' view, we create OURDOMAIN.internal.db for use with the 'internal' view. So, you'll also need to create OURDOMAIN.internal.db.head. When we generate the zone files, only publically-routable addresses are put into OURDOMAIN.db. OURDOMAIN.internal.db contains all addresses, both public and private. So basically, you'll want to put the public addresses for boss, ops, etc. into OURDOMAIN.db.head, and their private addresses into OURDOMAIN.internal.db.head.

Step 5b - Named recursive queries.

Current versions of Bind default to not allowing recursive lookups. This is a useful security feature, but means that nodes cannot lookup outside hosts! One simple fix to this is to explicitly allow recursion. In your named.conf file, find the options section and add the following line:

allow-recursion { any; };

allow-recursion { 127.0.0.1; 10.1.0.0/24; };

Step 6 - Miscellaneous configuration

There are a few things that we haven't been able to completely automate just yet, though we hope to soon.

hosts file - You want to put boss/ops name/IP addresses in /etc/hosts on both boss and ops to avoid boottime circular dependencies (because of NFS cross mounts). This is done for you in emulab-install, but you should confirm that it was done properly.

SSL Certificates - Our apache config file expects to find SSL certificates in:

/usr/local/etc/apache/ssl.crt/www.<sitename>.crt
/usr/local/etc/apache/ssl.key/www.<sitename>.key

Where <sitename> is OURDOMAIN from the configure defs file, which defaults to boss' domain name.

emulab-install already generated a temporary no-passphrase certificate for you and placed it in the locations specified above. However, we recommend that you get a "real" certificate from Verisign (or one of the others).

DHCPD - emulab-install generated a dhcpd.conf.template file and installed it in /usr/local/etc (the template is derived from information you provided in the defs file). It then generated an actual dhcpd.conf file and started up dhcpd for you. Do not edit the dhcpd.conf file directly! If you need to make changes, change the template instead and then run:

/usr/testbed/sbin/dhcpd_makeconf -i -r

locate database - It can be useful to update the 'locate' database to help you find files as you're learning the system. This normally happens nightly, but you can force it to happen now by running the following as root:

/usr/libexec/locate.updatedb &

This will run in the background for several minutes. You can then find 'foo.conf' by running 'locate foo.conf'.

dhclient - If you don't control the public network to which boss and ops (and perhaps tipserv) are attached, your network admins may want you to run dhclient on those interfaces (and you may want to as well so that your boss /etc/resolv.conf contains the correct nameserver returned by dhclient, to avoid configuring it statically). However, boss runs its own nameserver that ops and the experimental nodes must contact to resolve internal names, so you must get dhclient to correctly write /etc/resolv.conf (i.e., boss must be the first nameserver). The easiest way to solve this problem is to add the following line to boss' /etc/dhclient.conf file:

prepend domain-name-servers 127.0.0.1;

and add the following line to ops' (and any other non-experimental nodes, such as tipservs) /etc/dhclient.conf:

prepend domain-name-servers <bossip>;

where <bossip> is replaced by a valid IP address for your boss. Make sure to restart dhclient on these machines since dhclient will continue to overwrite /etc/resolv.conf incorrectly.

Step 7 - Quotas (Optional)

If you are planning to run quotas for Emulab users on the ops (or fs) server, you will need to establish a default quota value for all users. Ideally, the quota would be configurable per-user, but for now all quotas are initialized from a "prototypical" user. Note that you can change individual user quotas later by running edquota(8) on ops (or fs). First, log into your ops node (or your fs node if you are using a separate file server node).

Now set the quota for elabman on each quota-enabled filesystem. For example:

edquota -e /proj:2000000:2000000 -e /users:1000000:1000000 elabman

would set a 1GB quota on /proj and 512MB on /users for the elabman (prototypical) user. Once the prototype user quotas are established, you should do:

edquota -p elabman 10000-15000

which would automatically apply the elabman quotas to any user created with uids between 10000 and 15000, assuming that you wanted to allow up to 5000 users.

Step 8 - Reboot boss

Okay, go ahead and reboot boss now, and make sure it comes up okay.

• Prev

  Installing FreeBSD on boss

• Next

  Creating the first project

• Home