Controlling Emulab Node Configuration Services

Introduction

Emulab uses a largely "node driven" strategy for node configuration; i.e., it is a pull-style process with the nodes requesting information from Emulab's central control servers and configuring themselves (see this page for more information). At times, it is useful for Emulab users and administrators to apply various controls to these services. For instance, a user might want to disable Emulab's standard mechanism for experimental interface configuration, and replace it with something else. Or perhaps the experimenter requires that Emulab shared volumes not be mounted via NFS, as we provide in a standard node configuration.

Emulab now supports several ways in which client side configuration scripts can be augmented, replaced, or enabled and disabled.  In addition, some services allow users to customize their configuration details.  For instance, users can customize the behavior of rc.ifconfig by telling it that the user will handle the configuration of specific interfaces, and that rc.ifconfig can ignore those interfaces; and users can customize the behavior of rc.hostnames by telling it to prepend and/or append to /etc/hosts.

Users can control node configuration scripts within their experiment description file (and by including such controls in their images); administrators can control configuration scripts based on node type, node, or OS. These scripts are called hook scripts and can be attached to service scripts via your NS file, or included in a custom image.

Administrators can allow their settings to be overridden by users, or not. These are service scripts and we describe briefly how to create them below.

Let's jump into the details!

Controllable Services

Emulab only allows you to control services that are supported on the client side. For now, this list is a superset of services, since not all services are supported on all OSes (for instance, Windows doesn't support NFS, so there is no point trying to add an rc.nfs hook or replacement for Windows!). However, we don't have a good way to encode these sorts of OS variabilities at present.

Here is a current list of service scripts implemented as part of Emulab. Users can add hooks to each one of these scripts and administrators can add entirely new service scripts.

(current as of 10/14/2016)

User Hook Script Interface

In order to hook their own scripts into services, experimenters can use the following commands in their NS file.  (There is not yet a geni-lib Cloudlab/ProtoGENI interface.)

The client side service control commands look like this:

    tb-set-node-service "rc.foo" \
        -node (""|$node) -env (boot|load) -whence (every|first) \
        -script "/path/to/script" -scriptblob "<blobid>" \
        -enable (1|0) -enablehooks (1|0) -fatal (1|0)

    tb-add-node-service-hook "rc.foo" \
        -node (""|$node) -env (boot|load) -whence (every|first) \
        -script "/path/to/script" -scriptblob "<blobid>" \
        -op (boot|shutdown|reconfig|reset) -point (pre|post) \
        -argv "" -fatal (1|0)

Services are controlled or hooked onto by name (use the names in the table above); the service name is the first argument to both commands.

All other options (prefixed by "-") are "optional". First, we discuss the common arguments.

• -node: defaults to "", meaning that the command applies to all nodes in the experiment. Can also be set to $node to select a specific node.

• -env: can be one of "boot" or "load". If set to "boot", the command applies in the traditional path when the node self-configures after each boot, within the booted OS. If set to "load", the command applies within the disk-loading environment, in which configuration scripts can modify the freshly-loaded disk (note that in this mode, the scripts must be aware that they are in the execution context of an Emulab MFS, not the OS loaded on the disk). "load" mode is not fully supported yet by Emulab's MFSes. REQUIRED PARAMETER

• -whence: can be "every" or "first". If "every", the script or hook specified by the command will be affected on every boot or load. (In practice, "every" is only useful with the "boot" env setting, because when the node is booting an MFS to load its disk or be debugged, each boot of the MFS is a fresh boot; there is no way to mark that it is the "first" boot. On the other hand, if the env setting is "boot", our framework records whether a configuration script or hook has been run, and if it is only to be run the "first" time, it does not run more than once. REQUIRED PARAMETER

• -script: A path to an executable in /proj, /users, or /share, that is accessible to the blob store. If given as an argument to the tb-set-node-service command, this *replaces* the Emulab configuration script. If given as an argument to the tb-add-node-service-hook command, it is run either before the Emulab configuration script associated with this service. This is implemented via the Emulab blob store, and when the experiment is swapped in, blobs for the specified files are created. These blobs are removed when the experiment is swapped out (of course, the executables themselves are not removed!). The files must exist when the experiment is created or modified. You cannot specify arguments to service replacement scripts; they are passed a single argument specifying which operation the script should be performing (see below in the description of the -op option for more details; this is important.

• -scriptblob: This is essentially the same as -script, except that you should specify the UUID of an existing blob that you have already created.

• -fatal: If set to 1, a script or hook that exits with non-zero return code is fatal to the configuration process (i.e., configuration will halt and the node will report a TBFAILED state to Emulab, which will likely cause an experiment swapin or modify to fail!). If set to 0, the script or hook that exits non-zero will not fail the configuration process. Defaults to 1.

There are a couple options specific to tb-set-node-service:

• -enable: This enables (1) or disables (0) the service. Note that disabling a script does *not* disable its hooks! Defaults to 1.

• -enablehooks: This enables (1) or disables (0) any hooks that might be associated with this service.

There are several options specific to tb-add-node-service-hook:

• -op: Specifies which operation this hook should be run for. The Emulab client side framework specifies four times at which configuration scripts and their hooks can be run: boot, shutdown, reconfig, reset. "boot" is the obvious time (each time the node boots into its OS). "shutdown" is also the obvious time (each time the node shuts down its OS). "reconfig" occurs when a node reconfigured without being rebooted. "reset" will happen to clean off any configuration in the node's filesystem; i.e., when the node is shutting down prior to being imaged. This option defaults to "boot".

• -point: Specifies when the hook should be run. If set to "pre", the hook will run *before* the service configuration script. If set to "post", the hook will run *after* the service configuration script. This option defaults to "post".

• -argv: Specify arguments to your hook. Make sure to escape them appropriately. This entire string becomes the command line for your hook script. Note that the operation is always appended to the arguments (i.e., "/bin/sh /path/to/script $argv boot").

Debugging Your Hook Script

The output of your script is sent to the console log and /var/emulab/logs/bootsetup.debug at the point when the script is run. To access this information, your experiment must swapin successfully. This means that you need to set fatal to 0 when debugging.

If your script fails to run at all, verify that your script exists on the ops filesystem, that the script has global read permissions, and that the directory it is in also has global read permissions.

Example NS File

Here is an example NS file used to add a hook to the rc.isup service. This invokes a 'timetest' script which checks to ensure that the system clock is synchronized within a margin or error to the clock on ops:

    set ns [new Simulator]
    source tb_compat.tcl
    set NodeA [$ns node]
    tb-set-node-os $NodeA UBUNTU16-64-STD
    tb-add-node-service-hook "rc.isup" -script "/proj/tbres/duerig/timetest.pl" -op boot -whence every -env boot -point post -fatal 1
    # Go!
    $ns run

This script will be run every time the node boots and gets to the isup script which is after most node setup is complete. Since fatal is on, the node will fail in booting if the test shows the clock to be too far out of synch. The script expects arguments of the form 'timetest.pl [accuracy] boot' where accuracy bounds the margin of error in seconds and defaults to half a second.

Customizing Individual Services

A few services support user customization.  rc.ifconfig, the service that configures your experiment network interfaces as you requested in your NS file or RSpec, can be customized to ignore specific network interfaces you wish to configure yourself.  rc.hostnames, the service that creates a custom /etc/hosts file with experiment-specific node names and other useful aliases, allows you to prepend and/or append a custom set of names each time it runs.

rc.ifconfig

At boot, rc.ifconfig reads the tmcc information (Emulab configuration for your experiment) for experiment network interfaces.  This information can be found cached in /var/emulab/boot/tmcc/ifconfig, or you can retrieve it by calling /usr/local/etc/emulab/tmcc ifconfig if it has not been retrieved and cached yet.  Here's a simple example of that file for a node in an experiment with two 10GbE LANs:

INTERFACE IFACETYPE=ixl INET=10.11.10.1 MASK=255.255.0.0 MAC=3cfdfe05a180 SPEED=10000Mbps DUPLEX=full IFACE= RTABID=0 LAN=flat-lan-1
INTERFACE IFACETYPE=ixl INET=10.12.10.1 MASK=255.255.0.0 MAC=3cfdfe05a182 SPEED=10000Mbps DUPLEX=full IFACE= RTABID=0 LAN=capnetlan-1

Each line begins with INTERFACE, and specifies many obvious fields.  You can use this information to configure your network interface as you would like.

On boot, to apply configuration before Emulab's rc.ifconfig does, you have two options.  First (and you can't do this on first boot obviously), you can place static configuration into your OS's network config files (i.e., /etc/network/interfaces on Debian/Ubuntu).  Second, you can add a pre-hook script to rc.ifconfig that configures the interfaces you care about, and then notifies rc.ifconfig that those interfaces are already configured.

You notify rc.ifconfig by writing a special file into /var/run/emulab.  Its name should be, for a given interface, interface-done-<MAC_ADDRESS>.  It can be empty; or it could have a single line with three space- or tab- separated values: <INTERFACE_NAME> <IPADDR> <MACADDR>.  If this information is present, rc.ifconfig will use it to fill out other files in /var/emulab/boot that it would normally fill out as part of its normal process.  You can place these files into /var/run/emulab; note that this dir may not exist when you try to write the file, so always try to create it if it does not yet exist --- this is because on modern Linux and FreeBSD systems, /var/run is a temporary filesystem whose contents are not preserved across reboot.

For instance, for an OpenStack experiment, I have a large script that gathers information about the network interfaces in the experiment, and moves their configuration into openvswitch bridges, and statically configures them in /etc/network/interfaces --- so that they can be used by the OpenStack software as members of virtual networks.  This is a two-part configuration.  Openvswitch remembers the creation of the bridge, and the fact that the physical interface was moved into it, and replays that configuration on each boot.  Then, the static configuration information in /etc/network/interfaces actually handles the IP configuration, and informs the Emulab rc.ifconfig script that it should ignore these interfaces.  So, in the first of my script, I run commands like

ifconfig enp4s0f1 0
ifconfig enp6s0f1 0
ovs-vsctl add-br br-flat-lan-1
ovs-vsctl add-port br-flat-lan-1 enp4s0f1
ovs-vsctl add-br br-capnetlan-1
ovs-vsctl add-port br-capnetlan-1 enp61s0f1

(assuming that enp4s0f1 is the device identified by the MAC 3cfdfe05a180 in the first line of the ifconfig tmcc info shown above, and enp6s0f1 is the device corresponding to the MAC in the second line)

That gets the interfaces into the bridge and removes their Emulab-configured IP configuration.  Then, I move the IP configuration to the bridge devices, something like this:

ifconfig br-flat-lan-1 10.11.10.1/16 up
ifconfig br-capnetlan-1 10.12.10.1/16 up

Finally, I write the following information into /etc/network/interfaces in the Debian style:

auto br-flat-lan-1
iface br-flat-lan-1 inet static
    address 10.11.10.2
    netmask 255.255.0.0
    up mkdir -p /var/run/emulab
    up echo "br-flat-lan-1 10.11.10.2 3cfdfe005a180" > /var/run/emulab/interface-done-3cfdfe05a180

auto enp4s0f1
iface enp4s0f1 inet static
    address 0.0.0.0

auto br-capnetlan-1
iface br-capnetlan-1 inet static
    address 10.11.10.2
    netmask 255.255.0.0
    up mkdir -p /var/run/emulab
    up echo "br-capnetlan-1 10.12.10.2 3cfdfe005a182" > /var/run/emulab/interface-done-3cfdfe05a182

auto enp6s0f1
iface enp6s0f1 inet static
    address 0.0.0.0

 Then, when the node is rebooted, the Debian/Ubuntu ifup scripts that process /etc/network/interfaces will bring the interfaces up according to the static configuration, but they will also run the additional commands that write specific information into /var/run/emulab .  This lets rc.ifconfig know that we have already configured some of the interfaces; recall, rc.ifconfig runs after OS network initialization.  It also allows us to tell rc.ifconfig the name of the interface, so it can do other normal operations on that interface.

rc.hostnames

rc.hostnames regenerates /etc/hosts on every boot, to ensure Emulab-specific node names are present.  It also checks for hosts.head and hosts.tail files in several directories, both static and dynamic (/etc, /etc/emulab/run, and /var/run/emulab); and prepends the hosts.head and appends hosts.tail to the normally-generated /etc/hosts content.  If multiple hosts.head and hosts.tail files exist, each in separate directories, they will all be read and prepended/appended.  /etc and /etc/emulab/run are static directories, so if you know you want specific names or aliases for a specific image, you can place them there.  If you place hosts.head or hosts.tail in /var/run/emulab, be aware that those files will be destroyed on reboot.  Thus, files in /var/run/emulab are only useful if they are generated via a pre-hook script to rc.hostnames.

For instance, in my OpenStack setup scripts, I create a simple, embedded pre-hook script (see the next section on embedding service control configurations) for rc.hostnames that creates /var/run/emulab/hosts.head --- and thus my hostnames take precedence over the Emulab-generated hostnames, which is what I require.  However, it also preserves specific Emulab hostnames that are necessary for some important Emulab services to function.  Look at the example in the next section.

Embedding Service Control Configurations Into Images

Sometimes, you'll know prior to experiment swapin that a particular image will need a special kind of custom configuration, and rather than force users to specify that configuration --- you will want to include it in the image.  You can do this by writing a manifest file into one of two directories: /var/run/emulab/rcmanifest.d or /etc/emulab/run/rcmanifest.d .  The latter is a static location that will be preserved if you capture an image; the former is a dynamic location that is wiped every boot.  In all likelihood, you will use the static location, unless you co-opt an earlier-in-boot mechanism (such as a udev event) that writes such a file before the Emulab scripts begin execution at boot.  Any manifest files must be present prior to Emulab rc script execution!

Normally, in the case where hook files are specified in your NS file, that information is fetched as one of the first steps in the Emulab rc boot process, and affects the remainder of the rc script boot process.  If you specify manifest information in the filesystem, those files take the same format as the information fetched from Emulab's control servers.  Here's a brief example:

root@cp-1:~# cat /etc/emulab/run/rcmanifest.d/0.openstack-rcmanifest.sh
HOOK SERVICE=rc.hostnames ENV=boot WHENCE=every OP=boot POINT=pre FATAL=0 FILE=/root/setup/bin/rc.hostnames-openstack ARGV=""

This specifies a hook for the rc.hostnames service, that executes on every boot, prior to the core of rc.hostnames, and is non-fatal (i.e., if the hook returns nonzero, the rc process will not abort).  The hook script file itself is /root/setup/bin/rc.hostnames-openstack .  Needless to say, all these files are executed as the root user; they are performing system configuration.

Here's an example of when I would use the technique of creating an embedded manifest.  In my complex Cloudlab profile that configures an OpenStack cluster in an experiment, because the default setup of OpenStack uses openvswitch or linux bridges, I have to take the Emulab experiment network configuration and morph it into an OVS or Linux bridged configuration.  I require that if the OpenStack physical nodes are rebooted, the network configuration comes up immediately, prior to other OpenStack services.  Emulab's configuration happens after the core OS's network configuration, so I must use the OS's default network configuration mechanisms to express my custom configuration.  I don't need a custom service configuration that expresses service hooks to handle network configuration, as I described above in the rc.ifconfig example --- but I do need it for handling rc.hostnames.  Thus, I created the manifest file shown above, and a simple script that places my special hostnames into /var/emulab/run/hosts.head, so that my names take priority over the Emulab-generated names --- and so that my names are always present (i.e. not overwritten by Emulab-generated names each boot).

Administration

Emulab administrators can add or remove controllable service tuples from the database, or place limits on controllability of controllable services.

To add or modify a service tuple so that users can control the service, add or update a row in the client_services table. The service, env, and whence fields mean the same thing as described above. The hooks_only field specifies that users can only modify hooks for the service, not the service itself.

Administrators can also add their own hooks, service replacement scripts, or service controls in a manner similar to the user functionality described above, by adding rows to the client_service_ctl or client_service_hooks tables. Each row applies to one of three kinds of Emulab system objects: a node_type, a particular physical node, or an OSID. For service controls (since only a single service control can apply to a node per service/env/whence tuple), if more than one control matches a particular node, the highest-priority control is applied (order of priority from highest to lowest is node, OSID, node_type). Finally, administrators can specify whether a user control (specified for an experiment with the tb-set-node-service command) can override their control setting, by setting the user_can_override field to 0.

If the administrator sets the user_can_override field to 0 for the client_service_hooks table, user attempts to disable administrator hooks of this type will fail during experiment runtime.