Skip to content. | Skip to navigation

Personal tools


You are here: Home / Wiki / Elabinelab


Emulab Tutorial - Emulab in Emulab

Emulab Tutorial - Emulab in Emulab

Emulab in Emulab (henceforth known as "ElabInElab") is a new feature that allows the creation of an entire (inner) testbed as an experiment running on the (outer) testbed. The "inner" Emulab is functionally equivalent to the "outer" Emulab, except for how it interacts with certain aspects of the hardware infrastructure (switches, power controllers, etc), which must be mediated by the outer Emulab to avoid improper access to devices that an experiment is not normally allowed to access directly. For example, in order for an inner Emulab to power cycle a node, it must ask the outer Emulab to do it via a proxy that ensures that the node in question is actually part of the inner Emulab (a node in the experiment that comprises the inner Emulab).
ElabInElab serves several purposes:

  • Can be used to provide an isolated environment (in conjunction with firewalling) for running "dangerous" experiments that include the use of worms and other malware. Instead of running the experiment on the outer Emulab, the experiment is run on the inner Emulab, and thus has access to all of the Emulab's services, but in a context that does not put the outer Emulab at risk from attack.
  • Allows testing and development of Emulab itself in a controlled environment, without needing a dedicated testbed. New features can be tested without affecting users of the main testbed. In fact, multiple independent inner Emulabs can be constructed with each one being used for the testing and development of different features.

There are a few things to keep in mind about ElabInElab:

  • While it is tempting to think of ElabInElab as "recursive", it should be noted that real recursion is not supported; you cannot create an inner Emulab inside of an inner Emulab.
  • The inner emulab has its own "boss" and "users" nodes, its own web server, its own file server, etc.
  • From the outer Emulab's perspective, all of the nodes that make up the inner emulab are simply nodes in an experiment.
  • All of the nodes consume one of their experimental network interfaces to use for the innner Emulab "control" network. Therefore, inner experimental nodes have one fewer experimental interface to use in experiments.

Here is a simple example that sets up a tiny ElabInElab experiment, with just a single inner experimental node. For the purposes of this discussion, the project is "testbed" and the experiment is called "myemulab."

	source tb_compat.tcl
	set ns [new Simulator]

	tb-elab-in-elab 1

	namespace eval TBCOMPAT {
	    set elabinelab_maxpcs 1

	$ns run

which is instantiated as shown in the visualization in Figure 1.


Figure 1

As you can see in Figure 1, most of the details are handled for you; the experiment includes a boss node, an ops node, and a single pc600, all of which are connected via a lan. (You may specify 0 to elabinelab_maxpcs to get only an inner boss and ops, or a positive integer for multiple experimental pc nodes.)

Once the experiment swaps in, you can log into, or you can log into the web server at There is also a single experimental node that can be used to create an experiment. In all aspects, the inner Emulab can be used the same way that the outer Emulab can be used.

Another example:

	source tb_compat.tcl
	set ns [new Simulator]

	tb-elab-in-elab 1
        tb-set-inner-elab-eid myexp
	$ns run

In this example, we have included a tb-set-inner-elab-eid directive, which says to automatically launch an experiment within the inner Emulab once it is set up. The "myexp" experiment must already exist in the same project; it must have already been created, but not swapped in. The system uses the NS file associated with the "myexp" experiment to construct an experiment on the inner Emulab and swap it in. You will be notified via email, first when the inner Emulab has been fully swapped in, and then again when the inner experiment has been swapped in. You can interact with the inner experiment normally, albeit from the inner boss (myboss) web interface and the inner users node (myops), or you can log into the inner experimental node (mypc1) directly.

You may specify a different Emulab source tarfile to be used in setting up the inner Elab, for example:

	namespace eval TBCOMPAT {
	    set elabinelab_source_tarfile "/proj/yourpid/emulab-src.tar.gz"

This source tarfile is created in your object tree (preferably one that is pure, without any other hacks.) Run "make elabinelab" and it will create the tarfile for you.

Or if you want to check out a specific tag of the Emulab source code:

	tb-set-elabinelab-cvstag SomeTag

which will generate a checkout from the CVS repository.

You may specify tarfiles to modify your inner inner boss and ops, for example:

  	namespace eval TBCOMPAT {
  	    set elabinelab_tarfiles("boss") "/usr/site /proj/yourpid/patch.tar.gz"

You can specify multiple tarfiles in the string, just as tb-set-node-tarfiles allows.

Using similar sytax, there are also set elabinelab_ versions of some of the tb-set- commands that control how the inner boss, ops, and experimental nodes are set up:

  • elabinelab_nodeos - Choose a node OS ID, similar to tb-set-node-os .
  • elabinelab_hardware - Choose a node hardware type, similar to tb-set-hardware .
  • elabinelab_fixnodes - Choose an exact node, similar to tb-fix-node .

Implementation Notes

Our goal was to make the inner Emulab look as much like a real Emulab as possible. To do that, we decided to use one of the experimental interfaces on each node as the (inner) control network lan (see Figure 1). This lan connects all of the nodes (myboss, myops, and mypc1) in much the same way boss, ops, and experimental nodes are attached in the outer Emulab. The main difference is that there are no firewalls or subnets on the inner control network since the security concerns are not as strict; breaking into the inner boss is not going to do any more damage then does having root on any experimental node within the (outer) testbed.

The (inner) control network is used for all of the same traffic that nodes in the outer Emulab would; DHCP traffic, multicast disk reloading, experiment setup, etc. While each node still has its outer control network interface, that interface is not even configured, except on inner boss and inner ops.

Since we want to be able to create experiments on the inner Emulab normally, it is also necessary to "proxy" access to the Emulab infrastructure. For example, when setting up an inner experiment, a number of vlans will need to be created on the switches. Obviously, we cannot let the inner boss access the switches (or any other protected resources) directly; it must be mediated via a proxy on the outer boss. The proxy on the outer boss checks to make sure that the actions are allowed (and make sense), and then proceeds to do them itself.

Setup time: Largely due to dynamic construction of extensive parts of the inner Emulab environment, it currently takes about 20 minutes to set up ElabinElab on pc850s. In the future we will reduce this time considerably by more caching of pre-built components. Some faster nodes (2 and 3 GHz) will soon be available, which will also help.