Emulab Tutorial - Setting up your own "delay" nodes

Typical Emulab Operation

The Emulab resource mapper will typically determine when you need a delay node (or nodes) in your topology, and automatically insert them for you. In most cases this works fine, but some users want more direct control, or want to set up their own delay pipes, but without having to set everything up from scratch.

Lets take a quick look at typical Emulab delay handling. Given a line in your NS file like:

	set link0 [$ns duplex-link $nodeA $nodeB 33Mb 15ms DropTail]

Emulab will see that you want a link with a delay in it, and insert a new node, called a delay node. At the physical level, your link actually looks like this:

               link0-A            link0-B
	NodeA <-------> tbdelay0 <-------> NodeB

where tbdelay0 is an Emulab node acting as a transparent ethernet bridge between the two physical links link0-A and link0-B. All packets received received on the left side of tbdelay0 are spit out on the right side, and all packets received on the right side are spit out on the left side.

In addition to delay properties, you may also set various queue properties. More info about Emulab delay node properties and queue characteristics, as well as link tracing, can be found in the Advanced Tutorial.

Inserting your own bridge nodes

When you insert your own bridge nodes, you are explicitly adding a node (or nodes) to your topology that will function as a bridge between two network segments. A bridge node can then act a delay node and/or a link tracing node. For example:

	set NodeA [$ns node]
	set NodeB [$ns bridge]
	set NodeC [$ns node]

	set link0 [$ns duplex-link $NodeA $NodeB * 0ms DropTail]
	set link1 [$ns duplex-link $NodeC $NodeB * 0ms DropTail]

The physical topology thus looks like:

                link0           link1
	NodeA <-------> NodeB <-------> NodeC

The difference is that NodeB booted as a bridge node, connecting link0 and link1 together. In the absence of any traffic shaping or link tracing directives in your NS file, packets are passed through the bridge unmolested.

Shaping your traffic using a bridge node

If you want your bridge node to shape your traffic as described in the first section of this document, you should set the delay parameters. Using the previous example:

	# Set the delay,bw,loss for link0 at the bridge.
	$link0 SetDelayParams $NodeB 10ms 33Mb 0.02
	# Set the delay,bw,loss for link1 at the bridge.
	$link1 SetDelayParams $NodeB 10ms 33Mb 0.02

The above statements set the shaping parameters for each of the two links attached to the bridge. In other words, the first statement sets the parameters for packets arriving from link0 at NodeB while the second statement sets the parameters for packets arriving from link1 at NodeB. The parameters may in fact be set differently in each direction.

Since this is essentially a duplex link between NodeA and NodeC, when you ping NodeC from NodeA, you will see ping times of 20ms, since each ping packet has to traverse the link in both directions, and is thus subject to a 10ms delay in each direction.

In addition to the basic shaping parameters delay, bandwidth, and loss rate, you may also set certain queue characteristics, as described in the Advanced Tutorial. The syntax to get the queue object is slightly different though:

	set queue0 [$link0 Queue $NodeB]
	$queue0 set limit_ 40
	set queue1 [$link1 Queue $NodeB]
	$queue1 set limit_ 30

The rest of the Queue characteristics described in the advanced tutorial can be set as well.

Checking the correctness of your shaping parameters

We urge all users to diligently check the correctness of their experimental setup. To verify the traffic shaping parameters on your bridge node, ssh into the node and do:

	nodeB> sudo ipfw pipe show

which will give you output like the following.

60111: 33.000 Mbit/s   10 ms   30 sl. 1 queues (1 buckets) droptail
    mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 udp          0.0.0.0/68    255.255.255.255/67       7     1084  0    0   0
60121: 33.000 Mbit/s   10 ms   40 sl. 1 queues (1 buckets) droptail
    mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip   205.110.109.156/0       193.86.216.24/6        5      414  0    0   0

The ipfw man page on your bridge node will contain a wealth on information about how to parse the above, but the simple stuff (bandwidth, delay, queue length, and lossrate) should be obvious.

In addition, you may also use Emulab's linktest support to verify that your experiment is setup properly.

Changing your traffic shaping parameters

There are several ways to change the traffic shaping the parameters on your bridge nodes:

• Log into your bridge node, read the ipfw man page, and then issue the appropriate ipfw commands. Not a very attractive option.
• Log into your bridge node and find the /var/emulab/boot/rc.delay shell script. Make a copy of that script, and edit as needed. This is only slightly more attractive then the previous method, but you can probably get away without reading the ipfw man page if you are not messing with any of the Queue parameters (you are just setting the delay, bandwidth, loss rate).
• Use the delay_config script on your "users" node. delay_config is a front end to the Emulab event system, as described in the Advanced Tutorial, but is somewhat easier to use.

We recommend the last option, for obvious reasons. One caveat, the syntax for using delay_config is slightly different when changing the shaping parameters on a bridge node. Using the example above, to change the shaping in both directions symmetrically:

	users> delay_config -b yourpid youreid NodeB DELAY=30

Note the -b option. This option is required when modifying the shaping on an explicit bridge; it tells delay_config you know what you are doing.

To change the shaping asymmetrically:

	users> delay_config -b -s link0 yourpid youreid NodeB DELAY=15
	users> delay_config -b -s link1 yourpid youreid NodeB DELAY=30

Note the -s option. This option indicates which side of the bridge (what direction) you want to apply the changes to.

Tracing your traffic using a bridge node

You may also make use of the link tracing mechanism described in the advanced tutorial. You can use link tracing in conjunction with traffic shaping, or by itself. Using the example above, to turn on tracing (in both directions):

	$link0 SetTraceParams $NodeB "header" 0 ""
	$link1 SetTraceParams $NodeB "header" 0 ""

which instructs the system to turn on packet tracing crossing the bridge in both directions. Additional options (trace type, snaplen, and trace expression) are as described in the advanced tutorial. Note that a snaplen of zero just means to use the default snaplen. (All of the options after the node are actually optional and will default as indicated above).

There are several other differences to be aware of. First, the log files that are store in /local/logs have a different naming scheme. In the example above, the four files that are created are:

• trace_NodeA-link0.xmit
• trace_NodeA-link0.recv
• trace_NodeC-link1.xmit
• trace_NodeC-link1.recv

where the .recv files hold the packets that were sent by the node and received by the bridge node. The .xmit files hold those packets that were transmitted by the delay node and received by the other other node. So, for packets sent from nodeA to nodeC, the packet would arrive at the bridge node and be recorded in trace_nodeA-link0.recv. Once the packet traverses the bridge node (subject to traffic shaping) and it is about to be transmitted, it is recorded in trace_nodeA-link0.xmit. By comparing these two files, you can see how the traffic shaping has affected your packets, in each direction.

Using bridge nodes on a Lan

You may also use bridge nodes in conjunction with a Lan, although this is a bit more complicated to set up. Lets look at the following NS file fragment:

	set NodeA   [$ns node]
	set NodeB   [$ns node]
	set NodeC   [$ns node]
	set bridge0 [$ns bridge]
	set bridge1 [$ns bridge]

	set link0 [$ns duplex-link $NodeA $bridge0 * 0ms DropTail]
	set link1 [$ns duplex-link $NodeB $bridge1 * 0ms DropTail]
	set lan0  [$ns make-lan "$bridge0 $bridge1 $NodeC" * 0ms]

	# Set the delay params for link0 at bridge0
	$link0 SetDelayParams $bridge0 10ms 100Mb 0
	# Set the delay params for lan0 at bridge0
	$lan0 SetDelayParams $bridge0 10ms 100Mb 0

	# Set the delay params for link1 at bridge1
	$link1 SetDelayParams $bridge1 20ms 100Mb 0
	# Set the delay params for lan0 at bridge1
	$lan0 SetDelayParams $bridge1 20ms 100Mb 0

This example constructs a lan of three nodes, two of which will be used as bridge nodes (delays) to other nodes. In other words, packets from NodeA and NodeB will transit through a bridge, while packets from NodeC will flow directly into the lan. You may set the shaping parameters in both directions, and while this is a powerful addition to Emulab, it can easily get you into trouble if you are not careful.

As with lans, you may also set the queue parameters on each outbound link into the lan.

	set queue0 [$link0 Queue $bridge0]
	$queue0 set limit_ 40
	set queue1 [$link1 Queue $bridge1]
	$queue1 set limit_ 30

And if you are really daring, you may also set them on the inbound side.

	set queue0 [$lan0 Queue $bridge0]
	$queue0 set limit_ 40
	set queue1 [$lan0 Queue $bridge1]
	$queue1 set limit_ 30

And as mentioned above, be sure to check your experimental setup to confirm that everything is as you think it should be (see Emulab's linktest support).

Caveats

• The current implementation supports chaining bridge nodes together, but we do not recommend it, and may in fact disappear, since it is very confusing and causes linktest to break.
• Be sure to test your setup with linktest; bridge support is a new feature and you want to be sure it is working as you expect
• Bridge nodes are mostly regular nodes in your topology. You may log into them, install software on them, reboot them, etc. However, they do not have any routes set up since they are intended to be invisible to the topology.