Skip to content. | Skip to navigation

Personal tools


You are here: Home / Wiki / Firewall


Firewalled Experiment Support in Emulab

Firewalled Experiment Support in Emulab


Emulab allows the ability to setup a control-network firewall between an experiment and the outside world. This is not a firewall between nodes within an experiment, that is, the firewall is not part of your NS-specified network topology. The purpose of an Emulab firewall is to prevent experimental traffic from escaping, via the control network, to the Internet due to a mis-configured application or router within an experiment. Control net firewalls are also a key component of Emulab high-security experiment environments.

The firewall is implemented by allocating an additional node to the experiment. Through switch-enforced VLANs, the experiment is given its own private control network connecting all nodes in the experiment. Network traffic between experiment nodes and any hosts outside the experiment (Emulab infrastructure or Internet hosts in general) must pass through the firewall node. The firewall is setup as a filtering layer2 bridge using IPFW2 on FreeBSD and can be configured in a number of ways to block or allow certain types of traffic.

Swapin actions

When a firewalled experiment is swapped in, the firewall is setup and activated before any experiment nodes are allowed to setup. Currently this adds approximately five more minutes to swapin time, as the firewall node must first be loaded with the FBSD-IPFW2 disk image.

Swapout actions

When a firewalled experiment is swapped out, extra precautions are taken to ensure that the nodes are decontaminated before the firewall is taken down. The nodes, including the firewall itself, are scheduled to boot into a network-loaded, memory-based version of FreeBSD (the "MFS") which will run a script to zero all the known bootblocks on the disk. This prevents a node from accidentally booting from its disk prior to being reloaded.

The "boot to MFS" process is done not by issuing a reboot command or even by doing a power cycle, but rather by simultaneously powering off all nodes and then later powering them back on. Reboot is not used since the reboot command could be spoofed on a contaminated node. Power cycling is not used since cycling is normally performed in batches to avoid network (or power) overload on restart. Skewed reboots like this open a window of vulnerability where nodes rebooted later might be able, before they are rebooted, to spoof the reload server for nodes that have just rebooted. So we first turn everyone off so that later batched power-ons are safe.

Only when all nodes are known to be running in the MFS and their disks neutered, will the nodes be released to the reloading experiment. Any failure in the above process results in all nodes for the experiment being left powered off, with the control net disabled, and marked as "paniced" so that only a testbed admin can swap it out.


To add a firewall to an Emulab experiment, you specify a Firewall object in your NS file (there is currently no way to add a firewall via the Emulab client or experiment creation GUI):

	set fw [new Firewall $ns]
	$fw set-type <type>
	$fw set-style <style>

This tells Emulab to add a node called "fw" to your experiment, and provides a handle for adding rules to the firewall. The "type" of the firewall should be ipfw2-vlan if you want to use a FreeBSD-based firewall, or iptables-vlan to use a Linux-based firewall. Another type, ipfw, has been deprecated and should not be used. In the future, there may be additional types for different firewall implementations. The "style" of the firewall is one of:

  • open: A completely open firewall allowing all traffic. This gives you a hook for setting up custom firewall rules (below).
  • closed: A closed firewall allowing no communication with the outside world or other experiments within Emulab. Nodes can still communicate in a limited fashion with the Emulab infrastructure.
  • basic: A mostly closed firewall allowing only ssh connections from the outside world, and HTTP/HTTPS connections from inside (primarily for Windows Update on Windows XP nodes).

The firewall can be augmented with user-specified rules as well:

	$fw add-rule <rule string>

For an iptables-based firewall, rules for packets coming from outside the firewall are handled by a chain called OUTSIDE, while packets coming from inside the firewall are handled by a chain called INSIDE. Packets going to or coming from the firewall itself are handled by the INPUT and OUTPUT chains respectively.

FreeBSD-based firewalls only: In the basic form, rules are numbered starting at 100. Thus, rules are interpreted in the firewall in the order in which they appear in the NS file. The default rules for a firewall style have numbers either less than 100 or greater than 60000, so user-specified rules are interpreted ahead of all but the most important default rules. There is also a method allowing explicit numbering of rules:

	$fw add-numbered-rule <ruleno> <rule string>

allowing more precise control over their interpretation and allowing for linked rules. A complete example is shown below.

It should be noted that user-specified rules will only apply to IP traffic passing through the firewall. Non-IP traffic is not allowed through the firewall, and traffic to the firewall node itself is controlled by a different set of rules which cannot be overridden with this mechanism. For example, in a closed firewall that does not allow ssh, you can add a rule to allow ssh access from a particular machine to one of the firewalled nodes, but not to allow access to the firewall itself.


One should note carefully the following issues.

  • The firewall is just another node in the topology. It is setup just like all other nodes, this includes enabling accounts and NFS access. Thus anyone who can login to a node, can login to the firewall. This is not as bad as it may sound since the firewall node is subject to its own rules. Thus, in all but the open firewall style, the firewall node rejects all packets sent to it from the inside, and in the closed firewall, it rejects all traffic from the outside as well. However, even with the firewall rules, the nodes and the firewall do share common filesystems in /users and /proj. In the future, firewalls will have much more constrained access.
  • The firewall must run FreeBSD with IPFW or Linux using iptables. However, this is not fundamental to the design and we hope that the firewall syntax will be general enough to support other implementations.
  • The firewall rule syntax is fixed. In particular, it allows only fixed strings with no form of per-user or per-experiment variable substitution. There is a limited form of substitution used to plug in Emulab-wide parameters such as the Utah control network subnet. Since IPFW allows hostnames, you can at least use symbolic host names (i.e., the node names you use in your NS file) in place of IP addresses in your rules.
  • Firewall setup is static. Rules are specified in the NS file at experiment creation time. Rules can be changed using experiment modify, or by logging into the firewall and changing rules by hand with IPFW or iptables. A more dynamic interface may be desirable, in particular integration with the event system. On the other hand, it is not necessarily a good thing to have a firewall that is too easy to (mis)configure.
  • There is always a firewall for experiments. Keep in mind that Emulab has an external firewall already which imposes some site-wide restrictions (yes, there should be a link here...) So just allowing some ports through your experimental firewall doesn't guarantee that the affected traffic will make it to the outside world.

Known Bugs

There is a mighty fine line between a "limitation" and a "bug". But this one probably crosses the line:

  • The firewall exposes more of the infrastructure than it should. The Emulab node self-configuration and monitoring infrastructure uses a lot of different services on the boss and ops nodes. At the current time, we allow all those through. Moreover, many of these, such as TFTP, cannot be pinned down too precisely in firewall rules, so the rules are more open than we would prefer. Additionally, we also currently preserve the shared filesystem access model inside firewalled experiments, which permits not only attacks on NFS but allows for trojans to be placed in the filesystems.

For particularly anti-social applications, you should check out Emulab in Emulab.

A Complete Example

Here is the NS code for a simple two node experiment with a FreeBSD firewall:

	source tb_compat.tcl
	set ns [new Simulator]

	set n1 [$ns node]
	tb-set-node-os $n1 FBSD-STD
	set n2 [$ns node]
	tb-set-node-os $n2 RHL-STD
	set link [$ns duplex-link $n1 $n2 100Mb 0ms DropTail]

	# create a firewall node
	set fw [new Firewall $ns]
	$fw set-type ipfw2-vlan
	$fw set-style closed

	# allow traceroute through
	$fw add-rule "allow udp from EMULAB_CNET to any 33434-33524"
	$fw add-rule "allow udp from any 33434-33524 to EMULAB_CNET"

	# and ssh sessions from "users" to one of the nodes
	$fw add-rule "allow tcp from to n1 setup keep-state"

	$ns run

and here is the NS code for a simple two node experiment with a Linux firewall:

	source tb_compat.tcl
	set ns [new Simulator]

	set n1 [$ns node]
	tb-set-node-os $n1 FBSD-STD
	set n2 [$ns node]
	tb-set-node-os $n2 RHL-STD
	set link [$ns duplex-link $n1 $n2 100Mb 0ms DropTail]

	# create a firewall node
	set fw [new Firewall $ns]
	$fw set-type iptables-vlan
	$fw set-style closed

	# allow traceroute through
	$fw add-rule "iptables -A OUTSIDE -p udp -s EMULAB_CNET --dport 33434:33524 -j ACCEPT"
	$fw add-rule "iptables -A INSIDE -p udp --sport 33434:33524 -d EMULAB_CNET -j ACCEPT"

	# and ssh sessions from "users" to one of the nodes
	$fw add-rule "iptables -A OUTSIDE -p tcp -s -d n1 -y -m conntrack --ctstate NEW -j ACCEPT"

	$ns run

EMULAB_CNET is an example of the limited variable replacement capability. In the Utah case, it expands to "155.98.36/22". Note that even though this rule names the entire control net space of all experiments, it effects only those nodes within this experiment since the firewall is per-experiment. The last add-rule invocation demonstrates using a symbolic node name, "n1", to reference the node identified by the object "$n1" in the NS file. This rule also takes advantage of the dynamic rule capability in IPFW2. The rule itself only allows through TCP packets in the "setup" state (SYN but no ACK), but instructs IPFW to construct a limited lifetime dynamic rule allowing bidirectional traffic between the source and destination IP addresses and TCP ports from the packet.

This NS specification yields a topology that looks like:
No surprises. Two nodes connected by a link, with a "disconnected" firewall off to the side. What isn't shown is that all three are connected via the control net. The firewall is accessible via a DNS name of similarly to other nodes. You can login to the firewall (if allowed by the rules), reboot it, etc. just as any other node. The Experiment Details web page for the experiment lists all the rules for the firewall:

Firewall information:
ID              Type       Style    Rule# Rule
--------------- ---------- -------- ----- -----------------------------------
fw              ipfw2-vlan closed       4 check-state
                                        9 skipto 80 all from any to any layer2 in
                                       10 deny all from any to me via vlan0
                                       11 allow all from me to me
                                       20 allow udp from me to EMULAB_NS 53 keep-state
                                       22 allow tcp from boss to me 22 setup keep-state
                                       24 allow ip from me to ntp1,ntp2 123 keep-state
                                       26 allow udp from me 514 to ops 514
                                       30 allow ip from me to fs 111 keep-state
                                       31 allow udp from me not 0-700 to fs keep-state
                                       32 allow udp from me to fs 900 keep-state
                                       33 allow udp from me to fs 2049 keep-state
                                       34 allow ip from me to fs frag
                                       35 allow ip from fs to me frag
                                       36 allow tcp from me to boss 5999 setup keep-state
                                       38 allow ip from me to ops 2917 keep-state
                                       40 allow udp from me to boss 8509
                                       50 allow icmp from boss to me icmptypes 6,8
                                       51 allow icmp from me to boss icmptypes 0
                                       70 allow ip from me to boss 7777 keep-state
                                       79 deny all from any to any
                                       80 deny not mac-type ip
                                       83 allow ip from EMULAB_GWIP to any in not via vlan0
                                       84 deny ip from any to EMULAB_CNET in via vlan0
                                       85 deny ip from EMULAB_CNET to any in not via vlan0
                                       88 deny ip from not,,EMULAB_CNET to any in via vlan0
                                      100 allow udp from EMULAB_CNET to any 33434-33524
                                      101 allow udp from any 33434-33524 to EMULAB_CNET
                                      102 allow tcp from to n1 setup keep-state
                                    60020 allow udp from any to EMULAB_NS 53 keep-state
                                    60022 allow tcp from boss to any 22 setup keep-state
                                    60024 allow ip from any to ntp1,ntp2 123 keep-state
                                    60026 allow udp from any 514 to ops 514
                                    60030 allow ip from any to fs 111 keep-state
                                    60031 allow udp from any not 0-700 to fs keep-state
                                    60032 allow udp from any to fs 900 keep-state
                                    60033 allow udp from any to fs 2049 keep-state
                                    60034 allow ip from any to fs frag
                                    60035 allow ip from fs to any frag
                                    60036 allow tcp from any to boss 5999 setup keep-state
                                    60038 allow ip from any to ops 2917 keep-state
                                    60040 allow udp from any to boss 8509
                                    60046 allow udp from any to EMULAB_MCADDR EMULAB_MCPORT in via vlan0
                                    60047 allow udp from boss EMULAB_MCPORT to any EMULAB_MCPORT
                                    60048 allow igmp from any to any
                                    60050 allow icmp from boss to any icmptypes 6,8
                                    60051 allow icmp from any to boss icmptypes 0
                                    60064 allow udp from any 68 to 67 recv vlan0
                                    60065 allow udp from any 67 to any 68 in not recv vlan0
                                    60066 allow udp from any to boss,ops 69 keep-state
                                    60067 allow udp from boss,ops not 0-1023 to any not 0-1023 keep-state
                                    60068 allow udp from any 9696 to boss 6969 keep-state
                                    60069 allow udp from boss 6970 to any 9696
                                    60070 allow ip from any to boss 7777 keep-state
                                    65534 deny all from any to any

The rules from 10 to 79 are for the firewall itself, from 80 to 65534 are for bridged ("layer2") packets, i.e., those that pass through the firewall. As user rules start at 100, Again, notice the rules with EMULAB_ variables. NS is the Emulab name server IP address, CNET the control network subnet, MCADDR and MCPORT, the multicast addresses and ports used by the Frisbee disk loader. Other rules involving the hosts boss, ops, fs, ntp1, and ntp2, are Emulab infrastructure related.

The deprecated ipfw firewall

An earlier, less secure, firewall implementation did not require support from the switching infrastructure. This "software" firewall solution still allocated an extra node to act as an IP firewall. This node was then set as the default route for all other nodes in the experiment. Thus, all outgoing, non-experimental traffic was passed through the firewall node. Inbound traffic directed to the nodes did not pass through the firewall. So in addition to the limitations above you can add the following for this deprecated version:

  • The firewall was implemented using OS-provided routing. Specifically, every node has its default route changed to point to the firewall node. Sufficiently powerful applications could accidentally or intentionally change the default route back to the Emulab router, thus circumventing all protection.
  • Intra-Emulab traffic was not firewalled. Traffic between nodes over the control net (155.98.36.x addresses) is not filtered since the shared control net is a LAN and all other nodes are directly reachable. Additionally, traffic between the nodes and the Emulab infrastructure (boss and ops) does not pass through the firewall. Host routes are explicitly setup to avoid the firewall.