Kb199

Emulab FAQ: Testbed Operations: What happens when I free a node?

> I am having trouble freeing my nodes.
> After I type "nfree emulab-ops hwdown pc1" on boss, I get the
> following message "Moving pc1 to emulab-ops/reloadpending."  However,
> the node does not reboot nor end up in the free pool.  If I manually
> reboot the node, it boots into /tftpboot/frisbee but it's still not in
> the free pool.
>
> The node was fine when it's in the "hwdown" state.  I can login using
> ssh or at prompt.
>
> Do I need to do anything after I "nfree" a node?
> How do I add nodes to the free pool?

Normally when a node gets freed up, its disk gets reloaded. This takes a while (potentially 10 or more minutes), so one possibility is that you just didn't give it enough time.

Here's the usual procedure of what happens when you free a node:

• nfree moves it into emulab-ops/reloadpending
• A daemon on boss called reload_daemon periodically (every minute, I think) scans for nodes in reloadpending. When it finds one, it sets up the database state necessary to load the node, and moves it into emulab-ops/reloading
• reload_daemon reboots the node, which boots into the Frisbee MFS and loads the disk with frisbee
• The node reports that it's done with frisbee, and it gets removed from the reloading experiment (making it free)

So, the two places in this sequence I would check are:

1. Check the reloadlog in /usr/testbed/log on boss. Any errors encountered by the reload daemon should show up in here. eg., if it had any problem rebooting the node, you should see something.
2. When you got the node booted into the frisbee MFS, did it do anything? Normally, Frisbee prints out a message to the effect of "playing frisbee", then prints out a bunch of dots as it runs. If it didn't even print "playing frisbee", then there was some problem getting loading information from boss, or the frisbeed on boss was not running for some reason. If it printed that message, but no dots, you likely have a problem with multicast on your control network.