Kb198

Emulab FAQ: Testbed Operations: Questions about node booting in Emulab.

> 1) After I "nfree" a hwdown node, does the node boot to
> /tftpboot/frisbee or the FBSD410+RHL90-STD image or the node boots
> frisbee which loads the FBSD410+RHL90-STD images?  The last time I
> tried nfree a hwdown node, it booted to /tftpboot/firsbee.  How do I
> get the nodes to boot the "real" image?

After the nfree, an Emulab daemon will arrange to reload the node to make sure it is "clean". That always happens after an nfree.

Once that is done, the node will reboot into a PXE loaded boot program that waits for Emulab to tell it what to do next. Typically, that happens when the node is allocated to an experiment, at which time the node will boot an OS on the disk, or arrange to load a custom OS if the user wants an OS that is not currently loaded.

If you want to boot an image, then you have to create an experiment to allocate the node.

> 2) In BIOS, do I always set PXE as first boot device? If an node
> already has an image loaded on its disk and the user types "reboot" at
> prompt, wouldn't the node network boot and reload a fresh image
> (overriding what was on disk)?

Yes, the node must PXE boot first. Emulab tells the node what to do after that.

> 3) Do I need to partition the disk before imageunzip the generic image
> onto the disk? Do I need to manually partition all nodes' disks?

No, you do not need to do anything. The generic image you get from us already has the MBR set up.

> When I boot the Generic image from disk, I get to the following menu:
> F1: FreeBSD
> F2: Linux
> F3: ??
>
> PXE Failed: _
>

This message comes from the special boot program we put in the MBR of our images. It usually does mean that the PXE boot failed and it has fallen back to booting from disk. But it can also mean that it didn't even try to boot from the network and is just booting straight from disk (e.g., the BIOS is misconfigured).

> Do I need to prepare my harddisk before adding it to emulab?
> I am asking this because my delay node was booting GRUB that I had
> previously installed on disk.

This is totally bizarre - I don't know how that stuff is getting left on your disk. These machines only have a single disk, right?

Ah, wait, I think I know what's going on. For whatever, reason, you've got single-slice images, meaning that they only load into one parition on the disk. So, they don't overwrite the MBR, leaving your GRUB from before around.

Did Mike leave you with instructions on how to make a full-disk image?

> I did a fdisk yesterday and imageunzip the RHL+FBSD image onto it.
> Now my delay node boots to the menu:
> F1: FreeBSD
> F2: Linux
> F3: ??
>
> Pxe failed: _

This is expected - we have a special little bootloader that we put into the MBR that basically just reboots. This way, if PXE fails, we reboot and try again.

> I was wondering, maybe I need to prepare my disk before adding it to
> emulab.  What I found was that emulab doesn't actually repartition the
> harddisk.  What was previously on the disk was still on disk.  The
> GRUB menu, Fedora, and Ubuntu was still on disk after a node has been
> allocated to an experiment.

Yeah, I think the thing we need to do is make you a full-disk image, and then I think we'll be okay.

> CLIENT MAC ADDR: 00 0E 0C0 68 90 79  GUID: 42CCBA8F 2A47 ....
> CLIENT IP: 198.162.51.12  MASK: 255.255.255.0  DHCP IP: 198.162.51.253
> GATEWAY IP: 198.162.51.254
> PXE LOADER 1.00

Okay, this makes sense - in bootinfo.log on boss, I see:

  Jul 27 12:00:50 boss bootinfo[240]: 198.162.51.12: REQUEST (vers 1)
  Jul 27 12:00:50 boss bootinfo[240]: 198.162.51.12: REPLY: boot from mfs 198.162.51.253:/tftpboot/frisbee

So, it has correctly been told to boot the frisbee loader. The frisbee loader, BTW, is as tiny as possible, so it does not have an sshd in it. So it's expected that you won't be able to ssh in.

I expect that the rest of the boot messages, along with an error message, went to the serial console. (I assume "screen" means the VGA)