Ukimage

Stuff I did to make a node image for University of Kentucky and Georgia Tech. Much of this overlaps with local image creation (see newimage.txt).

Start with one of our freshly loaded images.

A. FreeBSD (while running single user):

* Make sure local filesystems are mounted:
  mount -at ufs

* /root/.cvsup/auth
  Customize host/domain, change the password.  Then run "cvpasswd" giving the
  password just used.  Put the line spit out by cvpasswd in the boss node's
  /usr/testbed/sup/cvsupd.passwd file.

* /root/.ssh
  Remove known_hosts if it exists.  Put in local boss root pub key.
  Leave in our pub key if acceptible.

* /etc/localtime
  Copy the correct file over from /usr/share/zoneinfo

* /usr/local/etc/emulab/master.passwd
  Change the root password, this file will get installed by prepare.

* /etc/ssh/ssh_host*
  Generate new host keys.  Actually, copy from an existing image if
  available (i.e., we use a single host key across all images and OSes
  within a testbed).

* /etc/emulab/{client,emulab}.pem
  Generate new ones.  This needs to be done on the site's boss node.
  Go into the source tree "ssl" subdirectory and edit the *.cnf.in files
  to update the "[ req_distinguished_name ]" section with the appropriate
  country, state, city, etc.  Then do a "gmake boss-installX" which generates
  the certs and installs the server-side.  Grab the emulab.pem and client.pem
  from that directory to put in the images.

* Unmount filesystems, and remount root read-only to be safe:
  cd /
  umount -h <fs node name>
  umount -at ufs
  mount -u -o ro /


B. While running Linux (single user):

* /root/.cvsup/auth
  Customize host/domain, leave password alone?

* /root/.ssh
  Remove known_hosts if it exists.  Put in local boss root pub key.
  Leave in our pub key if acceptible.

* /etc/localtime
  Copy the correct file over from /usr/share/zoneinfo

* /etc/ssh/ssh_host*
  Generate new host keys.  Actually, copy from an existing image if
  available (i.e., we use a single host key across all images and OSes
  within a testbed).

* /etc/testbed/shadow
  Change the root password, this file will get installed by prepare.

* /etc/testbed/{client,emulab}.pem
  Generate new ones.  This needs to be done on the site's boss node.
  Go into the source tree "ssl" subdirectory and edit the *.cnf.in files
  to update the "[ req_distinguished_name ]" section with the appropriate
  country, state, city, etc.  Then go to the build directory and do a
  "gmake boss-installX" which generates the certs and installs the
  server-side.  Grab the emulab.pem and localhost.pem from that directory
  to put in the images as emulab.pem and client.pem.  [ NOTE: we can
  get by without the certs if the client tmcc and server tmcd are built
  without SSL support (tmcc-nossl and tmcd-nossl targets).

* Remount root filesystem read-only (IMPORTANT!)
  cd /
  mount -o remount,ro /

* Fsck it for good luck.  Actually, not only good luck but also resets
  some time stamp that forces an fsck periodically
  fsck -f <root device>

D. Frisbee MFS (boss:/tftpboot/frisbee/boot/mfsroot):

E. FreeBSD MFS (boss:/tftpboot/boot/mfsroot):

F. Dealing with SCSI disks (or RAID).

* Since we have IDE disks, it takes a little more work to create an
  image on IDE disks for a site with SCSI disks.

* For FreeBSD, you need to make sure that the kernel includes support
  for the host adaptor in question.  You can look at output from the
  "newnode" (aka GENERIC) kernel if necessary.  If the HA is not already
  supported in our various TESTBED configs, add it and rebuild and install
  the kernels in the image.  Then, in the image, you need to change /etc/fstab
  to refer to "da" disks rather than "ad" disks.

* For Linux, it is even more of a PITA.  As with BSD, you need to configure
  our kernel with generic SCSI support and support for the HA.  Note that
  by default, Linux includes SCSI as a module, but to load a module into the
  Linux kernel at boot time, you need to have a little MFS (initrd) that
  contains the appropriate modules.  We just build a kernel with the SCSI
  stuff hardwired instead.  Anyway, once you have a kernel, modify /etc/fstab
  to talk about "sda" rather than "hda".  Now you also have to modify
  /etc/lilo.conf, again, to reference "sda" rather than "hda".  But, once
  you have done that, you cannot rerun lilo as it will complain about not
  having an sda disk; i.e., lilo looks at the config file to figure out
  what disk to write the new stuff to, and that file now says "sda" instead
  of the "hda" that you are constructing the image on!  Go ahead and create
  the image without re-running lilo.  When you boot Linux from the image for
  the first time, interrupt lilo (type anything once lilo starts loading the
  image) and then type:

	 linux root=/dev/sda2

  (assuming Linux is in partition 2 on the disk).  This will boot it up
  using SCSI as the root disk.  Now, assuming your kernel has the appropriate
  SCSI support, the machine will come up ok.  Then login as root and run
  "lilo" to fix things up for good (remember to create a new image afterward).