Skip to content. | Skip to navigation

Personal tools

Navigation

You are here: Home / Wiki / Kb101

Kb101

Emulab FAQ: Troubleshooting: I am running into errors while working in my home and/or project directory (or subdirectories). What's going on?

Emulab FAQ: Troubleshooting: I am running into errors while working in my home and/or project directory (or subdirectories). What's going on?

You may have been bitten by undesireable behavior present in the way FreeBSD handles NFS mount permissions updates. Each time Emulab swaps an experiment in or out, it must modify the set of allowed mounts on the Emulab fileserver to match the permissions required by the current set of experiments. This means adding mount permissions for nodes swapping in as part of an experiment, or removing permissions for nodes belonging to an experiment that is swapping out.

Unfortunately, the way that the mount daemon (mountd) and the FreeBSD kernel currently perform this update involves momentarily clearing the entire mount permissions table, followed by installing the new, updated one. What happens to external users during that update time period depends on the version of FreeBSD that is running. Prior to FreeBSD 9 (i.e., most current Emulab installations), if you are interacting with files that happen to be located on an NFS mounted filesystem exported from the Emulab fileserver (e.g., files on /users or /proj), the lapse in permissions will result in errors such as "permission denied", "stale NFS handle", and "No such file or directory." Starting with FreeBSD 9, mountd will suspend the NFS server while it is doing the update. To the user, this will appears as a slight delay when accessing a file, rather than an error.

For those working on an older installation, there are a few things you can do to minimize the impact of this problem. Most importantly, don't make a habit of running processes (especially long-running ones) that use files on NFS mounted filesystems. The best approach is to make a local copy of the things you need and operate on that local copy. This also helps cut down on the fileserver load. There are several tools to help make this easier:

  • Use the Emulab tarball feature to lay down your files.
  • Pull back changes with the loghole facility.
  • Use rsync to pull/push over your files. Most Emulab nodes have an empty fourth disk partition you can setup and use if you need more space to work in. Follow this link for more information.
  • See also this link for general information about where to place your data for experiments in Emulab.