Kb81

Emulab FAQ: Using the Testbed: How much permanent disk is available for my files?

Filesystems:

Emulab provides a reasonable amount of disk space for experimenters to store file. This space is physically located on the "fs" machine (e.g., fs.emulab.net at the Utah Emulab) and is always accessible from the "users" machine (which is usually the same machine as "fs"). There are several directory trees you have access to. Remember, all of these directories can be accessed on your experimental nodes via NFS as well.

/proj: Every project gets a sub directory in /proj. This is the primary location in which you should store files needed by your experiments and/or results from your experiments.
/groups: Every project subgroup gets a sub directory in /groups. If you have files that need to be restricted to just your subgroup members, place them here.
/users: Every user gets a home directory in /users. Your home directory is not the place to store large quantities of project files. A user's quota on /users is typically small, on the order of hundreds of MBs.
/scratch: (Optional FS) Every project gets a sub directory in /scratch. Disk quotas are higher on this filesystem, but the space is not backed up to tape. This directory is intended for large, short-term storage. NOTE: Prior to January 2015, the Utah Emulab had a /scratch filesystem, but that is no longer available. If you had files there, you can retrieve them by logging into users.emulab.net and accessing them via /oldops/scratch. This availability is temporary, so please copy the files to /proj instead.

Quotas:

NOTE: as of January 2015, the Utah Emulab has switched to using ZFS for its filesystems. This affects the way that quotas are enforced. Skip to the ZFS Quotas section below if you are a Utah Emulab user.

Since we have many experimenters, all wanting more disk space then we physically have, we run quotas on all filesystems to prevent users from filling up the disks. To see how big your quota is on each filesystem, simply run the quota command on users.emulab.net:

	users$ quota
	Disk quotas for user leebee (uid 30379): 
	     Filesystem   usage    quota     limit   grace   files   quota   limit   grace
	         /users      54    50000     60000              25       0       0
	             /q  967226  2000000   2500000           30348       0       0
                     /z      10 90000000 100000000               5       0       0

You will notice several important facts about the above quota display.

Your diskspace resides on users.emulab.net. To delete files when overquota, login to users.emulab.net. See this entry on remote access for how to login to an Emulab machine.
For any particular directory, you can use the Unix "du" command to locate subdirectories that are using the space. For example:

cd /users/foo
du -sk * .[a-zA-Z]*

Note the argument to catch files/subdirectories that start with '.', '*' does not match those!

If you are over quota on the filesystem containing /users, look in your home directory for files to remove. Do not remove the .ssh or .ssl directories; they contain Emulab-created credentials.
If you are over quota on the filesystem containing /proj, look for files you own in /proj/project, for every Emulab project you are a member of. Do not remove anything from /proj/project/exp; it could cause experiment swapin failures.
If you are over quota on the filesystem containing /groups and it is distinct from the filesystem for /proj, look in /groups/project/group for every subgroup group of every Emulab project you are a member of.
If you are over quota on the filesystem containing /scratch and it is distinct from the other filesystems, look in /scratch/project for every Emulab project you are a member of.
Custom OS images are the most common source of excessive disk usage, they are stored in /proj/project/images or /groups/project/group/images.
If you are over quota, please first consider getting rid of files you do not need! Emulab is not intended for long-term storage of your stuff.
If you are over quota in your home directory and absolutely need to keep the files, consider making a /scratch/project/username directory for some project that you are a member of and moving files there.
Before removing files from one of the shared directories (i.e., in /proj, /groups, or /scratch) make sure you are the owner of the files, otherwise it won't do you any good!
Quotas are displayed in 1K byte blocks.
Quotas are per-user, not per-project or per-group. Just joining a new project or group isn't going to get you any more disk space!
You have a separate quota on each physical filesystem. Note however that this does not necessarily mean a separate quota for each of /users, /proj, /groups and /scratch.
You have a very small quota on /users, which is intended to encourage users to use the project disk space in /proj or /scratch.

If you find that your existing quota is not big enough for your needs, please send email to testbed-ops explaining your needs.

ZFS quotas:

On the Utah Emulab, we use per-user and per-project ZFS filesystems and quotas are enforced by setting the size of the filesystem. Thus, unlike traditional BSD UFS quotas, limits are no longer per-user. This has three important implications:

Any user that can write to a filesystem can fill it up. In particular, you can no longer get around quotas by writing files as "root" from experiment nodes. This was possible before because space usage was controlled by a per-user quota and root had no quota.
Space for a project under the /proj and /groups directories is per-project, so any user in the project can fill it up. This is especially important for subgroups in /groups. Since all subgroups in a project share the same total amount of space, someone in another subgroup of your project could eat up all the space even though they cannot access your subgroup directory.
On the plus side, if you are in multiple projects, you get a distinct quota for each project!

To see your user filesystem "quota" you can use the df command on your home directory:

	users$ df /users/leebee
	Filesystem     1K-blocks  Used   Avail   Capacity Mounted on
	z/users/leebee   1048576 26764 1021812       3%   /users/leebee

Ditto for seeing how much space is available in one of your projects:

	users$ df /proj/LeeBeeTest
	Filesystem        1K-blocks Used     Avail Capacity  Mounted on
	z/proj/LeeBeeTest 104857600   21 104857579     0%    /proj/LeeBeeTest

or subgroups of a project:

users$ df /groups/LeeBeeTest
       Filesystem          1K-blocks Used   Avail Capacity  Mounted on
       z/groups/LeeBeeTest   5242880   19 5242861     0%    /groups/LeeBeeTest

If you fill up one of your filesystems, here are some hints and useful facts.

Your diskspace resides on users.emulab.net. To delete files, login there. See this entry on remote access for how to login to an Emulab machine.
For any particular directory, you can use the Unix "du" command to locate subdirectories that are using the space. For example:

cd /users/foo
du -sk * .[a-zA-Z]*

Note the argument to catch files/subdirectories that start with '.', '*' does not match those!

If you are cleaning up your home directory, look in the /users/user filesystem for files to remove. Do not remove the .ssh or .ssl subdirectories; they contain Emulab-created credentials.
If you are cleaning up a particular project, look in the /proj/project filesystem. Do not remove anything from the exp subdirectory; it could cause experiment swapin failures.
If you are cleaning up a particular subgroup of a project, look in the /groups/project filesystem in the subgroup subdirectory. However, recall that all subgroups share the same filesystem, so there is no guarantee that you will be able to free up space; it could be someone in another subgroup that is using up space.
Custom OS images are the most common source of excessive disk usage in projects or subgroups, they are stored in /proj/project/images or /groups/project/group/images.
If you are out of space, please first consider getting rid of files you do not need rather than asking for a larger quota! Emulab is not intended for long-term storage of your stuff.
If you are out of space in your home directory and absolutely need to keep the files, consider making a /proj/project/username directory for some project that you are a member of and moving files there.
Your /users filesystem is relatively small, which is intended to encourage users to use the project disk space in /proj.

If you find that your existing quota is not big enough for your needs, please send email to testbed-ops explaining your needs.