Disk space policy: Difference between revisions
No edit summary |
No edit summary |
||
| Line 1: | Line 1: | ||
This is the lab disk space policy and applies to those with ssh access to our cluster only. This policy applies to [[cluster 2]] | This is the lab disk space policy and applies to those with ssh access to our cluster only. This policy currently applies to [[cluster 2]]. It will apply to [[cluster 1]] effective April 1, 2014 and to [[cluster 0]] with caveats by May 1, 2014. | ||
{{TOCright}} | {{TOCright}} | ||
= | = Disk Types = | ||
We ask you to observe the following disk usage policy for your account on the cluster. We think this policy will cover 90% of the users, 90% of the time. We do understand that research can have unanticipated needs, and we are willing to work with you to create a plan that works for everyone. | |||
When you log in to a cluster | When you log in to a cluster computer, you are in your home directory. This would normally be /nfs/home/<userid>. Lab tradition is to have a ~/code/ directory for all software and a ~/work/ directory for all of your research. Your home directory is backed up weekly, and has a quota of 500 GB. Keeping the quota at this level will allow us to provide a high performance backup system that runs regularly and completes in a timely fashion. | ||
If you require more than your quota for ~/work/, we will create a special directory for you, e.g. /nfs/work/<userid>, which we suggest you symlink to ~/work/. /nfs/work will never be backed up. We recommend you repatriate key files from time to time from /nfs/work/<userid> to e.g. ~/worksave/. We can help you write scripts to automate this. | If you require more than your quota for ~/work/, we will create a special directory for you, e.g. /nfs/work/<userid>, which we suggest you symlink to ~/work/. /nfs/work will never be backed up. We recommend you repatriate key files from time to time from /nfs/work/<userid> to e.g. ~/worksave/. We can help you write scripts to automate this. You are also welcome to make your own supplementary backups to a USB drive on your workstation or laptop. We have USB drives you may have for this purpose. | ||
If you create database files that persist and are larger than your quota, ask us to create dedicated space for you, e.g. /nfs/db/<userid>. | If you create database files that persist and are larger than your quota, ask us to create dedicated space for you, e.g. /nfs/db/<userid>. The hallmarks of database files is that they are large, they are used by more than one person, and they are persistant. Database files are backed up quarterly beginning on the first day of the quarter. | ||
If you have large files from the synchrotron, ask us to create dedicated space for you, e.g. /nfs/store/<userid>. Storage files are backed up quarterly beginning on the first day of the quarter. | If you have large files from the synchrotron or other large data files, ask us to create dedicated space for you, e.g. /nfs/store/<userid>. Storage files are backed up quarterly beginning on the first day of the quarter. | ||
If after using /nfs/db, /nfs/work and /nfs/store to offload from /nfs/home/ you still need more space, please work with the [[sysadmin]]s to help you. | If after using /nfs/db, /nfs/work and /nfs/store to offload from /nfs/home/ you still need more space, please work with the [[sysadmin]]s to help you. | ||
When you log in to your workstation, you may be in a local directory on the workstation. Our policy is to never back up workstations. You may back up your own workstation, and you may make additional backups of your data on the cluster by writing to USB disks mounted on your desktop or laptop. We offer as a parting gift from the lab two multi-TB drives to which you may copy your files. Make one copy to take with you and one copy for us to keep safe for you. Then, delete the files from the server as you leave the lab, provided the project has been published. | When you log in to your workstation, you may be in a local directory on the workstation. Our policy is to never back up workstations. You may back up your own workstation, and you may make additional backups of your data on the cluster by writing to USB disks mounted on your desktop or laptop. We offer as a parting gift from the lab two multi-TB drives to which you may copy your files. Make one copy to take with you and one copy for us to keep safe for you. Label them clearly in ink. Then, delete the files from the server as you leave the lab, provided the project has been published. | ||
= Summary = | = Summary = | ||
We back up home directories weekly, and selected database, crystallography and other archived files quarterly. We do not back up /nfs/work files and we do not back up workstations or laptops. To backup work | We back up home directories weekly, and selected database, crystallography and other archived files quarterly. We do not back up /nfs/work files and we do not back up workstations or laptops. To backup files from /nfs/work, you must repatriate the key files to your home directory whence they will be backed up as part of normal procedures. To backup laptop and desktop, use USB disks, which we can give you. | ||
Our backup systems cannot keep up with the growth in disk space and the voracious appetite for disk of group members. This is a pragmatic policy that, while requiring a little organization, is more likely to deliver restored files you can actually rely on. | Our backup systems cannot keep up with the growth in disk space and the voracious appetite for disk of group members. This is a pragmatic policy that, while requiring a little organization, is more likely to deliver restored files you can actually rely on. | ||
As always, we welcome [[feedback]]. | As always, we welcome [[feedback]]. | ||
{| class="wikitable" | {| class="wikitable" | ||
Revision as of 19:46, 18 March 2014
This is the lab disk space policy and applies to those with ssh access to our cluster only. This policy currently applies to cluster 2. It will apply to cluster 1 effective April 1, 2014 and to cluster 0 with caveats by May 1, 2014.
Disk Types
We ask you to observe the following disk usage policy for your account on the cluster. We think this policy will cover 90% of the users, 90% of the time. We do understand that research can have unanticipated needs, and we are willing to work with you to create a plan that works for everyone.
When you log in to a cluster computer, you are in your home directory. This would normally be /nfs/home/<userid>. Lab tradition is to have a ~/code/ directory for all software and a ~/work/ directory for all of your research. Your home directory is backed up weekly, and has a quota of 500 GB. Keeping the quota at this level will allow us to provide a high performance backup system that runs regularly and completes in a timely fashion.
If you require more than your quota for ~/work/, we will create a special directory for you, e.g. /nfs/work/<userid>, which we suggest you symlink to ~/work/. /nfs/work will never be backed up. We recommend you repatriate key files from time to time from /nfs/work/<userid> to e.g. ~/worksave/. We can help you write scripts to automate this. You are also welcome to make your own supplementary backups to a USB drive on your workstation or laptop. We have USB drives you may have for this purpose.
If you create database files that persist and are larger than your quota, ask us to create dedicated space for you, e.g. /nfs/db/<userid>. The hallmarks of database files is that they are large, they are used by more than one person, and they are persistant. Database files are backed up quarterly beginning on the first day of the quarter.
If you have large files from the synchrotron or other large data files, ask us to create dedicated space for you, e.g. /nfs/store/<userid>. Storage files are backed up quarterly beginning on the first day of the quarter.
If after using /nfs/db, /nfs/work and /nfs/store to offload from /nfs/home/ you still need more space, please work with the sysadmins to help you.
When you log in to your workstation, you may be in a local directory on the workstation. Our policy is to never back up workstations. You may back up your own workstation, and you may make additional backups of your data on the cluster by writing to USB disks mounted on your desktop or laptop. We offer as a parting gift from the lab two multi-TB drives to which you may copy your files. Make one copy to take with you and one copy for us to keep safe for you. Label them clearly in ink. Then, delete the files from the server as you leave the lab, provided the project has been published.
Summary
We back up home directories weekly, and selected database, crystallography and other archived files quarterly. We do not back up /nfs/work files and we do not back up workstations or laptops. To backup files from /nfs/work, you must repatriate the key files to your home directory whence they will be backed up as part of normal procedures. To backup laptop and desktop, use USB disks, which we can give you.
Our backup systems cannot keep up with the growth in disk space and the voracious appetite for disk of group members. This is a pragmatic policy that, while requiring a little organization, is more likely to deliver restored files you can actually rely on.
As always, we welcome feedback.
| Name | Location | Quota | Backup? | Usage |
|---|---|---|---|---|
| Home | /nfs/home/<userid> | 500 GB | Weekly | General use - not large database and not docking runs. |
| Work | /nfs/work/<userid> | 2TB* | Never | Docking runs or calculations that produce large output |
| DB | /nfs/db/<userid> | 2TB* | Quarterly | Database files, often persistent, rarely modifed |
| Store | /nfs/store/<userid> | 2TB* | Quarterly | Crystallography files or other archived files, rarely modified. |
| Desktop | /home/<userid> | N/A | Never | Copy critical files from desktop to home directory on server. |
| Laptop | / | N/A | Never | Copy critical files from laptop to home directory on server. |
- More space may be requested from the sysadmins.