Disk space policy - Revision history

Frodo: asdf

2015-06-03T04:06:12Z

asdf

@@ Line 4: / Line 4: @@
 {{TOCright}}
 = Checking Disk Usage =
@@ Line 38: / Line 64: @@
 = Workstation and Laptop =
 When you log in to your workstation, you may be in a local directory on the workstation. Our policy is to never back up workstations. You may back up your own workstation, and you may make additional backups of your data on the cluster by writing to USB disks mounted on your desktop or laptop.  We offer as a parting gift from the lab two multi-TB drives to which you may copy your files.  Make one copy to take with you and one copy for us to keep safe for you. Label them clearly in ink. Then, delete the files from the server as you leave the lab, provided the project has been published.
 = See Also =

Frodo: /* Summary */ afsd

2015-06-03T04:05:23Z

Summary: afsd

← Older revision		Revision as of 04:05, 3 June 2015
Line 50:		Line 50:
	! Name !! Location !! Quota !! Backup? !! Usage		! Name !! Location !! Quota !! Backup? !! Usage
	\|-		\|-
	\| Home \|\| /nfs/home/<userid> \|\| 500 GB \|\| ~~Weekly~~ \|\| General use - not large database and not docking runs.		\| Home \|\| /nfs/home/<userid> \|\| 500 GB \|\| Near-daily \|\| General use - not large database and not docking runs.
	\|-		\|-
	\| Work \|\| /nfs/work/<userid> \|\| 500GB* \|\| Never \|\| Docking runs or calculations that produce large output		\| Work \|\| /nfs/work/<userid> \|\| 500GB* \|\| Never \|\| Docking runs or calculations that produce large output
	\|-		\|-
	\| DB \|\| /nfs/db/<userid> \|\| 2TB* \|\| Quarterly \|\| Database files, often persistent, rarely modifed		\| DB \|\| /nfs/db/<userid> \|\| 2TB* \|\| Quarterly at best \|\| Database files, often persistent, rarely modifed
	\|-		\|-
	\| Store \|\| /nfs/store/<userid> \|\| 500GB* \|\| Quarterly \|\| Crystallography files or other archived files, rarely modified.		\| Store \|\| /nfs/store/<userid> \|\| 500GB* \|\| Quarterly at best \|\| Crystallography files or other archived files, rarely modified.
	\|-		\|-
	\| Desktop \|\| /home/<userid> \|\| N/A \|\| Never \|\| Copy critical files from desktop to home directory on server.		\| Desktop \|\| /home/<userid> \|\| N/A \|\| Never \|\| Copy critical files from desktop to home directory on server.

Frodo: dsf

2015-06-03T04:04:07Z

dsf

← Older revision		Revision as of 04:04, 3 June 2015
Line 1:		Line 1:
	This is the lab disk space policy and applies to those with ssh access to our cluster only. This policy currently applies to [[cluster 2]]. ~~It will apply to [[cluster 0]] with caveats by June 1, 2014.~~		This is the lab disk space policy and applies to those with ssh access to our cluster only. This policy currently applies to [[cluster 2]].
	~~We ask you to~~ observe the following disk usage policy for your account on the cluster. ~~We think this~~ policy ~~will~~ cover 90% of ~~the~~ users, 90% of the time. We do understand that research can have unanticipated needs, and we are willing to work with you to create a plan that works for everyone.
			Please observe the following disk usage policy for your account on the cluster. This policy should cover 90% of users, 90% of the time. We understand that research can have unanticipated needs, and we are willing to work with you to create a plan that works for everyone.

	{{TOCright}}		{{TOCright}}
Line 20:		Line 21:

	= Treatment by disk type =		= Treatment by disk type =
	When you log in to a cluster computer, you are in your home directory. This would normally be /nfs/home/<userid>. Lab tradition is to have a ~/code/ directory for all software and a ~/work/ directory for all of your research. Your home directory is backed up weekly, and has a quota of 500 GB. Keeping the quota at this level will allow us to provide a high performance backup system that runs regularly and completes in a timely fashion.		When you log in to a cluster computer, you are in your home directory. This would normally be /nfs/home/<userid>. Lab tradition is to have a ~/code/ directory for all software and a ~/work/ directory for all of your research. Your home directory is backed up weekly, and has a quota of 500 GB. Keeping the quota at this level will allow us to provide a high performance backup system that runs regularly and completes in a timely fashion. /nfs/home is backed up around 300 days a year. Use it!

	== Docking jobs ==		== Docking jobs ==
	If you require more than your quota for ~/work/, we will create a special directory for you, e.g. /nfs/work/<userid>, which we suggest you symlink to ~/work/. We expect most lab members engaged in multiple docking projects will require space on /nfs/work. /nfs/work will ~~never~~ be backed up. We recommend you repatriate ~~key~~ files from time to time from /nfs/work/<userid> to e.g. ~/worksave/. We can help you write scripts to automate this. ~~You are also welcome~~ to make your own supplementary backups to a USB drive on your workstation or laptop. We have USB drives you may have for this purpose.		If you require more than your quota for ~/work/, we will create a special directory for you, e.g. /nfs/work/<userid>, which we suggest you symlink to ~/work/. We expect most lab members engaged in multiple docking projects will require space on /nfs/work. /nfs/work will NEVER be backed up. We recommend you repatriate critical files from time to time from /nfs/work/<userid> to e.g. ~/worksave/. We can help you write scripts to automate this. We encourage you to make your own supplementary backups to a USB drive on your workstation or laptop. We have USB drives you may have for this purpose.

	~~Historical~~ note: in the past, we used to attempt to backup nearly every file on every disk, including millions of docking job output files that had a transient existence. This resulted in slow computers - due to the backup process - and a lack of clarity of what was backed up. It was unsatisfactory. One problem has been clarity: how to understand what is backed up and what is not backed up. By backing up only /nfs/home/<userid> we are forcing you to choose the key input files from which you can re-create your output. You may also wish to use /nfs/store/<userid> to retain important results. But we want to get away from the idea of backing up every file on every disk, particularly those of transient existence and marginal value.		Fun historical note: In the past, we used to attempt to backup nearly every file on every disk, including millions of docking job output files that had a transient existence. This resulted in slow computers - due to the backup process - and a lack of clarity of what was backed up. It was unsatisfactory. One problem has been clarity: how to understand what is backed up and what is not backed up. By backing up only /nfs/home/<userid> we are forcing you to choose the key input files from which you can re-create your output. You may also wish to use /nfs/store/<userid> to retain important results. But we want to get away from the idea of backing up every file on every disk, particularly those of transient existence and marginal value.

	== Dockable database files ==		== Dockable database files ==
	If you create dockable database files that persist and are larger than your quota, ask us to create dedicated space for you, e.g. /nfs/db/<userid>. The hallmarks of database files is that they are large, they are often used by more than one person, and they are persistent. Database files are backed up quarterly beginning on the first day of the quarter.		If you create dockable database files that persist and are larger than your quota, ask us to create dedicated space for you, e.g. /nfs/db/users/<userid>. The hallmarks of database files is that they are large, they are often used by more than one person, and they are persistent. Database files are backed up quarterly beginning on the first day of the quarter.


	~~[[Category:Policy]]~~
	~~[[Category:Internal]]~~

	== Large datasets, completed projects ==		== Large datasets, completed projects ==
	If you have large files from the synchrotron or elsewhere, or a project has finished but you want to keep it online, ask us to create dedicated space for you, e.g. /nfs/store/<userid>. Storage files are backed up quarterly beginning on the first day of the quarter.		If you have large files from the synchrotron or elsewhere, or a project has finished but you want to keep it online, ask us to create dedicated space for you, e.g. /nfs/store/<userid>. Storage files are backed up quarterly beginning on the first day of the quarter.

	If after using /nfs/db, /nfs/work and /nfs/store to offload from /nfs/home/ you still need more space, please work with the [[sysadmin]]s ~~to help you~~.		If after using /nfs/db, /nfs/work and /nfs/store to offload from /nfs/home/ you still need more space, please work with the [[sysadmin]]s.

	= Workstation and Laptop =		= Workstation and Laptop =
Line 43:		Line 40:

	= Summary =		= Summary =
	We back up home directories ~~weekly~~, and selected database, crystallography and other archived files quarterly. We do not back up /nfs/work files and we do not back up workstations or laptops. To backup files from /nfs/work, ~~you must~~ repatriate the key files to your home directory whence they will be backed up as part of normal procedures. To backup laptop and desktop, use USB disks, which we can give you.		We back up cluster home directories daily, and selected database, crystallography and other archived files quarterly. We do not back up /nfs/work files and we do not back up workstations or laptops. To backup files from /nfs/work, repatriate the key files to your home directory whence they will be backed up as part of normal procedures. To backup laptop and desktop, use USB disks, which we can give you.

	Our backup systems cannot keep up with the growth in disk space and the voracious appetite for disk of group members. This is a pragmatic policy that, while requiring a little organization, is more likely to deliver restored files you can actually rely on.		Our backup systems cannot keep up with the growth in disk space and the voracious appetite for disk of group members. This is a pragmatic policy that, while requiring a little organization, is more likely to deliver restored files you can actually rely on.
Line 70:		Line 67:
	* [[Backups]]		* [[Backups]]

			[[Category:Policy]]

	[[Category:Internal]]		[[Category:Internal]]

Teague Sterling: /* Changing quotas from hard limits to soft limits and fixing * bullet point */

2014-06-03T17:08:52Z

Changing quotas from hard limits to soft limits and fixing * bullet point

← Older revision		Revision as of 17:08, 3 June 2014
Line 55:		Line 55:
	\| Home \|\| /nfs/home/<userid> \|\| 500 GB \|\| Weekly \|\| General use - not large database and not docking runs.		\| Home \|\| /nfs/home/<userid> \|\| 500 GB \|\| Weekly \|\| General use - not large database and not docking runs.
	\|-		\|-
	\| Work \|\| /nfs/work/<userid> \|\| ~~512GB~~* \|\| Never \|\| Docking runs or calculations that produce large output		\| Work \|\| /nfs/work/<userid> \|\| 500GB* \|\| Never \|\| Docking runs or calculations that produce large output
	\|-		\|-
	\| DB \|\| /nfs/db/<userid> \|\| 2TB* \|\| Quarterly \|\| Database files, often persistent, rarely modifed		\| DB \|\| /nfs/db/<userid> \|\| 2TB* \|\| Quarterly \|\| Database files, often persistent, rarely modifed
	\|-		\|-
	\| Store \|\| /nfs/store/<userid> \|\| ~~512GB~~* \|\| Quarterly \|\| Crystallography files or other archived files, rarely modified.		\| Store \|\| /nfs/store/<userid> \|\| 500GB* \|\| Quarterly \|\| Crystallography files or other archived files, rarely modified.
	\|-		\|-
	\| Desktop \|\| /home/<userid> \|\| N/A \|\| Never \|\| Copy critical files from desktop to home directory on server.		\| Desktop \|\| /home/<userid> \|\| N/A \|\| Never \|\| Copy critical files from desktop to home directory on server.
Line 65:		Line 65:
	\| Laptop \|\| / \|\| N/A \|\| Never \|\| Copy critical files from laptop to home directory on server.		\| Laptop \|\| / \|\| N/A \|\| Never \|\| Copy critical files from laptop to home directory on server.
	\|}		\|}
	* More space may be requested from the [[sysadmin]]s.		* * More space may be requested from the [[sysadmin]]s.

	= See Also =		= See Also =

Teague Sterling at 17:16, 30 May 2014

2014-05-30T17:16:37Z

← Older revision		Revision as of 17:16, 30 May 2014
Line 5:		Line 5:

	= Checking Disk Usage =		= Checking Disk Usage =
	You can check your disk usage on the cluster NFS (excluding /nfs/db) drives by running the quota command on any cluster machine.		You can check your disk usage on the cluster NFS (excluding /nfs/db) drives by running the '''quota''' command on any cluster machine.

	Example Output:		Example Output:

Teague Sterling: /* Checking Disk Usage */

2014-05-30T17:01:33Z

Checking Disk Usage

← Older revision		Revision as of 17:01, 30 May 2014
Line 5:		Line 5:

	= Checking Disk Usage =		= Checking Disk Usage =
	You can check your disk usage on the cluster NFS (excluding /nfs/db) drives by running the `quota` command on any cluster machine.		You can check your disk usage on the cluster NFS (excluding /nfs/db) drives by running the quota command on any cluster machine.

	Example Output:		Example Output:
Line 17:		Line 17:
	114M 500G 512G 3 0 0		114M 500G 512G 3 0 0
	nfs_store:/export/store/		nfs_store:/export/store/
	4 500G 512G 1 0 0		4 500G 512G 1 0 0

	= Treatment by disk type =		= Treatment by disk type =

Teague Sterling at 17:01, 30 May 2014

2014-05-30T17:01:02Z

@@ Line 3: / Line 3: @@
 {{TOCright}}
 = Treatment by disk type =
 When you log in to a cluster computer, you are in your home directory.  This would normally be /nfs/home/<userid>. Lab tradition is to have a ~/code/ directory for all software and a ~/work/ directory for all of your research. Your home directory is backed up weekly, and has a quota of 500 GB. Keeping the quota at this level will allow us to provide a high performance backup system that runs regularly and completes in a timely fashion.
 == Docking jobs ==
 If you require more than your quota for ~/work/, we will create a special directory for you, e.g. /nfs/work/<userid>, which we suggest you symlink to ~/work/.  We expect most lab members engaged in multiple docking projects will require space on /nfs/work.  /nfs/work will never be backed up. We recommend you repatriate key files from time to time from /nfs/work/<userid> to e.g. ~/worksave/. We can help you write scripts to automate this. You are also welcome to make your own supplementary backups to a USB drive on your workstation or laptop. We have USB drives you may have for this purpose.

Teague Sterling: /* Summary */

2014-05-29T21:13:10Z

Summary

← Older revision		Revision as of 21:13, 29 May 2014
Line 43:		Line 43:
	\| DB \|\| /nfs/db/<userid> \|\| 2TB* \|\| Quarterly \|\| Database files, often persistent, rarely modifed		\| DB \|\| /nfs/db/<userid> \|\| 2TB* \|\| Quarterly \|\| Database files, often persistent, rarely modifed
	\|-		\|-
	\| Store \|\| /nfs/store/<userid> \|\| ~~1TB~~* \|\| Quarterly \|\| Crystallography files or other archived files, rarely modified.		\| Store \|\| /nfs/store/<userid> \|\| 512GB* \|\| Quarterly \|\| Crystallography files or other archived files, rarely modified.
	\|-		\|-
	\| Desktop \|\| /home/<userid> \|\| N/A \|\| Never \|\| Copy critical files from desktop to home directory on server.		\| Desktop \|\| /home/<userid> \|\| N/A \|\| Never \|\| Copy critical files from desktop to home directory on server.

Teague Sterling: /* Summary */

2014-05-29T21:12:49Z

Summary

← Older revision		Revision as of 21:12, 29 May 2014
Line 39:		Line 39:
	\| Home \|\| /nfs/home/<userid> \|\| 500 GB \|\| Weekly \|\| General use - not large database and not docking runs.		\| Home \|\| /nfs/home/<userid> \|\| 500 GB \|\| Weekly \|\| General use - not large database and not docking runs.
	\|-		\|-
	\| Work \|\| /nfs/work/<userid> \|\| ~~2TB~~* \|\| Never \|\| Docking runs or calculations that produce large output		\| Work \|\| /nfs/work/<userid> \|\| 512GB* \|\| Never \|\| Docking runs or calculations that produce large output
	\|-		\|-
	\| DB \|\| /nfs/db/<userid> \|\| 2TB* \|\| Quarterly \|\| Database files, often persistent, rarely modifed		\| DB \|\| /nfs/db/<userid> \|\| 2TB* \|\| Quarterly \|\| Database files, often persistent, rarely modifed
	\|-		\|-
	\| Store \|\| /nfs/store/<userid> \|\| ~~2TB~~* \|\| Quarterly \|\| Crystallography files or other archived files, rarely modified.		\| Store \|\| /nfs/store/<userid> \|\| 1TB* \|\| Quarterly \|\| Crystallography files or other archived files, rarely modified.
	\|-		\|-
	\| Desktop \|\| /home/<userid> \|\| N/A \|\| Never \|\| Copy critical files from desktop to home directory on server.		\| Desktop \|\| /home/<userid> \|\| N/A \|\| Never \|\| Copy critical files from desktop to home directory on server.

Frodo at 13:50, 23 April 2014

2014-04-23T13:50:02Z

← Older revision		Revision as of 13:50, 23 April 2014
Line 13:		Line 13:
	== Dockable database files ==		== Dockable database files ==
	If you create dockable database files that persist and are larger than your quota, ask us to create dedicated space for you, e.g. /nfs/db/<userid>. The hallmarks of database files is that they are large, they are often used by more than one person, and they are persistent. Database files are backed up quarterly beginning on the first day of the quarter.		If you create dockable database files that persist and are larger than your quota, ask us to create dedicated space for you, e.g. /nfs/db/<userid>. The hallmarks of database files is that they are large, they are often used by more than one person, and they are persistent. Database files are backed up quarterly beginning on the first day of the quarter.


			[[Category:Policy]]
			[[Category:Internal]]

	== Large datasets, completed projects ==		== Large datasets, completed projects ==