CoreHPC: Difference between revisions
Jump to navigation
Jump to search
m (asdf) |
m (asdf) |
||
| (12 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
== Learning about FAC/coreHPC == | |||
You need one account on coreHPC. You access "FAC" via a disk mounted on coreHPC only: /mnt/fac/CX900007_DS1/ | You need one account on coreHPC. You access "FAC" via a disk mounted on coreHPC only: /mnt/fac/CX900007_DS1/ | ||
| Line 8: | Line 8: | ||
* there is /mnt/scratch/user/<yourid>/ scratch space. This is very fast disk, but not as fast as /tmp. The data will likely persist for weeks or months, but there is no guarantee. | * there is /mnt/scratch/user/<yourid>/ scratch space. This is very fast disk, but not as fast as /tmp. The data will likely persist for weeks or months, but there is no guarantee. | ||
* the FAC storage is your best bet for long term, but we "only" have 50 TB. So you cannot store big chunks of ZINC for a long time. Please get in the habit of compressing and moving your data off FAC back to our clusters when your project is complete. | * the FAC storage is your best bet for long term, but we "only" have 50 TB. So you cannot store big chunks of ZINC for a long time. Please get in the habit of compressing and moving your data off FAC back to our clusters when your project is complete. | ||
* to make a request to the CoreHPC managers, use this link: [https://tiny.ucsf.edu/Gethpc] | |||
| Line 19: | Line 20: | ||
* The pipe to FAC is currently at best 10 Gbps. We are working on getting more. | * The pipe to FAC is currently at best 10 Gbps. We are working on getting more. | ||
* to get to corehpc | == to get to the DMZ (jjiteam only) == | ||
facbastp9007.fac.ucsf.edu | |||
ssh n7vu01 | |||
our certs: /etc/letsencrypt/live/docking-irwinlab / *.pem | |||
== to get to corehpc == | |||
ssh <user>@chpc-ucsf-bastion-vm1.corehpc.ucsf.edu | ssh <user>@chpc-ucsf-bastion-vm1.corehpc.ucsf.edu | ||
| Line 25: | Line 32: | ||
ssh chpc-ucsf-login-vm1 | ssh chpc-ucsf-login-vm1 | ||
== disks are : == | |||
/mnt/scratch/user/<user>. | /mnt/scratch/user/<user>. | ||
/mnt/home/<user> | /mnt/home/<user> | ||
/home/remote/<user> ; you land here when you login | /home/remote/<user> ; you land here when you login | ||
== To check disk usage == | |||
** For /home/remote and /scratch | ** For /home/remote and /scratch | ||
*:: <source> quota -s </source> | *:: <source> quota -s </source> | ||
| Line 36: | Line 43: | ||
*:: <source> df -h /mnt/fac/CX900007_DS1 </source> | *:: <source> df -h /mnt/fac/CX900007_DS1 </source> | ||
== | == Read more == | ||
https://wiki.library.ucsf.edu/spaces/CHPC/pages/720396955/CoreHPC+Access+Primer | https://wiki.library.ucsf.edu/spaces/CHPC/pages/720396955/CoreHPC+Access+Primer | ||
https://it.ucsf.edu/service/corehpc | https://it.ucsf.edu/service/corehpc | ||
| Line 61: | Line 68: | ||
It is 1 TB. It is group writable for our group and our group alone. | It is 1 TB. It is group writable for our group and our group alone. | ||
== how we 'fix' the dmz machine == | |||
* apt install inetutils-telnet | |||
== how to do samba mounts == | |||
* on files2 | |||
** sudo dnf install cifs-utils -y | |||
** mount -t cifs //capstorage.fac.ucsf.edu/CX900007_DS1. /nfs/fac -o username=<username>,password=<password> | |||
*** mount error(13): Permission denied | |||
** mount -t cifs //capstorage.fac.ucsf.edu/CX900007_DS1 /nfs/fac -o username=jirwin3,vers=3.1.1 | |||
*** mount error(13): Permission denied | |||
** smbclient -L //capstorage.fac.ucsf.edu -U jirwin3 | |||
*** session setup failed: NT_STATUS_LOGON_FAILURE | |||
Bharat suggests: | |||
* mount -t cifs //capstorage.fac.ucsf.edu/CX900007_DS1. /nfs/fac -o username=<username>,password=<password>,domain=campus | |||
== how to move files rapidly to and from FAC/coreHPC == | |||
* on epyc2 on cluster 2 and home01 on cluster 7. mounted as /nfs/fac/ in both places. | |||
* in principle you can copy files to and from there. | |||
* in principle this is an easy way to move files to/from Santa Clara. | |||
* We are still working out the file permissions issue. | |||
* but you can use it right now. Please do, which will force us to think harder about the permissions issue. | |||
[[Category:Cluster]] | [[Category:Cluster]] | ||
Latest revision as of 20:38, 4 November 2026
Learning about FAC/coreHPC
You need one account on coreHPC. You access "FAC" via a disk mounted on coreHPC only: /mnt/fac/CX900007_DS1/
- We have 50TB on FAC for permanent storage. Use this for software, dockfiles, databases that you need to last for more than a few weeks.
- coreHPC is where you run jobs, using Slurm.
- your home directory is tiny. Use it for scripts and notes. It won't hold much else.
- there are two kinds of scratch space. /tmp on each node allows you to write data locally while the job is running. You must repariate it back to longer-term storage because /tmp gets wiped when your job ends.
- there is /mnt/scratch/user/<yourid>/ scratch space. This is very fast disk, but not as fast as /tmp. The data will likely persist for weeks or months, but there is no guarantee.
- the FAC storage is your best bet for long term, but we "only" have 50 TB. So you cannot store big chunks of ZINC for a long time. Please get in the habit of compressing and moving your data off FAC back to our clusters when your project is complete.
- to make a request to the CoreHPC managers, use this link: [1]
- We collectively refer to FAC/coreHPC as "Cluster 8".
- There is an open question of how to work efficiently with coreHPC. We are currently in an experimental phase.
- generally, the idea is: Our clusters in Byers Hall are persistent storage.
- move data you need to coreHPC, run the jobs, collect the output, and then move/delete to shrink your footprint when done.
- ZINC thus lives on our cluster (currently hosted on cluster 2, cross mounted read only to cluster 7).
- a copy of 50% of ZINC lives on Wynton/cluster 3.
- The pipe to FAC is currently at best 10 Gbps. We are working on getting more.
to get to the DMZ (jjiteam only)
facbastp9007.fac.ucsf.edu ssh n7vu01 our certs: /etc/letsencrypt/live/docking-irwinlab / *.pem
to get to corehpc
ssh <user>@chpc-ucsf-bastion-vm1.corehpc.ucsf.edu
- to get to slurm head node
ssh chpc-ucsf-login-vm1
disks are :
/mnt/scratch/user/<user>. /mnt/home/<user> /home/remote/<user> ; you land here when you login
To check disk usage
- For /home/remote and /scratch
quota -s
- For /mnt/fac/CX900007_DS1
df -h /mnt/fac/CX900007_DS1
Read more
https://wiki.library.ucsf.edu/spaces/CHPC/pages/720396955/CoreHPC+Access+Primer https://it.ucsf.edu/service/corehpc https://wiki.library.ucsf.edu/spaces/CHPC/pages/736476672/CoreHPC+Submission+Examples https://wiki.library.ucsf.edu/spaces/CHPC/pages/751928275/CoreHPC+Technical+Overview
Anyone who uses coreHPC, please take notes, update this and other pages, and teach the rest of the lab how to use it. Thank you.
how to do remote mounts
https://wiki.library.ucsf.edu/spaces/CHPC/pages/720396955/CoreHPC+Access+Primer
https://it.ucsf.edu/how-to/fac-drive-mapping-instructions?check_logged_in=1#linux
smb://capstorage.fac.ucsf.edu/CX900007_DS1. lab storage /mnt/fac/CX900007_DS1 fac
- tiny.ucsf.edu/getfac - this is how you file tickets for FAC.
our shared storage is on /home/remote/jirwin3/fac
It is 1 TB. It is group writable for our group and our group alone.
how we 'fix' the dmz machine
- apt install inetutils-telnet
how to do samba mounts
- on files2
- sudo dnf install cifs-utils -y
- mount -t cifs //capstorage.fac.ucsf.edu/CX900007_DS1. /nfs/fac -o username=<username>,password=<password>
- mount error(13): Permission denied
- mount -t cifs //capstorage.fac.ucsf.edu/CX900007_DS1 /nfs/fac -o username=jirwin3,vers=3.1.1
- mount error(13): Permission denied
- smbclient -L //capstorage.fac.ucsf.edu -U jirwin3
- session setup failed: NT_STATUS_LOGON_FAILURE
Bharat suggests:
- mount -t cifs //capstorage.fac.ucsf.edu/CX900007_DS1. /nfs/fac -o username=<username>,password=<password>,domain=campus
how to move files rapidly to and from FAC/coreHPC
- on epyc2 on cluster 2 and home01 on cluster 7. mounted as /nfs/fac/ in both places.
- in principle you can copy files to and from there.
- in principle this is an easy way to move files to/from Santa Clara.
- We are still working out the file permissions issue.
- but you can use it right now. Please do, which will force us to think harder about the permissions issue.