CoreHPC: Difference between revisions

From DISI
Jump to navigation Jump to search
m (asdf)
 
(14 intermediate revisions by 2 users not shown)
Line 1: Line 1:
We are still learning about FAC/coreHPC.
== Learning about FAC/coreHPC ==


you will need to get separate (?) FAC and coreHPC accounts.  
You need one account on coreHPC. You access "FAC" via a disk mounted on coreHPC only: /mnt/fac/CX900007_DS1/
* FAC is for storage
* We have 50TB on FAC for permanent storage.  Use this for software, dockfiles, databases that you need to last for more than a few weeks.
* coreHPC is for compute
* coreHPC is where you run jobs, using Slurm.  
* it is more complicated than that, but it is a reasonable simplification.  
* your home directory is tiny. Use it for scripts and notes. It won't hold much else.
* there are two kinds of scratch space.  /tmp on each node allows you to write data locally while the job is running. You must repariate it back to longer-term storage because /tmp gets wiped when your job ends.
* there is /mnt/scratch/user/<yourid>/ scratch space. This is very fast disk, but not as fast as /tmp. The data will likely persist for weeks or months, but there is no guarantee.
* the FAC storage is your best bet for long term, but we "only" have 50 TB. So you cannot store big chunks of ZINC for a long time.  Please get in the habit of compressing and moving your data off FAC back to our clusters when your project is complete.
* to make a request to the CoreHPC managers, use this link: [https://tiny.ucsf.edu/Gethpc]


* We collectively refer to FAC/coreHPC as "Cluster 8".  
* We collectively refer to FAC/coreHPC as "Cluster 8".  
* We are attempting two strategies for cluster 8:  
* There is an open question of how to work efficiently with coreHPC. We are currently in an experimental phase.
** upload ZINC on the fly, dock it, and then throw it away.  
* generally, the idea is: Our clusters in Byers Hall are persistent storage.
** nfs mount our cluster on coreHPC, data gets pulled as needed
* move data you need to coreHPC, run the jobs, collect the output, and then move/delete to shrink your footprint when done.  


* ZINC thus lives on our cluster (currently hosted on cluster 2, cross mounted read only to cluster 7).
* ZINC thus lives on our cluster (currently hosted on cluster 2, cross mounted read only to cluster 7).
* a copy of 50% of ZINC lives on Wynton/cluster 3.
* a copy of 50% of ZINC lives on Wynton/cluster 3.
* The pipe to FAC is 100 Gbps. Assming we get 50% of that, allows 1 TB in 3 minutes, and 3 PB is 6 days.
* The pipe to FAC is currently at best 10 Gbps. We are working on getting more.  
* thus, ftp/dock/delete.  


* to get to corehpc
== to get to the DMZ (jjiteam only) ==
facbastp9007.fac.ucsf.edu
ssh n7vu01
our certs: /etc/letsencrypt/live/docking-irwinlab / *.pem
 
 
== to get to corehpc ==
  ssh <user>@chpc-ucsf-bastion-vm1.corehpc.ucsf.edu
  ssh <user>@chpc-ucsf-bastion-vm1.corehpc.ucsf.edu


Line 22: Line 32:
  ssh chpc-ucsf-login-vm1
  ssh chpc-ucsf-login-vm1


* disks are  
== disks are : ==
  /mnt/scratch/user/<user>.  
  /mnt/scratch/user/<user>.  
  /mnt/home/<user>  
  /mnt/home/<user>  
  /home/remote/<user>  ; you land here when you login
  /home/remote/<user>  ; you land here when you login


== Learn more ==  
== To check disk usage ==
** For /home/remote and /scratch
*:: <source> quota -s </source>
** For /mnt/fac/CX900007_DS1
*:: <source> df -h /mnt/fac/CX900007_DS1 </source>
 
== Read more ==  
  https://wiki.library.ucsf.edu/spaces/CHPC/pages/720396955/CoreHPC+Access+Primer
  https://wiki.library.ucsf.edu/spaces/CHPC/pages/720396955/CoreHPC+Access+Primer
  https://it.ucsf.edu/service/corehpc
  https://it.ucsf.edu/service/corehpc
Line 52: Line 68:
It is 1 TB. It is group writable for our group and our group alone.  
It is 1 TB. It is group writable for our group and our group alone.  


== how we 'fix' the dmz machine ==
* apt install inetutils-telnet
== how to do samba mounts ==
* on files2
** sudo dnf install cifs-utils -y
** mount -t cifs //capstorage.fac.ucsf.edu/CX900007_DS1. /nfs/fac -o username=<username>,password=<password>
*** mount error(13): Permission denied
** mount -t cifs //capstorage.fac.ucsf.edu/CX900007_DS1 /nfs/fac -o username=jirwin3,vers=3.1.1
*** mount error(13): Permission denied
** smbclient -L //capstorage.fac.ucsf.edu -U jirwin3
*** session setup failed: NT_STATUS_LOGON_FAILURE
Bharat suggests:
* mount -t cifs //capstorage.fac.ucsf.edu/CX900007_DS1. /nfs/fac -o username=<username>,password=<password>,domain=campus
== how to move files rapidly to and from FAC/coreHPC ==
* on epyc2 on cluster 2 and home01 on cluster 7.  mounted as /nfs/fac/ in both places.
* in principle you can copy files to and from there.
* in principle this is an easy way to move files to/from Santa Clara.
* We are still working out the file permissions issue.
* but you can use it right now. Please do, which will force us to think harder about the permissions issue.
[[Category:Cluster]]
[[Category:Cluster]]

Latest revision as of 20:38, 4 November 2026

Learning about FAC/coreHPC

You need one account on coreHPC. You access "FAC" via a disk mounted on coreHPC only: /mnt/fac/CX900007_DS1/

  • We have 50TB on FAC for permanent storage. Use this for software, dockfiles, databases that you need to last for more than a few weeks.
  • coreHPC is where you run jobs, using Slurm.
  • your home directory is tiny. Use it for scripts and notes. It won't hold much else.
  • there are two kinds of scratch space. /tmp on each node allows you to write data locally while the job is running. You must repariate it back to longer-term storage because /tmp gets wiped when your job ends.
  • there is /mnt/scratch/user/<yourid>/ scratch space. This is very fast disk, but not as fast as /tmp. The data will likely persist for weeks or months, but there is no guarantee.
  • the FAC storage is your best bet for long term, but we "only" have 50 TB. So you cannot store big chunks of ZINC for a long time. Please get in the habit of compressing and moving your data off FAC back to our clusters when your project is complete.
  • to make a request to the CoreHPC managers, use this link: [1]


  • We collectively refer to FAC/coreHPC as "Cluster 8".
  • There is an open question of how to work efficiently with coreHPC. We are currently in an experimental phase.
  • generally, the idea is: Our clusters in Byers Hall are persistent storage.
  • move data you need to coreHPC, run the jobs, collect the output, and then move/delete to shrink your footprint when done.
  • ZINC thus lives on our cluster (currently hosted on cluster 2, cross mounted read only to cluster 7).
  • a copy of 50% of ZINC lives on Wynton/cluster 3.
  • The pipe to FAC is currently at best 10 Gbps. We are working on getting more.

to get to the DMZ (jjiteam only)

facbastp9007.fac.ucsf.edu
ssh n7vu01
our certs: /etc/letsencrypt/live/docking-irwinlab / *.pem


to get to corehpc

ssh <user>@chpc-ucsf-bastion-vm1.corehpc.ucsf.edu
  • to get to slurm head node
ssh chpc-ucsf-login-vm1

disks are :

/mnt/scratch/user/<user>. 
/mnt/home/<user> 
/home/remote/<user>  ; you land here when you login

To check disk usage

    • For /home/remote and /scratch
     quota -s
    • For /mnt/fac/CX900007_DS1
     df -h /mnt/fac/CX900007_DS1

Read more

https://wiki.library.ucsf.edu/spaces/CHPC/pages/720396955/CoreHPC+Access+Primer
https://it.ucsf.edu/service/corehpc
https://wiki.library.ucsf.edu/spaces/CHPC/pages/736476672/CoreHPC+Submission+Examples
https://wiki.library.ucsf.edu/spaces/CHPC/pages/751928275/CoreHPC+Technical+Overview

Anyone who uses coreHPC, please take notes, update this and other pages, and teach the rest of the lab how to use it. Thank you.


how to do remote mounts

https://wiki.library.ucsf.edu/spaces/CHPC/pages/720396955/CoreHPC+Access+Primer
https://it.ucsf.edu/how-to/fac-drive-mapping-instructions?check_logged_in=1#linux


smb://capstorage.fac.ucsf.edu/CX900007_DS1. lab storage
/mnt/fac/CX900007_DS1 fac


  • tiny.ucsf.edu/getfac - this is how you file tickets for FAC.


our shared storage is on /home/remote/jirwin3/fac It is 1 TB. It is group writable for our group and our group alone.


how we 'fix' the dmz machine

  • apt install inetutils-telnet


how to do samba mounts

  • on files2
    • sudo dnf install cifs-utils -y
    • mount -t cifs //capstorage.fac.ucsf.edu/CX900007_DS1. /nfs/fac -o username=<username>,password=<password>
      • mount error(13): Permission denied
    • mount -t cifs //capstorage.fac.ucsf.edu/CX900007_DS1 /nfs/fac -o username=jirwin3,vers=3.1.1
      • mount error(13): Permission denied
    • smbclient -L //capstorage.fac.ucsf.edu -U jirwin3
      • session setup failed: NT_STATUS_LOGON_FAILURE


Bharat suggests:

  • mount -t cifs //capstorage.fac.ucsf.edu/CX900007_DS1. /nfs/fac -o username=<username>,password=<password>,domain=campus

how to move files rapidly to and from FAC/coreHPC

  • on epyc2 on cluster 2 and home01 on cluster 7. mounted as /nfs/fac/ in both places.
  • in principle you can copy files to and from there.
  • in principle this is an easy way to move files to/from Santa Clara.
  • We are still working out the file permissions issue.
  • but you can use it right now. Please do, which will force us to think harder about the permissions issue.