CoreHPC: Difference between revisions

From DISI
Jump to navigation Jump to search
m (asdf)
m (asdf)
 
(10 intermediate revisions by the same user not shown)
Line 1: Line 1:
We are still learning about FAC/coreHPC.
== Learning about FAC/coreHPC ==


You need one account on coreHPC. You access "FAC" via a disk mounted on coreHPC only: /mnt/fac/CX900007_DS1/
You need one account on coreHPC. You access "FAC" via a disk mounted on coreHPC only: /mnt/fac/CX900007_DS1/
Line 20: Line 20:
* The pipe to FAC is currently at best 10 Gbps. We are working on getting more.  
* The pipe to FAC is currently at best 10 Gbps. We are working on getting more.  


* to get to corehpc
== to get to the DMZ (jjiteam only) ==
facbastp9007.fac.ucsf.edu
ssh n7vu01
our certs: /etc/letsencrypt/live/docking-irwinlab / *.pem
 
 
== to get to corehpc ==
  ssh <user>@chpc-ucsf-bastion-vm1.corehpc.ucsf.edu
  ssh <user>@chpc-ucsf-bastion-vm1.corehpc.ucsf.edu


Line 26: Line 32:
  ssh chpc-ucsf-login-vm1
  ssh chpc-ucsf-login-vm1


* disks are  
== disks are : ==
  /mnt/scratch/user/<user>.  
  /mnt/scratch/user/<user>.  
  /mnt/home/<user>  
  /mnt/home/<user>  
  /home/remote/<user>  ; you land here when you login
  /home/remote/<user>  ; you land here when you login


* To check disk usage
== To check disk usage ==
** For /home/remote and /scratch
** For /home/remote and /scratch
*:: <source> quota -s </source>
*:: <source> quota -s </source>
Line 37: Line 43:
*:: <source> df -h /mnt/fac/CX900007_DS1 </source>
*:: <source> df -h /mnt/fac/CX900007_DS1 </source>


== Learn more ==  
== Read more ==  
  https://wiki.library.ucsf.edu/spaces/CHPC/pages/720396955/CoreHPC+Access+Primer
  https://wiki.library.ucsf.edu/spaces/CHPC/pages/720396955/CoreHPC+Access+Primer
  https://it.ucsf.edu/service/corehpc
  https://it.ucsf.edu/service/corehpc
Line 62: Line 68:
It is 1 TB. It is group writable for our group and our group alone.  
It is 1 TB. It is group writable for our group and our group alone.  


== how we 'fix' the dmz machine ==
* apt install inetutils-telnet
== how to do samba mounts ==
* on files2
** sudo dnf install cifs-utils -y
** mount -t cifs //capstorage.fac.ucsf.edu/CX900007_DS1. /nfs/fac -o username=<username>,password=<password>
*** mount error(13): Permission denied
** mount -t cifs //capstorage.fac.ucsf.edu/CX900007_DS1 /nfs/fac -o username=jirwin3,vers=3.1.1
*** mount error(13): Permission denied
** smbclient -L //capstorage.fac.ucsf.edu -U jirwin3
*** session setup failed: NT_STATUS_LOGON_FAILURE
Bharat suggests:
* mount -t cifs //capstorage.fac.ucsf.edu/CX900007_DS1. /nfs/fac -o username=<username>,password=<password>,domain=campus
== how to move files rapidly to and from FAC/coreHPC ==
* on epyc2 on cluster 2 and home01 on cluster 7.  mounted as /nfs/fac/ in both places.
* in principle you can copy files to and from there.
* in principle this is an easy way to move files to/from Santa Clara.
* We are still working out the file permissions issue.
* but you can use it right now. Please do, which will force us to think harder about the permissions issue.
[[Category:Cluster]]
[[Category:Cluster]]

Latest revision as of 20:38, 4 November 2026

Learning about FAC/coreHPC

You need one account on coreHPC. You access "FAC" via a disk mounted on coreHPC only: /mnt/fac/CX900007_DS1/

  • We have 50TB on FAC for permanent storage. Use this for software, dockfiles, databases that you need to last for more than a few weeks.
  • coreHPC is where you run jobs, using Slurm.
  • your home directory is tiny. Use it for scripts and notes. It won't hold much else.
  • there are two kinds of scratch space. /tmp on each node allows you to write data locally while the job is running. You must repariate it back to longer-term storage because /tmp gets wiped when your job ends.
  • there is /mnt/scratch/user/<yourid>/ scratch space. This is very fast disk, but not as fast as /tmp. The data will likely persist for weeks or months, but there is no guarantee.
  • the FAC storage is your best bet for long term, but we "only" have 50 TB. So you cannot store big chunks of ZINC for a long time. Please get in the habit of compressing and moving your data off FAC back to our clusters when your project is complete.
  • to make a request to the CoreHPC managers, use this link: [1]


  • We collectively refer to FAC/coreHPC as "Cluster 8".
  • There is an open question of how to work efficiently with coreHPC. We are currently in an experimental phase.
  • generally, the idea is: Our clusters in Byers Hall are persistent storage.
  • move data you need to coreHPC, run the jobs, collect the output, and then move/delete to shrink your footprint when done.
  • ZINC thus lives on our cluster (currently hosted on cluster 2, cross mounted read only to cluster 7).
  • a copy of 50% of ZINC lives on Wynton/cluster 3.
  • The pipe to FAC is currently at best 10 Gbps. We are working on getting more.

to get to the DMZ (jjiteam only)

facbastp9007.fac.ucsf.edu
ssh n7vu01
our certs: /etc/letsencrypt/live/docking-irwinlab / *.pem


to get to corehpc

ssh <user>@chpc-ucsf-bastion-vm1.corehpc.ucsf.edu
  • to get to slurm head node
ssh chpc-ucsf-login-vm1

disks are :

/mnt/scratch/user/<user>. 
/mnt/home/<user> 
/home/remote/<user>  ; you land here when you login

To check disk usage

    • For /home/remote and /scratch
     quota -s
    • For /mnt/fac/CX900007_DS1
     df -h /mnt/fac/CX900007_DS1

Read more

https://wiki.library.ucsf.edu/spaces/CHPC/pages/720396955/CoreHPC+Access+Primer
https://it.ucsf.edu/service/corehpc
https://wiki.library.ucsf.edu/spaces/CHPC/pages/736476672/CoreHPC+Submission+Examples
https://wiki.library.ucsf.edu/spaces/CHPC/pages/751928275/CoreHPC+Technical+Overview

Anyone who uses coreHPC, please take notes, update this and other pages, and teach the rest of the lab how to use it. Thank you.


how to do remote mounts

https://wiki.library.ucsf.edu/spaces/CHPC/pages/720396955/CoreHPC+Access+Primer
https://it.ucsf.edu/how-to/fac-drive-mapping-instructions?check_logged_in=1#linux


smb://capstorage.fac.ucsf.edu/CX900007_DS1. lab storage
/mnt/fac/CX900007_DS1 fac


  • tiny.ucsf.edu/getfac - this is how you file tickets for FAC.


our shared storage is on /home/remote/jirwin3/fac It is 1 TB. It is group writable for our group and our group alone.


how we 'fix' the dmz machine

  • apt install inetutils-telnet


how to do samba mounts

  • on files2
    • sudo dnf install cifs-utils -y
    • mount -t cifs //capstorage.fac.ucsf.edu/CX900007_DS1. /nfs/fac -o username=<username>,password=<password>
      • mount error(13): Permission denied
    • mount -t cifs //capstorage.fac.ucsf.edu/CX900007_DS1 /nfs/fac -o username=jirwin3,vers=3.1.1
      • mount error(13): Permission denied
    • smbclient -L //capstorage.fac.ucsf.edu -U jirwin3
      • session setup failed: NT_STATUS_LOGON_FAILURE


Bharat suggests:

  • mount -t cifs //capstorage.fac.ucsf.edu/CX900007_DS1. /nfs/fac -o username=<username>,password=<password>,domain=campus

how to move files rapidly to and from FAC/coreHPC

  • on epyc2 on cluster 2 and home01 on cluster 7. mounted as /nfs/fac/ in both places.
  • in principle you can copy files to and from there.
  • in principle this is an easy way to move files to/from Santa Clara.
  • We are still working out the file permissions issue.
  • but you can use it right now. Please do, which will force us to think harder about the permissions issue.