Curation of ZINC-22 3D: Difference between revisions

From DISI
Jump to navigation Jump to search
(asdf)
 
 
(4 intermediate revisions by the same user not shown)
Line 4: Line 4:
== General remarks ==  
== General remarks ==  
* I suggest using screens on gimel and then logging in to disk-hosting machine.  
* I suggest using screens on gimel and then logging in to disk-hosting machine.  
* /nfs/exd is where files live and README describing the layers


== Once per day - 10 mins? ==
== Once per day - 10 mins? ==
* check for disk full conditions (our cluster and Wynton)
* check for disk full conditions (our cluster and Wynton)
df -h /nfs/ex? /nfs/db? | sort -k 5 -r |head
beegfs-ctl --getquota --gid bks
* check that the queues on our cluster and Wynton are full
* check that the queues on our cluster and Wynton are full
* check whether any preparation processes have finished (and move on to next step)  
 
* check whether any preparation processes have finished (and move on to next step)


== Maintain current info in spreadsheet ==  
== Maintain current info in spreadsheet ==  
Line 17: Line 22:
* blue = 3D build running
* blue = 3D build running
* yellow = attention required when resources allow
* yellow = attention required when resources allow
* bold = unpack/repack in process.  
* bold = unpack/repack in process.
* orange = on hold /suspended.


== Copies ==  
== Copies ==  

Latest revision as of 19:10, 17 June 2022

These are notes from John to Max to help guide the curation of ZINC-22 3D. Each topic has its own subject heading.


General remarks

  • I suggest using screens on gimel and then logging in to disk-hosting machine.
  • /nfs/exd is where files live and README describing the layers

Once per day - 10 mins?

  • check for disk full conditions (our cluster and Wynton)
df -h /nfs/ex? /nfs/db? | sort -k 5 -r |head
beegfs-ctl --getquota --gid bks
  • check that the queues on our cluster and Wynton are full
  • check whether any preparation processes have finished (and move on to next step)

Maintain current info in spreadsheet

https://docs.google.com/spreadsheets/d/14QpCiB78feWPQKT5LkpfKDNR6VCAMbEs7Md-QMX0glc/edit#gid=298497589

color coding

  • green = complete
  • blue = 3D build running
  • yellow = attention required when resources allow
  • bold = unpack/repack in process.
  • orange = on hold /suspended.

Copies

  • we make one copy to AWS
  • one copy to OCI
  • one copy to Wynton
  • we have been retaining 50-mers. We should delete them.


Unpack/repack process

  • ls -ld /nfs/ex?/zinc-22?/H??/H??[PM]???* - this tells you where the 50-mers live
  • ls /nfs/ex?/zinc-22?/H??/H??[PM]???*/out - this tells you how many top level dirs.
  • look at the last one. ls -ld * | wc -l helps you check finished / fully transferred
  • csh cmd-? H??P???... unpack
  • sdi6.script H?? sdi
  • csh master-f? H?? <a-z>. repack
  • mv H?? H??.old when repack complete
  • rm -rf H??.old when sure complete
  • start another one.