How To Load New ZINC Databases - Revision history

Khtang at 18:50, 8 September 2021

2021-09-08T18:50:19Z

← Older revision		Revision as of 18:50, 8 September 2021
Line 1:		Line 1:
			'''NOT UPDATED'''
	== Start ==		== Start ==
	* log in to machine		* log in to machine

Khtang at 21:00, 27 October 2020

2020-10-27T21:00:10Z

← Older revision		Revision as of 21:00, 27 October 2020
Line 148:		Line 148:
	Ben		Ben
	</nowiki>		</nowiki>


			=== Updated procedure 10/25 ===
			TBD

Btingle: /* Email to Abhinav and Khanh, 6/22 */

2020-06-22T22:59:47Z

Email to Abhinav and Khanh, 6/22

← Older revision		Revision as of 22:59, 22 June 2020
Line 130:		Line 130:
	H??P???_H??P???/ - the root folder for an individual partition database. partitions can contain one ore more tranches		H??P???_H??P???/ - the root folder for an individual partition database. partitions can contain one ore more tranches
	src/ - the important output of scripts is stored here. This folder should usually not be messed with		src/ - the important output of scripts is stored here. This folder should usually not be messed with
	H??P???/ - the final table data for individual tranches in this partition. Also contain archives that record what has been added to the database		H??P???/ - the final table data for individual tranches in this partition. Also contain archives that record what has been added to the database
	stage/ - this is where we do our "calculations" for each step. you are safe to delete files in this directory once you're sure you don't need them anymore		stage/ - this is where we do our "calculations" for each step. you are safe to delete files in this directory once you're sure you don't need them anymore
	preprocessing/		preprocessing/

Btingle: /* Email to Abhinav and Khanh, 6/22 */

2020-06-22T22:59:10Z

Email to Abhinav and Khanh, 6/22

Btingle: /* Email to Khanh, 6/15 */

2020-06-22T22:57:36Z

Email to Khanh, 6/15

← Older revision		Revision as of 22:57, 22 June 2020
Line 108:		Line 108:

	to be clear, you would only want to export any of the variables for a particular stage if that stage succeeded the previous time you ran the script. It saves time to not have to re-do any work.		to be clear, you would only want to export any of the variables for a particular stage if that stage succeeded the previous time you ran the script. It saves time to not have to re-do any work.
			</nowiki>

			= Email to Abhinav and Khanh, 6/22 =

			<nowiki>
			Abhinav and Khanh,

			It turns out that the loading on n-1-17 was being polluted by other scripts john was running on the same machine.
			Also, there was bug in the script which caused it to fail when loading multiple files onto the same database.
			This only affected the loads on n-1-17.
			I've started a test run of partition 134 s on that machine, to test a fix I wrote for the script.
			If you started a load that had multiple files in it before you run you will need to:
			1. wipe the database for that port
			2. purge the loading source directory. Let me explain:
			The loading script produces a number of intermediate files when preparing them for loading into the database. These root directory for these files is the /local2/load directory.
			You will see a number of directories named after a tranche range. Here are the important locations:
			/local2/load
			H??P???_H??P???/ - the root folder for an individual partition database. partitions can contain one ore more tranches
			src/ - the important output of scripts is stored here. This folder should usually not be messed with
			H??P???/ - the final table data for individual tranches in this partition. Also contain archives that record what has been added to the database
			stage/ - this is where we do our "calculations" for each step. you are safe to delete files in this directory once you're sure you don't need them anymore
			preprocessing/
			postprocessing/ - (check the wiki page on when and how to skip certain steps of the script)
			resolve/
			loading/
			tmp/ - source files are copied here temporarily
			config.txt - contains the port number for this partition database
			The script was assigning incorrect id numbers to new entries in the database when the partition had more than one tranche (input file), causing numerous indexes to fail on rebuilding.
			In this case, we need to wipe the database and completely delete the contents of the source directory for each database because the final output was corrupted.
			Further runs of the script depend on this final output to be correct.
			You are safe to skip all steps except resolution and loading when re-trying a script that had this problem.

			Hope this has cleared some things up.

			Ben
	</nowiki>		</nowiki>

Btingle: /* Email to Khanh, 6/15 */

2020-06-15T18:46:10Z

Email to Khanh, 6/15

← Older revision		Revision as of 18:46, 15 June 2020
Line 107:		Line 107:
	...		...

	to be clear, you would only want to export any of the variables for a particular stage if that stage succeeded the previous time you ran the script. It saves time to not re-do any work.		to be clear, you would only want to export any of the variables for a particular stage if that stage succeeded the previous time you ran the script. It saves time to not have to re-do any work.
	</nowiki>		</nowiki>

Btingle: /* Email to Khanh, 6/15 */

2020-06-15T18:45:53Z

Email to Khanh, 6/15

← Older revision		Revision as of 18:45, 15 June 2020
Line 107:		Line 107:
	...		...

	to be clear, you would only want to export any of the variables for a particular stage if that stage succeeded the previous time you ran the script. ~~This is~~ to ~~save time~~ re-~~doing~~ work ~~that's already been done~~.		to be clear, you would only want to export any of the variables for a particular stage if that stage succeeded the previous time you ran the script. It saves time to not re-do any work.
	</nowiki>		</nowiki>

Btingle: /* Email to Khanh, 6/15 */

2020-06-15T18:44:21Z

Email to Khanh, 6/15

← Older revision		Revision as of 18:44, 15 June 2020
Line 104:		Line 104:
	Lmk if you have questions.		Lmk if you have questions.
	Ben		Ben

			...

			to be clear, you would only want to export any of the variables for a particular stage if that stage succeeded the previous time you ran the script. This is to save time re-doing work that's already been done.
	</nowiki>		</nowiki>

Btingle: /* Logging */

2020-06-15T18:09:00Z

Logging

← Older revision		Revision as of 18:09, 15 June 2020
Line 73:		Line 73:

	== Logging ==		== Logging ==
	check batch_logs/<step name>/<job id>_<array id>.out for the log output of various jobs		check batch_logs/<step name>_<job id>_<array id>.out for the log output of various jobs

	<step name> : <description>		<step name> : <description>

	run_catalog_load : main job output		run_catalog_load : main job output, loading and resolution script output

	pre_process : pre processing job output		pre_process : pre processing job output

	post_process : post processing job output		post_process : post processing job output

			= Email to Khanh, 6/15 =

			<nowiki>
			Khanh
			The reason there was an error is because of misconfigured environment variables, but I've fixed them now. Your script seems to have succeeded in pre-processing the molecules, but failed on the following
			stages. No big deal, you can just re-run the script with some changes.

			If the pre-processing stage went through but there was an error in any of the following stages I added the option to skip that(those) stage(s) in the following run. You can also skip the resolution and
			post processing stages if something goes wrong there. To see if there was an error with any of the stages you can check .err, but you can also check .out to see what the main script output is. If you
			don't see any jobs pop up for a particular stage, or if the output indicates it processed zero entries for that stage you can be sure something went wrong. You can re-run the script with these
			environment variables to skip that stage. Only export this environment variable if there was an error during a previous time you ran the script, for example you would not want to export any of these
			when running the script for a particular partition the first time.

			export SKIP_PRE_PROCESS="TRUE"
			export SKIP_RESOLUTION="TRUE"
			export SKIP_POST_PROCESS="TRUE"
			export SKIP_LOADING="TRUE"

			(I have no idea why you would ever want to use that last one, but it's there for completeness's sake)
			Lmk if you have questions.
			Ben
			</nowiki>

Btingle: /* ZINC_PORT */

2020-06-11T22:02:44Z

ZINC_PORT

← Older revision		Revision as of 22:02, 11 June 2020
Line 51:		Line 51:
	the port number of the postgres database		the port number of the postgres database

	if you're loading a database from a different source but the same partition number as another database, you must use the same port number as that database		if you're loading a database from a different source but the same partition number as another database, you must use the same host and port number as that database

	== Running the Script ==		== Running the Script ==

← Older revision		Revision as of 22:59, 22 June 2020
Line 116:		Line 116:

	It turns out that the loading on n-1-17 was being polluted by other scripts john was running on the same machine.		It turns out that the loading on n-1-17 was being polluted by other scripts john was running on the same machine.
	Also, there was bug in the script which caused it to fail when loading multiple files onto the same database.		Also, there was bug in the script which caused it to fail when loading multiple files onto the same database. This only affected the loads on n-1-17.
	This only affected the loads on n-1-17.
	I've started a test run of partition 134 s on that machine, to test a fix I wrote for the script.		I've started a test run of partition 134 s on that machine, to test a fix I wrote for the script.
	If you started a load that had multiple files in it before you run you will need to:		If you started a load that had multiple files in it before you run you will need to:

	1. wipe the database for that port		1. wipe the database for that port
	2. purge the loading source directory. ~~Let me explain:~~		2. purge the loading source directory (I'll explain)
			3. re-run all scripts for that database

	The loading script produces a number of intermediate files when preparing them for loading into the database. These root directory for these files is the /local2/load directory.		The loading script produces a number of intermediate files when preparing them for loading into the database. These root directory for these files is the /local2/load directory.
	You will see a number of directories named after a tranche range. Here are the important locations:		You will see a number of directories named after a tranche range. Here are the important locations:

	/local2/load		/local2/load
	H??P???_H??P???/ - the root folder for an individual partition database. partitions can contain one ore more tranches		H??P???_H??P???/ - the root folder for an individual partition database. partitions can contain one ore more tranches
Line 135:		Line 138:
	tmp/ - source files are copied here temporarily		tmp/ - source files are copied here temporarily
	config.txt - contains the port number for this partition database		config.txt - contains the port number for this partition database

	The script was assigning incorrect id numbers to new entries in the database when the partition had more than one tranche (input file), causing numerous indexes to fail on rebuilding.		The script was assigning incorrect id numbers to new entries in the database when the partition had more than one tranche (input file), causing numerous indexes to fail on rebuilding.
	In this case, we need to wipe the database and completely delete the contents of the source directory for each database because the final output was corrupted.		In this case, we need to wipe the database and completely delete the contents of the source directory for each database because the final output was corrupted.