Another window will pop up with the resource file displayed.
KEYWORD = VALUE ! COMMENTTokens are space delimited. Stand-alone comment lines begin with "!". The comment field is optional, along with the "!". The value fields may include spaces only if the value is delimited by single quotes:
KEYWORD = 'value with spaces' !An example which includes spaces
A special syntax is used for keywords that should be set in the environment of the process. For those keywords, the string ENV. is prepended to the keyword name. This is the primary mechanism through which an external polling process gains access to values in the process resource and path files.
SYSTEM = name_of_your_project CLASS = subsystem TASK = < xpoll -p $PATH_FILE -r taskname > !the OPUS method for invoking !the processThe first two, SYSTEM and CLASS, are now required to build the tree of processes in the PMG.
For so-called "external polling" processes, which include scripts and programs with no knowledge of how the OPUS pipeline works, these keywords are required also:
COMMAND = name_of_script !the script (in your path) that !runs this pipeline process XPOLL_STATE.01 = SUCCESS !maps a successful exit status (01 in this !case) from the process !so that XPOLL does not think the process !encountered an error XPOLL_ERROR.pp = X !unknown status exitIf the MINBLOCKS keyword is provided, the OUTPATH keyword must also be included:
MINBLOCKS = 50000 !disk space in 512-byte blocks required for !the process to begin execution OUTPATH = output_directory !defines the disk device where MINBLOCKS of !free space must exist in order to start the !processSimple examples of resource files for many types of external pollers are included with the sample pipeline.
CLASS - a designator for the class of data that the process operates on. All processes are ordered first by SYSTEM and then by CLASS to build the process tree for the PMG.
COMMAND - the name of your script to be executed by the OPUS XPOLL task (when XPOLL appears in the TASK keyword value).
DELTA_TIME - time added to the current time at which the pipeline process will awaken and run again (DDD:HH:MM:SS).
DESCRIPTION - a short description of the process that appears with the process name as a tooltip in the PMG's process tree. If this string contains spaces, it must be enclosed within single quotes.
ENV.keyword - enables a task to have access to an environment variable defined in your opus_login.csh
FILE_ABSENT - defines the modifier applied to the FILE_OBJECT when a file polling process goes absent.
FILE_DIRECTORYn - where n=1...(Number of file searches); identifies the directory to search for files which are used as triggers.
FILE_ERROR. - Process resource file status definitions can update any field or combination of fields for OSF's and files. For example, the status definition
FILE_ERROR.DIRECTORY = /data/complete/will update a file's directory (i.e. copy it to a new directory). The field names for files are DIRECTORY, ROOTNAME, EXTENSION, DANGLE. More than one field can be updated per status definition, as usual, except for the special cases OSF_PROCESSING and FILE_PROCESSING
FILE_SUCCESS.Update actions on successful completion. See FILE_ERROR. above.
FILE_OBJECTn - where n=1...(Number of file searches); identifies the filename or file mask used to search for files.
FILE_RANK - priority of FILE events in the OPUS event queue (1...N, where N is a positive integer).
FILE_MAXTARGSn - where n=1...(Number of triggers); identifies how many files an application will handle per event (the application must support multiple files per event if > 1). If this key is not defined, a single file will be included per event.
FILE_ACTION - specifies that the task indicated by the value should be executed upon FILE_SUCCESS.
FILE_ACTION_OK - specifies that the exit status of a FILE_ACTION task be tested and if not equal to this value, an error should be posted to the process log file.
MAX_ERROR - the number of times that XPOLL will accept an error status return from a pipeline process and continue to activate the process on the next event, AS LONG AS THE EXIT STATUS IS NOT IN THE FATAL RANGE (100-127 inclusive). Once this count of errors is exceeded or a single occurrence of a status in the fatal range occurs, XPOLL will stop activating the process (in pipeline terms, the process will go ABSENT).
MINBLOCKS - number of 512-byte blocks that must be free on the output device specified by OUTPATH before a pipeline process will begin doing work. While less than this number of blocks are free, the process remains in IOWAIT state.
OSF_PROCESSING.XX - identifies the OSF stage (XX) and value assigned while a pipeline process is doing work.
OSF_RANK - priority of OSF events in the OPUS event queue (1...N, where N is a positive integer).
OSF_TRIGGERn.XX - where n=1...(Number of triggers); identifies the OSF stage (XX) and value that must be matched before a process begins doing work.
OSF_TRIGGERn.MAXTARGS - where n=1...(Number of triggers); identifies how many OSF's an application will handle per event (the application must support multiple OSF's per event if > 1). If this key is not defined, a single OSF will be included per event.
OSF_ABSENT - defines the modifier applied to the OSF_PROCESSING column of an OSF when a process goes absent during processing of that OSF.
OUTPATH - output path (must be present if MINBLOCKS is defined).
PASSWORD - indicates to the Process Manager (PMG) that it should prompt the user for a password when starting this task. The password entered is encrypted and passed to the task as a pair of extra parameters on the task's COMMAND line. If you include a PASSWORD resource item, you should also include a PASSWORD_PROMPT item, especially if you start multiple processes that require a password.
PASSWORD_PROMPT - displayed by the PMG when the user is asked for a password before starting this task. This value can help clarify for which process the password is needed, and it is also used by the PMG to "group" password-required processes. This grouping results in the user only being prompted for a password once for each unique combination of PASSWORD_PROMPT and node name for a set of processes. The password entered by the user is automatically passed to each of the processes in the same group that are started on the same node. Processes started on the same node which do not provide a PASSWORD_PROMPT resource value are assigned to the default process group, and all get assigned the same password as if they were grouped together by a common blank PASSWORD_PROMPT value.
POLLING_TIME - interval time (seconds) at which the OPUS pipeline checks for a new process "event" (i.e., checks to see if the process has work to do). A good default for this value is 10 seconds. Do not set this to be too small. The normal work flow of the servers can be hampered if they do not have sufficient time to process and distribute events before the next interval starts.
RESUME_BLOCKS - can be used to specify the number of blocks needed for a process to exit IOWAIT state.
START_TIME - time at which the pipeline process will first awaken and run (DDD:HH:MM:SS).
SUB[env_var_name] - where env_var_name is an environment variable. For example, a keyword/value entry in a path file might be written:
DATA_PATH = SUB[DPATH]in order to assign the value of the environment variable DPATH to the path file keyword DATA_PATH. One use for such a facility is to maintain static or configured versions of path files that can be used in different pipelines- the path file values can be tailored to each pipeline through the environment of the process in such a setup.
SYSTEM - the primary key used to sort processes for display in the PMG's process selection dialog. Processes with the same value for SYSTEM are grouped together in this dialog. The order in which SYSTEM groups are displayed depends alphabetically on this value.
TASK - indicates the command syntax that the OPUS pipeline will use to invoke the pipeline process. Use the "-r" option to specify the process resource file rootname. Use the "-p" option to specify the path file rootname.
Side Note: If the process is an internal poller, you may also use the "-ssrl" option in the TASK line to specify a message reporting level (using the MSG_REPORT_LEVEL values, e.g. "-ssrl INFO") for any server-side messages the process may generate. Such server-side messages are sent to a special process log file with a ".ssplog" extension, though they can be turned off with "-ssrl NONE". Client-side message reporting levels for C++ internal pollers are based on the regular MSG_REPORT_LEVEL environment variable. Client-side message reporting levels for any Java internal pollers which use the JMsg Java class are based on that client's specific Java calls to JMsg.setReportLevel(), which, for HST pipeline tasks for example, are based on a resource value called "MSG_RPT_LVL" - you may use your own scheme. Client-side message reporting levels for Python or other internal pollers are not yet defined by OPUS, and so such reporting is completely specified by the Python client code that you write.
TIME_RANK - priority of TIME events in the OPUS event queue (1...N, where N is a positive integer).
XPOLL_ERROR.XX - identifies the OSF stage (XX) and value assigned when the process exit status does not map to the known exit status values given in the XPOLL_STATE.NN keywords.
XPOLL_STATE.NN - maps the process exit status NN (zero-fill, e.g. 01) to a set of process resource file keywords that indicate the pipeline action performed when receiving that exit status.
OSF triggers cause a pipeline process to begin executing when an OSF with certain characteristics is posted on the OPUS blackboard. For example, a telemetry-parsing pipeline stage could be triggered by an OSF indicating that a data-receipt pipeline stage has completed.
File triggers occur when a file matching a certain filename mask appears in a particular directory. For example, a process could be triggered when files named f*.pod appear in the directory /mydata/incoming/.
A time trigger occurs either at specific calendar dates and times, at some delta time offset from the current time, or at some delta time offset from some time in the future. For example, a process could trigger every day at 08:00 AM, while another might trigger every 15 minutes around the clock. Yet another process might trigger every hour starting the following Sunday. Only one time trigger definition is allowed per process resource file.
OSF_RANK = 1 !if multiple trigger types are used in !combination, this determines what order !this trigger will be checked relative to !the others OSF_TRIGGER1.XX = w !when OSF column XX = "w" (waiting) OSF_TRIGGER1.YY = c !and when OSF column YY = "c" (complete) OSF_TRIGGER1.DATA_ID = zzz !and the OSF data id = "zzz"... OSF_PROCESSING.XX = p !set this OSF stage upon being triggeredWhen this combination of OSF characteristics is met by an OSF on the blackboard, the process will trigger (i.e. start up, or awaken). The other columns of the OSF do not matter. Only the columns mentioned in the trigger are checked. Once the process is triggered, the OSF column XX will be set to "p" (processing) to indicate that work is being done. This will also prevent the same OSF from triggering other copies of this process, if multiple copies of this process are currently running.
The OSF_RANK, OSF_TRIGGERn.XX, and OSF_PROCESSING.xx keys are required to form a valid OSF trigger.
FILE_RANK = 1 !if multiple trigger types are used in !combination, this determines what order !this trigger will be checked relative to !the others FILE_DIRECTORY1 = /mydisk/mydata/ !incoming data found here FILE_OBJECT1 = *.incoming !trigger on files that match the pattern FILE_PROCESSING = _proc ! file renamed to include this on end ! when found (prevents retriggering) FILE_SUCCESS.DIRECTORY = /mydisk/done/ ! If successful, file is moved ! here (still has _proc on name) FILE_ERROR.DIRECTORY = /mydisk/bad/ ! If unsuccessful, file is moved ! here (still has _proc on name)
The FILE_RANK, FILE_DIRECTORYn, FILE_OBJECTn, FILE_PROCESSING, FILE_SUCCESS, and FILE_ERROR keys are required to form a valid file trigger.
TIME_RANK = 1 !if multiple trigger types are used in !combination, this determines what order !this trigger will be checked relative to !the others DELTA_TIME = 000:00:30:00 !trigger every 30 minutes from the initial !time at which the process is started
TIME_RANK = 1 !if multiple trigger types are used in !combination, this determines what order !this trigger will be checked relative to !the others DELTA_TIME = 001:00:00:00 !trigger every 24 hours from the initial !time at which the process is started
TIME_RANK = 1 !if multiple trigger types are used in !combination, this determines what order !this trigger will be checked relative to !the others START_TIME = 12:00:00 !start triggering at 12 noon, local time DELTA_TIME = 000:01:00:00 !trigger every hour from the START_TIME !onward
FILE_RANK = 1 !Multiple trigger types are being used in !combination: this determines that the FILE !trigger will be checked before the OSF !trigger. FILE_DIRECTORY1 = /mydisk/mydata/ !Incoming data found here FILE_OBJECT1 = luv*.incoming ! trigger on files that match this pattern FILE_DIRECTORY2 = /mydisk/mydata/ !Incoming data found here FILE_OBJECT2 = war*.incoming ! trigger on files that match this pattern FILE_PROCESSING = _proc !File renamed to include this on end ! when found (prevents retriggering) FILE_SUCCESS.DIRECTORY = /mydisk/done/ !If successful, file is moved ! here (still has _proc on name) FILE_ERROR.DIRECTORY = /mydisk/bad/ !If unsuccessful, file is moved here but FILE_ERROR = _bad ! _proc on name is replaced with this string OSF_RANK = 2 !Multiple trigger types are being used in !combination: this determines that the OSF !trigger will be checked after !the FILE trigger OSF_TRIGGER1.XX = c !When when OSF column XX = "c" (complete) OSF_TRIGGER1.YY = s ! and OSF stage (column) YY = "s" (start) OSF_TRIGGER1.DATA_ID = zzz ! and the OSF data id = "zzz"... OSF_PROCESSING.YY = p ! set this OSF stage to processingWhen this process is started, OPUS will first look in FILE_DIRECTORY1 for files matching FILE_OBJECT1 and process them, if found. When no matching files are found (i.e. all matching files were processed, or if none were ever found), it next looks in FILE_DIRECTORY2 for for files matching FILE_OBJECT2 and repeats the procedure. Lastly, OPUS will look at the OSFs and if an OSF is found with the specified DATA_ID and stage XX is 'c' AND stage YY is 's', it will set the YY column to 'p' and process the data for that OSF.
Note that only one FILE_RANK key is required although there are more than one pairs of FILE_DIRECTORYn and FILE_OBJECTn keys in that trigger group. Complete FILE_DIRECTORYn and FILE_OBJECTn pairs are required, even if some information is the shared by multiple keys.
For example, given an external polling process triggering on OSF stage GC, the following keywords define three possible exit status values (1, 3, and 5) and their mappings to other keywords in the process resource file as well as how the GC OSF stage should be updated for exit status values other than 1, 3, or 5:
XPOLL_STATE.01 = OSF_SUCCESS1 XPOLL_STATE.03 = OSF_ERROR1 XPOLL_STATE.05 = OSF_SUCCESS2 XPOLL_ERROR.GC = x !any exit status != 1,3,5 sets OSF stage !GC to xThe corresponding keywords that map the exit status values to OSF update actions might look like:
OSF_SUCCESS1.GC = c !if exit status=1 set OSF stage GC to c... OSF_SUCCESS1.CA = n !...and set OSF stage CA to n (not necessary) OSF_SUCCESS2.GC = c !if exit status=5 set OSF stage CA to c... OSF_SUCCESS2.CA = w !...and set OSF stage CA to w (waiting) OSF_ERROR1.GC = e !if exit status=3 set OSF stage GC to eIn this case, should the external polling process exit with status = 1, the GC stage of the OSF it was working on would be updated to "c" (for complete, perhaps) and the CA stage would be updated to "n", possibly indicating that the CA stage is not necessary for this data set. An exit status of 5 results in a similar update except that CA is changed to "w". On the other hand, should an exit status of 3 be returned, the GC stage of the OSF would be set to "e" indicating that an error occurred during processing; the CA stage would not be changed since GC did not complete normally. Finally, an exit status not equal to 1, 3, or 5 would result in the GC stage being marked "x".
The process must define EVENT_STATE_FILE in the environment (the best way to do this is through an ENV. entry in the process resource file). The value of this environment variable should be the name of a file in OPUS_HOME_DIR in which the external poller will report the individual exit status values for each item in the event.
The external poller must place one line per event item in this file in the following format:
OSF_STATUS = s0 OSF_STATUS1 = s1 . . . OSF_STATUSn = s(n-1)for OSF events or
EVENT_STATUS = s0 EVENT_STATUS1 = s1 . . . EVENT_STATUSn = s(n-1)for file events where n is the number of items in the event, the OSF_STATUS or EVENT_STATUS definitions in the file correspond to the items defined in the environment of the same name, and s0...s(n-1) are the exit status values to be mapped for that item.
Finally, the external poller must signal that the file should be read by exiting with a status that maps to an XPOLL_STATE value of CHECK_FILE. For example, if the process resource file defines
XPOLL_STATE.01 = CHECK_FILEthe external poller should exit with a value of 1 in order to have the file read.
The FILE_ACTION keyword can be defined for a process with a file trigger. The value of this keyword should specify a script or process to run if the FILE_SUCCESS status is applied. The command-line arguments for the script can include the token ^f or ^F for which the file name that was processed is substituted.
For example, an entry of the form:
FILE_ACTION = '/bin/rm -f ^f'will cause the program /bin/rm to run if FILE_SUCCESS is signaled upon completion of file event processing. The file name is substituted in place of ^f, so this file action results in deletion of the file.
Optionally, the keyword FILE_ACTION_OK can accompany the definition of FILE_ACTION to define the expected return status of the file action command. If the return status does not match the value of FILE_ACTION_OK, an error message is reported in the process log file.
ENV.OUTPATH = DATOUTand this path file entry
DATOUT /mydisk/mydata/the value of OUTPATH is translated "/mydisk/data/". If the DATOUT entry in the path file were removed, the value of OUTPATH would not be translated the second time and would remain equal to "DATOUT". Path file keyword entries cannot be set as environment variables directly using ENV.; this only works for process resource file keywords.
The path file usually provides a mapping of symbolic names listed in the process resource file entries to specific file paths. This allows a pipeline process to be run on different types of data with different input and output directories (i.e., in parallel pipelines) without altering the process resource file values. For example, STScI runs a real-time pipeline on one path and a production pipeline on another path using the same set of process resource files: all pipeline-specific data are in the path files and are retrieved by the processes through indirection. The path file construct increases the flexibility of OPUS, particularly in multiple pipelines environments.
DATABASE = PATH_DATABASEand one in a path file like
PATH_DATABASE = REQUIREDresults in a value of REQUIRED for DATABASE when the process is run in this path.
To override the value of a keyword in a process resource file for a path, you would prepend the name of the process to the keyword in the path file and supply the overriding value. For example, a path file might contain the entry:
g2f.DATABASE = OPTIONALNo matter what value is assigned to DATABASE in the process resource file of g2f, its value will be OPTIONAL when run in this path. If you wish to override the process resource keyword value for all processes in a particular path, you can use "*" as the process name prefix to the keyword in the path file as in:
*.DATABASE = PROHIBITED
The process resource file keyword value should have a symbolic name. That name should appear as a keyword in the path file. Different path files, then, can assign different values to that symbolic name.
However, if the process resource file keyword values for COMMAND (external pollers), TASK or FILE_ACTION contain the substring SUB[env_var_name], where env_var_name is the name of an environment variable, then the value of that environment variable is substituted for the SUB expression before the command is executed.
This same mechanism is supported for all path file values, so a process resource file can obtain values from the environment indirectly using the technique described here.
There is a built-in keyword pair designed to provide a disk space check. By specifying the keywords,
MINBLOCKS = 50000 !number of free 512-byte blocks required on !output device OUTPATH = /mydir/outgoing !directory where output products are placedThe OPUS system will automatically verify that sufficient disk space exists on the output device defined by OUTPATH before each "event" (new input file available, or previous pipeline step complete, or process wake-up time arrives). If sufficient free space does NOT exist, the pipeline process will be suspended until enough space is made available. The pipeline process status entry will have its status field set to "iowait" in this case. This will show up in the Process Manager (PMG) display.
Be careful in using this capability. If you have seven of the same process running in the same pipeline, and if each checks for a value of MINBLOCKS which is just sufficient for its own output, then what if all seven run simultaneously? If there is only enough space for the output of a single process the other tasks may run out of space in processing their data.