Fast/Unload SOUL Interface
The Fast/Unload SOUL Interface, purchased as a separate Fast/Unload option, allows one to invoke Fast/Unload from a SOUL program. With this approach, an application builds a set of records to be unloaded, using standard SOUL statements; for example, you can reduce unload time by restricting the set of records using indexed Model 204 fields.
There are two ways two invoke Fast/Unload from a SOUL program:
- By passing the label of the record set or list to the $Funload function.
- By invoking the FastUnload or the FastUnloadTask method against a Recordset object.
$Funload and the FastUnload/FastUnloadTask methods must be linked into the ONLINE or BATCH204 load module as part of the Model 204 installation process. $Funload or the FastUnload/FastUnloadTask methods send the record set to the Fast/Unload PST, which invokes the Fast/Unload load module in a subtask (or PST, under CMS) of Model 204. The full power of the FUEL language is available, including sorts, UAI, etc.
When invoked via $Funload or the FastUnload/FastUnloadTask methods, Fast/Unload only accesses the records in the passed record set, and only issues reads for disk tracks containing those records. Whether or not the Fast/Unload SOUL Interface is employed, Fast/Unload always issues reads only for disk tracks containing the records accessed, but, except for records skipped by the EVERY and SKIPREC parameters, a standalone Fast/Unload accesses all Table B records in a file until Fast/Unload processing ends.
The Fast/Unload SOUL Interface also provides the ability to unload data from a Model 204 group.
Note that the Fast/Unload SOUL Interface allows unloading to be performed (at the thread level) either synchronously, so that all processing is completed before control is returned from $Funload or the FastUnload/FastUnloadTask methods, or asynchronously so that control is returned to the User Language program as soon as the $function or method parameters are verified and accepted. (Fast/Unload processing is never synchronous at the Online level: threads not waiting for the completion of a Fast/Unload SOUL Interface requests will continue to run while one or more unloads are progressing.) When processing is asynchronous (at the thread level) it is up to the user to verify successful completion of the unload much as a user would verify successful completion of a batch job.
A User Language procedure is provided as part of the installation process to monitor and control Fast/Unload requests from the ONLINE region. Specifically, a system manager can cancel or examine Fast/Unload requests. In addition, each user can perform the same functions on asynchronous requests that he or she initiated.
To make it easier to diagnose problems where Fast/Unload SOUL Interface is involved, under Sirius Mods</class> version 6.7 and and Model 204 version 7.5 and later, messages are also sent to the Model 204 journal/audit trail at each Fast/Unload request made and at each completed by the Fast/Unload SOUL Interface.
The message when the requests are started look like:
MSIR.0890: Asynchronous request 2 made by $funload
And the message when the requests are completed look like:
MSIR.0891: Asynchronous request 2 completed, RC = 0
When to use the Fast/Unload SOUL Interface
If you have the Fast/Unload SOUL Interface, you can use it, rather then a standalone Fast/Unload, to perform a given unload; in fact, you can usually use the Fast/Unload SOUL Interface and expect it will run as well or better than the comparable standalone Fast/Unload. There are many factors which make use of the Fast/Unload SOUL Interface clearly advantageous over a standalone Fast/Unload, such as:
| Sparse index-driven record set | If you can use the FIND statement with indexed fields (i.e., no Table B scan) to determine a small fraction of the file which then can be processed by Fast/Unload, this will be a significant performance advantage. |
|---|---|
| Model 204 Boolean processing | The capabilities of Model 204 record list processing, and the ability to combine sets of records in the FIND statement, provide not only efficient index-driven record restriction, but also very powerful application capabilities. |
| File enqueue | When a file is open for update in an ONLINE environment, the only way to access that file in a standalone job is to bypass file enqueueing (with the NOENQ parameter, as described in NOEnq). A standalone Fast/Unload with NOENQ will only examine pages that have been written to disk, and so will not see any modified pages still in the buffer pool. Performing the unload with the Fast/Unload SOUL Interface allows you to use normal Model 204 record-level locking so that a file can be unloaded while it is open for update, obtaining a current image of all pages accessed. There is a tradeoff, of course, involved in holding record locks over the duration of an unload job. You can use an unlocked set of records (record lists, or the User Language FIND WITHOUT LOCKS statement) with the Fast/Unload SOUL Interface, and the current disk buffer page image is still used. Note, however, that unloading without a locked record set while the file is open for update is suitable for creating some kind of report or other file extract that can tolerate a degree of inconsistent data, but is generally not advisable if you are unloading in order to reorganize the file. |
| Groups (prior to version 4.4) | The Fast/Unload SOUL Interface allows you to process a set of records derived from one or more files in a PERManent, TEMPorary, or ad hoc Model 204 group. Prior to version 4.4, a standalone Fast/Unload can only access the single file designated on the Fast/Unload OPEN directive. |
| User Language pre-/post-processing, scheduling | You can use the features of User Language to prepare the Fast/Unload input program. The program can be dynamically generated, or static and stored in an Html/Text block or in a separate procedure that's read with $procopn and $procdat. The Html/Text block is particularly useful for generating Fast/Unload input programs that are largely static but have some dynamic parts. The Fast/Unload SOUL Interface can be used to manipulate and combine the results of one or more $Funload or FastUnload/FastUnloadTask calls. Besides offering the convenience of post-processing with User Language, this can be useful for merging, matching or cross-checking applications. The results can be processed conveniently and efficiently either with $lists, with the $FunImg function, with Stringlist objects, or with a FastUnloadTask object. You can combine scheduling and control of all "background" processing in your Model 204 online with initiating and controlling Fast/Unload SOUL Interface processing. |
The CPU time consumed by the Fast/Unload task running under Model 204 will not interfere with online Model 204 users: the z/OS dispatching priority of the Fast/Unload task is 4 less than that of the Model 204 main task.
In some cases, the only appropriate environment to run Fast/Unload is a batch environment, but you can still obtain the relevant advantages of the Fast/Unload SOUL Interface by invoking it in a single-user Model 204 job ("BATCH204"). Some of the reasons for running in a batch environment are:
| Virtual storage use | The virtual storage requirements of a Fast/Unload task running under Model 204 could significantly affect the virtual storage requirements of the whole region, especially if an external sort is being invoked by Fast/Unload as part of its processing. Care should be taken to ensure that these requirements do not cause paging in the Model 204 region, paging having potentially disastrous performance impact on a multi-user Model 204 region. |
|---|---|
| Below-the-line storage | If the below-the-line storage requirements of Fast/Unload are large, it may need to run in a separate address space from a Model 204 online job, and if they are extremely demanding, this might not even permit a BATCH204 environment. |
| Communication for current buffers | As mentioned above under "File enqueue", the Fast/Unload SOUL Interface provides the current copy of any modified pages that the Fast/Unload subtask accesses. If a very large number of modified pages are in the buffer pool when $Funload or the FastUnload/FastUnloadTask method is called, this could, in some cases, lead to a noticable amount of overhead; one measure of this would be the PST wait time, described in Fast/Unload job statistics. If this overhead is significant, you might evaluate the tradeoffs in either unloading the file in a standalone unload with the NOENQ parameter, or unloading it when there are fewer dirty pages in the buffer pool, for example, when updating activity is lower. |
Setting up the Fast/Unload SOUL Interface environment
To use the Fast/Unload SOUL Interface the system manager must perform the following steps:
- Create a custom ONLINE or BATCH204 load module that has the Sirius Functions linked in.
- Either concatenate the load library containing the Fast/Unload
load module to the load library containing the ONLINE or BATCH204
load module on the STEPLIB
DD or copy the Fast/Unload load module into the load library containing the
ONLINE or BATCH204 load module.
For example, if the Model 204 ONLINE load module
resides in M204.LOADLIB and the Fast/Unload load module resides in
SIRIUS.LOAD then
//STEPLIB DD DSN=M204.LOADLIB,DISP=SHR // DD DSN=SIRIUS.LOAD,DISP=SHR
would be an appropriate STEPLIB DD for the ONLINE JCL.
Under CMS, the Fast/Unload TEXT file must be on a disk accessed by the Model 204 ONLINE service machine.
- Modify the JCL (or EXEC) used to run the Model 204 ONLINE or BATCH204 so that it invokes the custom load module from step one and so that it contains a DD card (or FILEDEF statement) for the Fast/Unload audit trail. This audit trail has DDNAME FUNAUDIT. For initial testing purposes, this DD could simply specify SYSOUT=*.
- Modify the user0 parm card to indicate the maximum number
of concurrent Fast/Unload tasks to be allowed.
This is specified with the
FUNTSKN parameter.
This parameter has a default of 0 and a maximum of 64.
If this parameter is set to 0, no user will be able to access Fast/Unload via
$Funload or the FastUnload/FastUnloadtask methods.
In addition, it might be necessary to specify the name of the
Fast/Unload load module (or TEXT file, in CMS).
This is done using the FUNPGM parameter.
The default value of FUNPGM is 'FUNLOAD'.
For example, the following would be valid parameters on user0's parm card :
FUNTSKN=8,FUNPGM='MYFUN'
- The system manager might also need to increase the value of NSUBTKS on user0's parm card. Fast/Unload requires a PST and when running under CMS an additional PST for every active Fast/Unload task. Thus, the NSUBTKS requirement is increased by 1 for Fast/Unload under z/OS and 1+FUNTSKN under CMS. The one exception to this rule is that when Fast/Unload is to be invoked in a single user run, it is possible to avoid the overhead of using the multi-user scheduler by setting NSUBTKS to 0. This will cause the single user to perform the work of the Fast/Unload PST whenever a $FunWait, $FunImg, $FunSStr or $FunSkip is executed.
Once these tasks have been performed by the system manager, Fast/Unload is ready for use by the programmer. Communication with Fast/Unload is achieved via the SOUL $functions or methods that are provided when a site purchases the Fast/Unload SOUL Interface.
System parameters for the Fast/Unload SOUL Interface
The parameters described in the following subsections provide useful controls for Fast/Unload SOUL Interface requests.
FUNPARM
The FUNPARM system parameter is a standard Model 204 bitmask-style parameter. introduced in Sirius Mods</class> version 6.7. Setting the X'01' bit, the only bit currently defined, specifies that a synchronous Fast/Unload request is not to be allowed while an updating transaction is active. This is to prevent a Fast/Unload request that might take a long time to complete from being run while a user has resources enqueued for an updating transaction. These resources would, of course, include the blocking of checkpoints.
If the FUNPARM X'01' bit is set, and a thread attempts a synchronous Fast/Unload SOUL Interface request (via $Funload or the FastUnload method of the Recordset class) in the middle of an updating transaction, the transaction is cancelled with a message like the following:
CANCELLING REQUEST: MSIR.0561: $FUNLOAD: Synchronous request during update transaction in line 43, procedure FUNTEST, file ALEXPROC
FUNMAXT
The FUNMAXT system parameter specifies the maximum amount of time, in seconds, a Fast/Unload SOUL Interface request is to be given to complete. The timer begins when the Fast/Unload SOUL Interface is requested, either by $Funload or by the Recordset class FastUnload or FastUnloadTask method.
Introduced in Sirius Mods</class> version 6.7, FUNMAXT is a numeric parameter with valid values from 0 to 36000. The default value of 0 means no time limit is placed on Fast/Unload SOUL Interface requests.
The purpose of FUNMAXT is to prevent user requests from being "hung up" indefinitely while queuing for busy Fast/Unload tasks or for unintentionally long-running requests.
To override FUNMAXT for specific requests, you can use either:
- The
MaxTimenamed parameter on the FastUnload and FastUnloadTask methods in the Recordset class. - The sixth parameter on $Funload:
* Make sure request completes in one minute %rc = %rs:fastUnload(%listi, %listo, - parameters='NEBUFF=10', maxTime=60) ... * Make sure request completes in one minute %rc = $funload('LABEL', %iList, %oList, , 'NEBUFF=10', 60)
It is a reasonable strategy to set FUNMAXT to a fairly low value, then to selectively set it higher for requests that need more time. Of course, it can be very difficult to ensure that short-running requests complete quickly if the Online also has long-running requests that might tie up all the Fast/Unload tasks. You improve the odds of quick completion if you specify more Fast/Unload tasks (FUNTSKN bigger), but this may still not be enough:
- All tasks might be tied up anyway, if there are many long-running requests.
- Some of the Fast/Unload tasks might have trouble getting dispatched, because there are more of them than CPUs to run them.