Fast/Unload SOUL Interface: Difference between revisions

Revision as of 21:58, 16 March 2015

Introduction

The Fast/Unload SOUL Interface, purchased as a separate Fast/Unload option, allows one to invoke Fast/Unload from a SOUL program. With this approach, an application builds a set of records to be unloaded, using standard SOUL statements. For example, you can reduce unload time by restricting the set of records using indexed Model 204 fields.

There are two ways to invoke Fast/Unload from a SOUL program:

By passing the label of the record set or list to the $FunLoad function.
By invoking the FastUnload or the FastUnloadTask method against a Recordset object.

$Funload or the FastUnload and FastUnloadTask methods send the record set to the Fast/Unload PST, which invokes the Fast/Unload load module in a subtask (or PST, under CMS) of Model 204. The full power of the FUEL language is available, including sorts, UAI, etc.

When invoked via $Funload or the FastUnload or FastUnloadTask methods, Fast/Unload only accesses the records in the passed record set, and only issues reads for disk tracks containing those records. Whether or not the Fast/Unload SOUL Interface is employed, Fast/Unload always issues reads only for disk tracks containing the records accessed, but, except for records skipped by the EVERY and SKIPREC parameters, a standalone Fast/Unload accesses all Table B records in a file until Fast/Unload processing ends.

The Fast/Unload SOUL Interface also provides the ability to unload data from a Model 204 group; standalone Fast/Unload provides that functionality by allowing multiple Model 204 file names on the OPEN statement.

Note that the Fast/Unload SOUL Interface allows unloading to be performed (at the thread level) either of these ways:

Synchronously, so that all processing is completed before control is returned from $Funload or the FastUnload and FastUnloadTask methods.
Asynchronously, so that control is returned to the SOUL program as soon as the $function or method parameters are verified and accepted.

Note: Fast/Unload processing is never synchronous at the Online level: threads not waiting for the completion of a Fast/Unload SOUL Interface requests will continue to run while one or more unloads are progressing. When processing is asynchronous (at the thread level), it is up to the user to verify successful completion of the unload much as a user would verify successful completion of a batch job.

A SOUL procedure is provided as part of the installation process to monitor and control Fast/Unload requests from the ONLINE region. Specifically, a system manager can cancel or examine Fast/Unload requests. In addition, each user can perform the same functions on asynchronous requests that he or she initiated.

To make it easier to diagnose problems where Fast/Unload SOUL Interface is involved, under Sirius Mods version 6.7 and and Model 204 version 7.5 and later, messages are also sent to the Model 204 journal/audit trail at each Fast/Unload request made and at each completed by the Fast/Unload SOUL Interface.

The message when the requests are started looks like:

MSIR.0890: Asynchronous request 2 made by $funload

And the message when the requests are completed looks like:

MSIR.0891: Asynchronous request 2 completed, RC = 0

When to use the Fast/Unload SOUL Interface

If you have the Fast/Unload SOUL Interface, you can use it, rather then a standalone Fast/Unload, to perform a given unload. In fact, you can usually use the Fast/Unload SOUL Interface and expect it will run as well or better than the comparable standalone Fast/Unload. Many factors make use of the Fast/Unload SOUL Interface clearly advantageous over a standalone Fast/Unload, such as:

Sparse index-driven record set	If you can use the `FIND` statement with indexed fields (that is, no Table B scan) to determine a small fraction of the file which then can be processed by `Fast/Unload`, this will be a significant performance advantage.
`Model 204` Boolean processing	The capabilities of `Model 204` record list processing, and the ability to combine sets of records in the `FIND` statement, provide not only efficient index-driven record restriction, but also very powerful application capabilities.
File enqueue	When a file is open for update in an ONLINE environment, the only way to access that file in a standalone job is to bypass file enqueueing (with the `NOENQ` parameter, as described in NOEnq). A standalone `Fast/Unload` with `NOENQ` will only examine pages that have been written to disk, and so will not see any modified pages still in the buffer pool. Performing the unload with the `Fast/Unload SOUL Interface` allows you to use normal `Model 204` record-level locking, so a file can be unloaded while it is open for update, obtaining a current image of all pages accessed. There is a tradeoff, of course, involved in holding record locks over the duration of an unload job. You can use an unlocked set of records (record lists, or the SOUL `FIND WITHOUT LOCKS` statement) with the `Fast/Unload SOUL Interface`, and the current disk buffer page image is still used. Note, however, that unloading without a locked record set while the file is open for update is suitable for creating some kind of report or other file extract that can tolerate a degree of inconsistent data, but it is generally not advisable if you are unloading in order to reorganize the file.
Groups (prior to version 4.4)	The `Fast/Unload SOUL Interface` allows you to process a set of records derived from one or more files in a permanent, temporary, or ad hoc `Model 204` group. Prior to version 4.4, a standalone `Fast/Unload` can only access the single file designated on the `Fast/Unload` `OPEN` directive.
SOUL pre-/post-processing, scheduling	You can use the features of SOUL to prepare the `Fast/Unload` input program. The program can be dynamically generated, or static and stored in an `Html` or `Text` statement block or in a separate procedure that is read with `$ProcOpn` and `$ProcDat`. The `Html` or `Text` block is particularly useful for generating `Fast/Unload` input programs that are largely static but have some dynamic parts. The `Fast/Unload SOUL Interface` can be used to manipulate and combine the results of one or more `$Funload` or `FastUnload` or `FastUnloadTask` calls. Besides offering the convenience of post-processing with SOUL, this can be useful for merging, matching, or cross-checking applications. The results can be processed conveniently and efficiently with $lists, with the `$FunImg` function, with `Stringlist` objects, or with a `FastUnloadTask` object. You can combine scheduling and control of all "background" processing in your `Model 204` Online with initiating and controlling `Fast/Unload SOUL Interface` processing.

The CPU time consumed by the Fast/Unload task running under Model 204 will not interfere with online Model 204 users: the z/OS dispatching priority of the Fast/Unload task is 4 less than that of the Model 204 main task.

In some cases, the only appropriate environment to run Fast/Unload is a batch environment, but you can still obtain the relevant advantages of the Fast/Unload SOUL Interface by invoking it in a single-user Model 204 job ("BATCH204"). Some of the reasons for running in a batch environment are:

Virtual storage use	The virtual storage requirements of a `Fast/Unload` task running under `Model 204` could significantly affect the virtual storage requirements of the whole region, especially if an external sort is being invoked by `Fast/Unload` as part of its processing. Care should be taken to ensure that these requirements do not cause paging in the `Model 204` region, paging having potentially disastrous performance impact on a multi-user `Model 204` region.
Below-the-line storage	If the below-the-line storage requirements of `Fast/Unload` are large, it may need to run in a separate address space from a `Model 204` Online job, and if they are extremely demanding, this might not even permit a `BATCH204` environment.
Communication for current buffers	As mentioned in the table above under "File enqueue", the `Fast/Unload SOUL Interface` provides the current copy of any modified pages that the `Fast/Unload` subtask accesses. If a very large number of modified pages are in the buffer pool when `$Funload` or the `FastUnload` or `FastUnloadTask` method is called, this could, in some cases, lead to a noticable amount of overhead. One measure of this would be the PST wait time, described in Fast/Unload job statistics. If this overhead is significant, you might evaluate the tradeoffs in either unloading the file in a standalone unload with the `NOENQ` parameter, or unloading it when there are fewer dirty pages in the buffer pool, for example, when updating activity is lower.

Setting up the Fast/Unload SOUL Interface environment

To use the Fast/Unload SOUL Interface the system manager must perform the following steps:

Create a custom ONLINE or BATCH204 load module that has the Sirius Functions linked in.
Concatenate the load library containing the Fast/Unload load module to the load library containing the ONLINE or BATCH204 load module on the STEPLIB DD, or copy the Fast/Unload load module into the load library containing the ONLINE or BATCH204 load module.
For example, if the Model 204 ONLINE load module resides in M204.LOADLIB, and the Fast/Unload load module resides in SIRIUS.LOAD, then this is an appropriate STEPLIB DD for the ONLINE JCL:

//STEPLIB DD DSN=M204.LOADLIB,DISP=SHR // DD DSN=SIRIUS.LOAD,DISP=SHR

Under CMS, the Fast/Unload TEXT file must be on a disk accessed by the Model 204 ONLINE service machine.
Modify the JCL (or EXEC) used to run the Model 204 ONLINE or BATCH204 so that it invokes the custom load module from step one, and so that it contains a DD card (or FILEDEF statement) for the Fast/Unload audit trail. This audit trail has DDNAME FUNAUDIT. For initial testing purposes, this DD could simply specify SYSOUT=*.
Modify the User 0 parm card to indicate the maximum number of concurrent Fast/Unload tasks to be allowed. This is specified with the FUNTSKN parameter. This parameter has a default of 0 and a maximum of 64. If this parameter is set to 0, no user will be able to access Fast/Unload via $Funload or the FastUnload or FastUnloadtask method.
In addition, it might be necessary to specify the name of the Fast/Unload load module (or TEXT file, in CMS). This is done using the FUNPGM parameter. The default value of FUNPGM is FUNLOAD.

For example, the following would be valid parameters on User 0's parm card:

FUNTSKN=8,FUNPGM='MYFUN'
The system manager might also need to increase the value of NSUBTKS on User 0's parm card. Fast/Unload requires a PST, and when running under CMS an additional PST, for every active Fast/Unload task. Thus, the NSUBTKS requirement is increased by 1 for Fast/Unload under z/OS, and by 1+FUNTSKN under CMS.
The one exception to this rule is that when Fast/Unload is to be invoked in a single-user run, you can avoid the overhead of using the multi-user scheduler by setting NSUBTKS to 0. This causes the single user to perform the work of the Fast/Unload PST whenever a $FunWait, $FunImg, $FunsStr, or $FunSkip is executed.

Once these tasks have been performed by the system manager, Fast/Unload is ready for use by the programmer. Communication with Fast/Unload is achieved by the SOUL $functions or methods that are provided when a site purchases the Fast/Unload SOUL Interface.

System parameters for the Fast/Unload SOUL Interface

The parameters described in the following subsections provide useful controls for Fast/Unload SOUL Interface requests.

FUNPARM

The FUNPARM system parameter is a standard Model 204 bitmask-style parameter. Setting the X'01' bit specifies that a synchronous Fast/Unload request is not to be allowed while an updating transaction is active. This is to prevent a Fast/Unload request that might take a long time to complete from being run while a user has resources enqueued for an updating transaction. These resources would, of course, include the blocking of checkpoints.

If the FUNPARM X'01' bit is set, and a thread attempts a synchronous Fast/Unload SOUL Interface request (with $Funload or the FastUnload method of the Recordset class) in the middle of an updating transaction, the transaction is cancelled with a message like the following:

CANCELLING REQUEST: MSIR.0561: $FUNLOAD: Synchronous request during update transaction in line 43, procedure FUNTEST, file ALEXPROC

The X'02', X'04', and X'08' bits of FUNPARM are supported as of version 4.7 of Fast/Unload. These settings control the Model 204 auditing of Fast/Unload messages.

FUNPRMU

The FUNPRMU system parameter is a standard Model 204 bitmask-style parameter, supported as of version 4.7 of Fast/Unload.

The X'02', X'04', and X'08' bits of FUNPRMU control the Model 204 auditing of Fast/Unload messages. As described in Fast/Unload report processing, FUNPRMU (and FUNPARM) allow the auditing of some of the messages from the Fast/Unload SOUL Interface job report to be captured on FUNAUDIT and/or the Model 204 audit trail, regardless of the report destination specified in the SOUL invocation of the Fast/Unload SOUL Interface.

FUNMAXT

The FUNMAXT system parameter specifies the maximum amount of time, in seconds, a Fast/Unload SOUL Interface request is to be given to complete. The timer begins when the Fast/Unload SOUL Interface is requested, by $Funload or by the Recordset class FastUnload or FastUnloadTask method.

FUNMAXT is a numeric parameter with valid values from 0 to 36000. The default value, 0, means no time limit is placed on Fast/Unload SOUL Interface requests.

The purpose of FUNMAXT is to prevent user requests from being "hung up" indefinitely while queuing for busy Fast/Unload tasks or for unintentionally long-running requests.

To override FUNMAXT for specific requests, you can use either of these:

The MaxTime named parameter on the FastUnload and FastUnloadTask methods in the Recordset class.
The sixth parameter on $Funload:
* Make sure request completes in one minute %rc = %rs:fastUnload(%listi, %listo, - parameters='NEBUFF=10', maxTime=60) ... * Make sure request completes in one minute %rc = $funload('LABEL', %iList, %oList, , 'NEBUFF=10', 60)

It is a reasonable strategy to set FUNMAXT to a fairly low value, then to selectively set it higher for requests that need more time. Of course, it can be very difficult to ensure that short-running requests complete quickly if the Online also has long-running requests that might tie up all the Fast/Unload tasks.

You improve the odds of quick completion if you specify more Fast/Unload tasks (FUNTSKN bigger), but this may still not be enough:

All tasks might be tied up anyway, if there are many long-running requests.
Some of the Fast/Unload tasks might have trouble getting dispatched, because there are more of them than CPUs to run them.

@@ Line 260: / Line 260: @@
 ==See also==
+<p><b>SOUL interface topics</b></p>
+{{Template:SOUL Interface topic list}}
+<p><b>Fast/Unload interface topics</b></p>
 {{Template:Fast/Unload topic list}}
 [[Category:Fast/Unload SOUL Interface]]