Fast/Unload SOUL Interface

From m204wiki
Revision as of 19:30, 17 June 2015 by JAL (talk | contribs) (maintask instead of main task)
Jump to navigation Jump to search

Introduction

The Fast/Unload SOUL Interface (FUSI), purchased as a separate Fast/Unload option, allows one to invoke Fast/Unload from a SOUL program. With this approach, an application builds a set of records to be unloaded, using standard SOUL statements. For example, you can reduce unload time by restricting the set of records using indexed Model 204 fields.

There are two ways to invoke Fast/Unload from a SOUL program:

$Funload or the FastUnload and FastUnloadTask methods send the record set to the Fast/Unload PST, which invokes the Fast/Unload load module in a subtask (or PST, under CMS) of Model 204. The full power of the FUEL language is available, including sorts, UAI, etc.

When invoked via $Funload or the FastUnload or FastUnloadTask methods, Fast/Unload only accesses the records in the passed record set, and it only issues reads for disk tracks containing those records. Whether or not the FUSI is employed, Fast/Unload always issues reads only for disk tracks containing the records accessed, but, except for records skipped by the EVERY and SKIPREC parameters, a standalone Fast/Unload accesses all Table B records in a file until Fast/Unload processing ends.

The FUSI also provides the ability to unload data from a Model 204 group; standalone Fast/Unload provides that functionality by allowing multiple Model 204 file names on the OPEN statement.

Note that the FUSI allows unloading to be performed (at the thread level) either of these ways:

  • Synchronously, so that all processing is completed before control is returned from $Funload or the FastUnload and FastUnloadTask methods.
  • Asynchronously, so that control is returned to the SOUL program as soon as the $function or method parameters are verified and accepted.

Note: Fast/Unload processing is never synchronous at the Online level: threads not waiting for the completion of a FUSI request will continue to run while one or more unloads are progressing. When processing is asynchronous (at the thread level), it is up to the user to verify successful completion of the unload, much as a user would verify successful completion of a batch job.

A SOUL procedure is provided as part of the installation process to monitor and control Fast/Unload requests from the ONLINE region. Specifically, a system manager can cancel or examine Fast/Unload requests. In addition, each user can perform the same functions on asynchronous requests that he or she initiated.

To make it easier to diagnose problems where the FUSI is involved, messages are also sent to the Model 204 journal/audit trail at each Fast/Unload request made and at each completed by the FUSI.

The message when the requests are started looks like:

MSIR.0890: Asynchronous request 2 made by $funload

And the message when the requests are completed looks like:

MSIR.0891: Asynchronous request 2 completed, RC = 0

When to use the FUSI

If you have the FUSI, you can use it, rather then a standalone Fast/Unload, to perform a given unload. In fact, you can usually use the FUSI and expect it will run as well or better than the comparable standalone Fast/Unload. Many factors make use of the FUSI clearly advantageous over a standalone Fast/Unload, such as:

Sparse index-driven record set If you can use the FIND statement with indexed fields (that is, no Table B scan) to determine a small fraction of the file which then can be processed by Fast/Unload, this will be a significant performance advantage.
Model 204 Boolean processing The capabilities of Model 204 record list processing, and the ability to combine sets of records in the FIND statement, provide not only efficient index-driven record restriction, but also very powerful application capabilities.
File enqueue When a file is open for update in an ONLINE environment, the only way to access that file in a standalone job is to bypass file enqueueing (with the NOENQ parameter, as described in NOEnq). A standalone Fast/Unload with NOENQ will only examine pages that have been written to disk, and so will not see any modified pages still in the buffer pool.

Performing the unload with the FUSI allows you to use normal Model 204 record-level locking, so a file can be unloaded while it is open for update, obtaining a current image of all pages accessed. There is a tradeoff, of course, involved in holding record locks over the duration of an unload job. You can use an unlocked set of records (record lists, or the SOUL FIND WITHOUT LOCKS statement) with the FUSI, and the current disk buffer page image is still used. Note, however, that unloading without a locked record set while the file is open for update is suitable for creating some kind of report or other file extract that can tolerate a degree of inconsistent data, but it is generally not advisable if you are unloading in order to reorganize the file.

Groups (prior to version 4.4) The FUSI allows you to process a set of records derived from one or more files in a permanent, temporary, or ad hoc Model 204 group. Prior to version 4.4, a standalone Fast/Unload can only access the single file designated on the Fast/Unload OPEN directive.
SOUL pre-/post-processing, scheduling You can use the features of SOUL to prepare the Fast/Unload input program. The program can be dynamically generated, or static and stored in an Html or Text statement block or in a separate procedure that is read with $ProcOpn and $ProcDat. The Html or Text block is particularly useful for generating Fast/Unload input programs that are largely static but have some dynamic parts.

The FUSI can be used to manipulate and combine the results of one or more $Funload or FastUnload or FastUnloadTask calls. Besides offering the convenience of post-processing with SOUL, this can be useful for merging, matching, or cross-checking applications. The results can be processed conveniently and efficiently with $lists, with the $FunImg function, with Stringlist objects, or with a FastUnloadTask object.

You can combine scheduling and control of all "background" processing in your Model 204 Online with initiating and controlling FUSI processing.

The CPU time consumed by the Fast/Unload task running under Model 204 will not interfere with online Model 204 users: the z/OS dispatching priority of the Fast/Unload task is 4 less than that of the Model 204 maintask.

In some cases, the only appropriate environment to run Fast/Unload is a batch environment, but you can still obtain the relevant advantages of the FUSI by invoking it in a single-user Model 204 job ("BATCH204"). Some of the reasons for running in a batch environment are:

Virtual storage use The virtual storage requirements of a Fast/Unload task running under Model 204 could significantly affect the virtual storage requirements of the whole region, especially if an external sort is being invoked by Fast/Unload as part of its processing. Care should be taken to ensure that these requirements do not cause paging in the Model 204 region, paging having potentially disastrous performance impact on a multi-user Model 204 region.
Below-the-line storage If the below-the-line storage requirements of Fast/Unload are large, it may need to run in a separate address space from a Model 204 Online job, and if they are extremely demanding, this might not even permit a BATCH204 environment.
Communication for current buffers As mentioned in the table above under "File enqueue", the FUSI provides the current copy of any modified pages that the Fast/Unload subtask accesses. If a very large number of modified pages are in the buffer pool when $Funload or the FastUnload or FastUnloadTask method is called, this could, in some cases, lead to a noticable amount of overhead.

One measure of this would be the PST wait time, described in Fast/Unload job statistics. If this overhead is significant, you might evaluate the tradeoffs in either unloading the file in a standalone unload with the NOENQ parameter, or unloading it when there are fewer dirty pages in the buffer pool, for example, when updating activity is lower.

Setting up the FUSI environment

Note: Using the FUSI with Model 204 version 7.4 requires Fast/Unload version 4.6 or higher.

To use the FUSI, the system manager must perform the following steps:

  1. If using a version of Model 204 prior to 7.5, include the Sirius Mods in your ONLINE or BATCH204 load module.
  2. Concatenate the load library containing the Fast/Unload load module to the load library containing the ONLINE or BATCH204 load module on the STEPLIB DD, or copy the Fast/Unload load module into the load library containing the ONLINE or BATCH204 load module.

    For example, if the Model 204 ONLINE load module resides in M204.LOADLIB, and the Fast/Unload load module resides in SIRIUS.LOAD, then this is an appropriate STEPLIB DD for the ONLINE JCL:

    //STEPLIB DD DSN=M204.LOADLIB,DISP=SHR // DD DSN=SIRIUS.LOAD,DISP=SHR

    Under CMS, the Fast/Unload TEXT file must be on a disk accessed by the Model 204 ONLINE service machine.

  3. Modify the JCL (or EXEC) used to run the Model 204 ONLINE or BATCH204 so that it invokes the custom load module from step one, and so that it contains a DD card (or FILEDEF statement) for the Fast/Unload audit trail. This audit trail has DDNAME FUNAUDIT. For initial testing purposes, this DD could simply specify SYSOUT=*.
  4. Modify the User 0 parm card to indicate the maximum number of concurrent Fast/Unload tasks to be allowed. This is specified with the FUNTSKN parameter. This parameter has a default of 0 and a maximum of 64. If this parameter is set to 0, no user will be able to access Fast/Unload via $Funload or the FastUnload or FastUnloadtask method.

    In addition, it might be necessary to specify the name of the Fast/Unload load module (or TEXT file, in CMS). This is done using the FUNPGM parameter. The default value of FUNPGM is FUNLOAD.

    For example, the following would be valid parameters on User 0's parm card:

    FUNTSKN=8,FUNPGM='MYFUN'

  5. The system manager might also need to increase the value of NSUBTKS on User 0's parm card. Fast/Unload requires a PST, and when running under CMS an additional PST, for every active Fast/Unload task. Thus, the NSUBTKS requirement is increased by 1 for Fast/Unload under z/OS, and by 1+FUNTSKN under CMS.

    The one exception to this rule is that when Fast/Unload is to be invoked in a single-user run, you can avoid the overhead of using the multi-user scheduler by setting NSUBTKS to 0. This causes the single user to perform the work of the Fast/Unload PST whenever a $FunWait, $FunImg, $FunsStr, or $FunSkip is executed.

Once these tasks have been performed by the system manager, Fast/Unload is ready for use by the programmer. Communication with Fast/Unload is achieved by the SOUL $functions or methods that are provided when your site purchases the FUSI product.

System parameters for the FUSI

The parameters described in the following subsections provide useful controls for FUSI requests.

FUNPARM

The FUNPARM system parameter is a standard Model 204 bitmask-style parameter. Setting the X'01' bit specifies that a synchronous Fast/Unload request is not to be allowed while an updating transaction is active. This is to prevent a Fast/Unload request that might take a long time to complete from being run while a user has resources enqueued for an updating transaction. These resources would, of course, include the blocking of checkpoints.

If the FUNPARM X'01' bit is set, and a thread attempts a synchronous FUSI request (with $Funload or the FastUnload method of the Recordset class) in the middle of an updating transaction, the transaction is cancelled with a message like the following:

CANCELLING REQUEST: MSIR.0561: $FUNLOAD: Synchronous request during update transaction in line 43, procedure FUNTEST, file ALEXPROC

The X'02', X'04', and X'08' bits of FUNPARM control the Model 204 auditing of Fast/Unload messages. As described in Fast/Unload report processing, FUNPARM (and the FUNPRMU user parameter) allow the auditing of some of the messages from the FUSI job report to be captured on FUNAUDIT and/or the Model 204 audit trail, regardless of the report destination specified in the SOUL invocation of the FUSI. The X'02', X'04', and X'08' bits of FUNPARM are supported as of version 4.7 of Fast/Unload.

FUNMAXT

The FUNMAXT system parameter specifies the maximum amount of time, in seconds, a FUSI request is to be given to complete. The timer begins when the FUSI is requested, by $Funload or by the Recordset class FastUnload or FastUnloadTask method.

FUNMAXT is a numeric parameter with valid values from 0 to 36000. The default value, 0, means no time limit is placed on FUSI requests.

The purpose of FUNMAXT is to prevent user requests from being "hung up" indefinitely while queuing for busy Fast/Unload tasks or for unintentionally long-running requests.

To override FUNMAXT for specific requests, you can use either of these:

  • The MaxTime named parameter on the FastUnload and FastUnloadTask methods in the Recordset class.
  • The sixth parameter on $Funload:

    * Make sure request completes in one minute %rc = %rs:fastUnload(%listi, %listo, - parameters='NEBUFF=10', maxTime=60) ... * Make sure request completes in one minute %rc = $funload('LABEL', %iList, %oList, , 'NEBUFF=10', 60)

It is a reasonable strategy to set FUNMAXT to a fairly low value, then to selectively set it higher for requests that need more time. Of course, it can be very difficult to ensure that short-running requests complete quickly if the Online also has long-running requests that might tie up all the Fast/Unload tasks.

You improve the odds of quick completion if you specify more Fast/Unload tasks (FUNTSKN bigger), but this might still not be enough:

  • All tasks might be tied up anyway, if there are many long-running requests.
  • Some of the Fast/Unload tasks might have trouble getting dispatched, because there are more of them than CPUs to run them.

See also

Fast/Unload SOUL Interface topics

Fast/Unload topics

Retrieved from "https://m204wiki.rocketsoftware.com/index.php?title=Fast/Unload_SOUL_Interface&oldid=78005"