Fast/Unload SOUL Interface: Difference between revisions

From m204wiki
Jump to navigation Jump to search
m (trim parameter info)
 
(2 intermediate revisions by one other user not shown)
Line 112: Line 112:
To use the <var class="product">FUSI</var>, the system manager must perform the following steps:
To use the <var class="product">FUSI</var>, the system manager must perform the following steps:
<ol>
<ol>
<li>If using a version of Model 204 prior to 7.5, include the [[Sirius Mods]] in your <code>ONLINE</code> or <code>BATCH204</code> load module.
<li>If using a version of Model 204 prior to 7.5:
</li>
<ul>
 
<li>Include the [[Sirius Mods]] in your <code>ONLINE</code> or <code>BATCH204</code> load module; a load module with the Sirius Mods is necessary to invoke the FUSI.</li>
<li>Concatenate the load library containing the <var class="product">Fast/Unload</var>
<li>Concatenate the load library containing the <var class="product">Fast/Unload</var>
load module to the load library containing the <code>ONLINE</code> or <code>BATCH204</code> load module on the <code>STEPLIB</code> DD, or copy the <var class="product">Fast/Unload</var> load module into the load library containing the <code>ONLINE</code> or <code>BATCH204</code> load module.
load module to the load library containing the <code>ONLINE</code> or <code>BATCH204</code> load module on the <code>STEPLIB</code> DD, or copy the <var class="product">Fast/Unload</var> load module into the load library containing the <code>ONLINE</code> or <code>BATCH204</code> load module.
Line 125: Line 125:
Under CMS, the <var class="product">Fast/Unload</var> TEXT file must be on a disk accessed by the
Under CMS, the <var class="product">Fast/Unload</var> TEXT file must be on a disk accessed by the
<var class="product">Model 204</var> <code>ONLINE</code> service machine.</p></li>
<var class="product">Model 204</var> <code>ONLINE</code> service machine.</p></li>
</ul>
</li>


<li>Modify the JCL (or EXEC) used to run the <var class="product">Model 204</var> <code>ONLINE</code> or <code>BATCH204</code> so that it invokes the custom load module from step one, and so that it contains a
<li>Modify the JCL (or EXEC) used to run the <var class="product">Model&nbsp;204</var> <code>ONLINE</code> or <code>BATCH204</code> so that it contains a
DD card (or FILEDEF statement) for the <var class="product">Fast/Unload</var> audit trail.
DD card (or FILEDEF statement) for the <var class="product">Fast/Unload</var> audit trail.
This audit trail has DDNAME <code>FUNAUDIT</code>.
This audit trail has DDNAME <code>FUNAUDIT</code>.
For initial testing purposes, this DD could simply specify <code>SYSOUT=*</code>.</li>
For initial testing purposes, this DD could simply specify <code>SYSOUT=*</code>.
<p>See the [[#audFmt|FUSI audit format below]].</p>
<p>See the [[#audFmt|FUSI audit format below]].</p></li>


<li>Modify the User 0 parm card to indicate the maximum number
<li>Modify the User 0 parm card to indicate the maximum number
Line 155: Line 157:
This causes the single user
This causes the single user
to perform the work of the <var class="product">Fast/Unload</var> PST whenever a <var>$FunWait</var>, <var>$FunImg</var>, <var>$FunsStr</var>, or <var>$FunSkip</var> is executed.</p></li>
to perform the work of the <var class="product">Fast/Unload</var> PST whenever a <var>$FunWait</var>, <var>$FunImg</var>, <var>$FunsStr</var>, or <var>$FunSkip</var> is executed.</p></li>
<li>If you want to exploit Model 204 zIIP support, see [[Fast/Unload overview#fuhpo|Fast/Unload HPO]].</li>
</ol>
</ol>


Line 205: Line 209:
The parameters described in the following subsections provide useful controls for <var class="product">FUSI</var> requests.
The parameters described in the following subsections provide useful controls for <var class="product">FUSI</var> requests.
   
   
===FUNMTASK===
===<b id="funmtask"></b>FUNMTASK===
The <var>FUNMTASK</var> parameter specifies whether Fast/Unload Soul Interface (FUSI) requests are to be executed on a z/OS subtask or on one of these: the Model 204 maintask, an MP/204 subtask, or the zIIP support SRB.
The <var>FUNMTASK</var> parameter specifies whether Fast/Unload Soul Interface (FUSI) requests are to be executed on a z/OS subtask or on one of these: the Model 204 maintask, an MP/204 subtask, or the zIIP support SRB.


Line 220: Line 224:
The X'02', X'04', and X'08' bits (and the <var>[[FUNPRMU parameter|FUNPRMU]]</var> user parameter) allow the auditing of some of the messages from the <var class="product">FUSI</var> job report to be captured on <code>FUNAUDIT</code> and/or the <var class="product">Model&nbsp;204</var> audit trail, regardless of the report destination specified in the SOUL invocation of the <var class="product">FUSI</var>.</p>
The X'02', X'04', and X'08' bits (and the <var>[[FUNPRMU parameter|FUNPRMU]]</var> user parameter) allow the auditing of some of the messages from the <var class="product">FUSI</var> job report to be captured on <code>FUNAUDIT</code> and/or the <var class="product">Model&nbsp;204</var> audit trail, regardless of the report destination specified in the SOUL invocation of the <var class="product">FUSI</var>.</p>
<p>
<p>
For more detailed usage information, see the <var>[[FUNPARM parameter|FUNPARM]]</var> page.
For more detailed usage information, see the <var>[[FUNPARM parameter|FUNPARM]]</var> page. </p>


===<b id="funmaxt"></b>FUNMAXT===
===<b id="funmaxt"></b>FUNMAXT===
The <var>[[FUNMAXT parameter|FUNMAXT]]</var> system parameter specifies the maximum amount of time, in seconds, a <var class="product">FUSI</var> request is to be given to complete.
The <var>FUNMAXT</var> system parameter specifies the maximum amount of time, in seconds, a <var class="product">FUSI</var> request is to be given to complete.
The timer begins when the <var class="product">FUSI</var> is requested,  
The timer begins when the <var class="product">FUSI</var> is requested,  
by <var>$Funload</var> or by the <var>Recordset</var> class <var>FastUnload</var> or <var>FastUnloadTask</var> method.
by <var>[[$Funload]]</var> or by the <var>[[FastUnload (Recordset function)|FastUnload]]</var> and <var>[[FastUnloadTask (Recordset function)|FastUnloadTask]]</var> methods the <var>Recordset</var> class.


<var>FUNMAXT</var> is a numeric parameter with valid values from 0 to 36000.
<var>FUNMAXT</var> is a numeric parameter with valid values from 0 to 36000.
Line 233: Line 237:
indefinitely while queuing for busy <var class="product">Fast/Unload</var> tasks or for unintentionally long-running requests.
indefinitely while queuing for busy <var class="product">Fast/Unload</var> tasks or for unintentionally long-running requests.


To override <var>FUNMAXT</var> for specific requests, you can use either of these:
For more usage information, see the <var>[[FUNMAXT parameter|FUNMAXT]]</var> page.  
<ul>
<li>The <var>MaxTime</var> named parameter on the <var>FastUnload</var> and <var>FastUnloadTask</var> methods in the
<var>Recordset</var> class. </li>


<li>The sixth parameter on <var>$Funload</var>:
==See also==
<p class="code"><nowiki>* Make sure request completes in one minute
%rc = %rs:fastUnload(%listi, %listo, parameters='NEBUFF=10', maxTime=60)
...
* Make sure request completes in one minute
%rc = $funload('LABEL', %iList, %oList, , 'NEBUFF=10', 60)
</nowiki></p></li>
</ul>
 
It is a reasonable strategy to set <var>FUNMAXT</var> to a fairly low value, then to
selectively set it higher for requests that need more time.
Of course, it can be very difficult to ensure that short-running requests
complete quickly if the Online also has long-running requests that
might tie up all the <var class="product">Fast/Unload</var> tasks.
 
You improve the odds of quick completion if you specify more <var class="product">Fast/Unload</var> tasks
(<var>FUNTSKN</var> bigger), but this might still not be enough:
<ul>
<li>All tasks might be tied up anyway, if there are many long-running requests.</li>


<li>Some of the <var class="product">Fast/Unload</var> tasks might have trouble getting
dispatched, because there are more of them than CPUs to run them. </li>
</ul>
==See also==
===Fast/Unload SOUL Interface topics===
===Fast/Unload SOUL Interface topics===
{{Template:Fast/Unload SOUL Interface topic list}}
{{Template:Fast/Unload SOUL Interface topic list}}

Latest revision as of 21:48, 21 November 2016

Introduction

The Fast/Unload SOUL Interface (FUSI), purchased as a separate Fast/Unload option, allows one to invoke Fast/Unload from a SOUL program. With this approach, an application builds a set of records to be unloaded, using standard SOUL statements. For example, you can reduce unload time by restricting the set of records using indexed Model 204 fields.

There are two ways to invoke Fast/Unload from a SOUL program:

$Funload or the FastUnload and FastUnloadTask methods send the record set to the Fast/Unload PST, which invokes the Fast/Unload load module in a subtask (or PST, under CMS) of Model 204. The full power of the FUEL language is available, including sorts, UAI, etc.

When invoked via $Funload or the FastUnload or FastUnloadTask methods, Fast/Unload only accesses the records in the passed record set, and it only issues reads for disk tracks containing those records. Whether or not the FUSI is employed, Fast/Unload always issues reads only for disk tracks containing the records accessed, but, except for records skipped by the EVERY and SKIPREC parameters, a standalone Fast/Unload accesses all Table B records in a file until Fast/Unload processing ends.

The FUSI also provides the ability to unload data from a Model 204 group; standalone Fast/Unload provides that functionality by allowing multiple Model 204 file names on the OPEN statement.

Note that the FUSI allows unloading to be performed (at the thread level) either of these ways:

  • Synchronously, so that all processing is completed before control is returned from $Funload or the FastUnload and FastUnloadTask methods.
  • Asynchronously, so that control is returned to the SOUL program as soon as the $function or method parameters are verified and accepted.

Note: Fast/Unload processing is never synchronous at the Online level: threads not waiting for the completion of a FUSI request will continue to run while one or more unloads are progressing. When processing is asynchronous (at the thread level), it is up to the user to verify successful completion of the unload, much as a user would verify successful completion of a batch job.

A SOUL procedure is provided as part of the installation process to monitor and control Fast/Unload requests from the ONLINE region. Specifically, a system manager can cancel or examine Fast/Unload requests. In addition, each user can perform the same functions on asynchronous requests that he or she initiated.

To make it easier to diagnose problems where the FUSI is involved, messages are also sent to the Model 204 journal/audit trail at each Fast/Unload request made and at each completed by the FUSI.

The message when the requests are started looks like:

MSIR.0890: Asynchronous request 2 made by $funload

And the message when the requests are completed looks like:

MSIR.0891: Asynchronous request 2 completed, RC = 0

Name changed from Fast/Unload User Language Interface

The Fast/Unload Soul Interface was previously called the The Fast/Unload User Language Interface. The product has remained the same.

When to use the FUSI

If you have the FUSI, you can use it, rather then a standalone Fast/Unload, to perform a given unload. In fact, you can usually use the FUSI and expect it will run as well or better than the comparable standalone Fast/Unload. Many factors make use of the FUSI clearly advantageous over a standalone Fast/Unload, such as:

Sparse index-driven record set If you can use the FIND statement with indexed fields (that is, no Table B scan) to determine a small fraction of the file which then can be processed by Fast/Unload, this will be a significant performance advantage.
Model 204 Boolean processing The capabilities of Model 204 record list processing, and the ability to combine sets of records in the FIND statement, provide not only efficient index-driven record restriction, but also very powerful application capabilities.
File enqueue When a file is open for update in an ONLINE environment, the only way to access that file in a standalone job is to bypass file enqueueing (with the NOENQ parameter, as described in NOEnq). A standalone Fast/Unload with NOENQ will only examine pages that have been written to disk, and so will not see any modified pages still in the buffer pool.

Performing the unload with the FUSI allows you to use normal Model 204 record-level locking, so a file can be unloaded while it is open for update, obtaining a current image of all pages accessed. There is a tradeoff, of course, involved in holding record locks over the duration of an unload job. You can use an unlocked set of records (record lists, or the SOUL FIND WITHOUT LOCKS statement) with the FUSI, and the current disk buffer page image is still used. Note, however, that unloading without a locked record set while the file is open for update is suitable for creating some kind of report or other file extract that can tolerate a degree of inconsistent data, but it is generally not advisable if you are unloading in order to reorganize the file.

Groups (prior to version 4.4) The FUSI allows you to process a set of records derived from one or more files in a permanent, temporary, or ad hoc Model 204 group. Prior to version 4.4, a standalone Fast/Unload can only access the single file designated on the Fast/Unload OPEN directive.
SOUL pre-/post-processing, scheduling You can use the features of SOUL to prepare the Fast/Unload input program. The program can be dynamically generated, or static and stored in an Html or Text statement block or in a separate procedure that is read with $ProcOpn and $ProcDat. The Html or Text block is particularly useful for generating Fast/Unload input programs that are largely static but have some dynamic parts.

The FUSI can be used to manipulate and combine the results of one or more $Funload or FastUnload or FastUnloadTask calls. Besides offering the convenience of post-processing with SOUL, this can be useful for merging, matching, or cross-checking applications. The results can be processed conveniently and efficiently with $lists, with the $FunImg function, with Stringlist objects, or with a FastUnloadTask object.

You can combine scheduling and control of all "background" processing in your Model 204 Online with initiating and controlling FUSI processing.

The CPU time consumed by the Fast/Unload task running under Model 204 will not interfere with online Model 204 users: the z/OS dispatching priority of the Fast/Unload task is 4 less than that of the Model 204 maintask.

In some cases, the only appropriate environment to run Fast/Unload is a batch environment, but you can still obtain the relevant advantages of the FUSI by invoking it in a single-user Model 204 job ("BATCH204"). Some of the reasons for running in a batch environment are:

Virtual storage use The virtual storage requirements of a Fast/Unload task running under Model 204 could significantly affect the virtual storage requirements of the whole region, especially if an external sort is being invoked by Fast/Unload as part of its processing. Care should be taken to ensure that these requirements do not cause paging in the Model 204 region, paging having potentially disastrous performance impact on a multi-user Model 204 region.
Below-the-line storage If the below-the-line storage requirements of Fast/Unload are large, it may need to run in a separate address space from a Model 204 Online job, and if they are extremely demanding, this might not even permit a BATCH204 environment.
Communication for current buffers As mentioned in the table above under "File enqueue", the FUSI provides the current copy of any modified pages that the Fast/Unload subtask accesses. If a very large number of modified pages are in the buffer pool when $Funload or the FastUnload or FastUnloadTask method is called, this could, in some cases, lead to a noticable amount of overhead.

One measure of this would be the PST wait time, described in Fast/Unload job statistics. If this overhead is significant, you might evaluate the tradeoffs in either unloading the file in a standalone unload with the NOENQ parameter, or unloading it when there are fewer dirty pages in the buffer pool, for example, when updating activity is lower.

Setting up the FUSI environment

Note: Using the FUSI with Model 204 version 7.4 requires Fast/Unload version 4.6 or higher. Version 4.7 or higher is required for full exploitation of Model 204 version 7.5.

To use the FUSI, the system manager must perform the following steps:

  1. If using a version of Model 204 prior to 7.5:
    • Include the Sirius Mods in your ONLINE or BATCH204 load module; a load module with the Sirius Mods is necessary to invoke the FUSI.
    • Concatenate the load library containing the Fast/Unload load module to the load library containing the ONLINE or BATCH204 load module on the STEPLIB DD, or copy the Fast/Unload load module into the load library containing the ONLINE or BATCH204 load module.

      For example, if the Model 204 ONLINE load module resides in M204.LOADLIB, and the Fast/Unload load module resides in SIRIUS.LOAD, then this is an appropriate STEPLIB DD for the ONLINE JCL:

      //STEPLIB DD DSN=M204.LOADLIB,DISP=SHR // DD DSN=SIRIUS.LOAD,DISP=SHR

      Under CMS, the Fast/Unload TEXT file must be on a disk accessed by the Model 204 ONLINE service machine.

  2. Modify the JCL (or EXEC) used to run the Model 204 ONLINE or BATCH204 so that it contains a DD card (or FILEDEF statement) for the Fast/Unload audit trail. This audit trail has DDNAME FUNAUDIT. For initial testing purposes, this DD could simply specify SYSOUT=*.

    See the FUSI audit format below.

  3. Modify the User 0 parm card to indicate the maximum number of concurrent Fast/Unload tasks to be allowed. This is specified with the FUNTSKN parameter. This parameter has a default of 0 and a maximum of 64. If this parameter is set to 0, no user will be able to access Fast/Unload via $Funload or the FastUnload or FastUnloadtask method.

    In addition, it might be necessary to specify the name of the Fast/Unload load module (or TEXT file, in CMS). This is done using the FUNPGM parameter. The default value of FUNPGM is FUNLOAD.

    For example, the following would be valid parameters on User 0's parm card:

    FUNTSKN=8,FUNPGM='MYFUN'

  4. The system manager might also need to increase the value of NSUBTKS on User 0's parm card. Fast/Unload requires a PST, and when running under CMS an additional PST, for every active Fast/Unload task. Thus, the NSUBTKS requirement is increased by 1 for Fast/Unload under z/OS, and by 1+FUNTSKN under CMS.

    The one exception to this rule is that when Fast/Unload is to be invoked in a single-user run, you can avoid the overhead of using the multi-user scheduler by setting NSUBTKS to 0. This causes the single user to perform the work of the Fast/Unload PST whenever a $FunWait, $FunImg, $FunsStr, or $FunSkip is executed.

  5. If you want to exploit Model 204 zIIP support, see Fast/Unload HPO.

Once these tasks have been performed by the system manager, Fast/Unload is ready for use by the programmer. Communication with Fast/Unload is achieved by the SOUL $functions or methods that are provided when your site purchases the FUSI product.

FUSI audit format

If FUNAUDIT is allocated to the Online, it will contain messages (for example, starting a Fast/Unload task for a request, or completing a request) in the following format:

Col Len Data --- --- -------------------------- 1 1 Carriage control character 2 11 Date/time (YYDDDHHMMSS) 13 1 Space 14 5 User number 19 1 Space 20 5 Request number 25 1 Space 26 5 Message number (within request) 31 1 Space 32 101 Output data

If FUNAUDIT is not allocated to the Online, the messages are instead produced in the Model 204 audit trail as RK messages with the following format:

Col Len Data --- --- -------------------------- 1 4 'FUN:' 5 1 Space 6 5 User number 11 1 Space 12 5 Request number 17 1 Space 18 5 Message number (within request) 23 1 Space 24 101 Output data

In the above a "request" is a Fast/Unload request, which corresponds to a single invocation of $Funload or of the FastUnload or the FastUnloadTask method.

System parameters for the FUSI

The parameters described in the following subsections provide useful controls for FUSI requests.

FUNMTASK

The FUNMTASK parameter specifies whether Fast/Unload Soul Interface (FUSI) requests are to be executed on a z/OS subtask or on one of these: the Model 204 maintask, an MP/204 subtask, or the zIIP support SRB.

For usage details, see the FUNMTASK page.

FUNPARM

The FUNPARM system parameter is a standard Model 204 bitmask-style parameter.

Setting the X'01' bit specifies that a synchronous Fast/Unload request is not to be allowed while an updating transaction is active. This is to prevent a Fast/Unload request that might take a long time to complete from being run while a user has resources enqueued for an updating transaction.

The X'02', X'04', and X'08' bits (and the FUNPRMU user parameter) allow the auditing of some of the messages from the FUSI job report to be captured on FUNAUDIT and/or the Model 204 audit trail, regardless of the report destination specified in the SOUL invocation of the FUSI.

For more detailed usage information, see the FUNPARM page.

FUNMAXT

The FUNMAXT system parameter specifies the maximum amount of time, in seconds, a FUSI request is to be given to complete. The timer begins when the FUSI is requested, by $Funload or by the FastUnload and FastUnloadTask methods the Recordset class.

FUNMAXT is a numeric parameter with valid values from 0 to 36000. The default value, 0, means no time limit is placed on FUSI requests.

The purpose of FUNMAXT is to prevent user requests from being "hung up" indefinitely while queuing for busy Fast/Unload tasks or for unintentionally long-running requests.

For more usage information, see the FUNMAXT page.

See also

Fast/Unload SOUL Interface topics

Fast/Unload topics

Retrieved from "https://m204wiki.rocketsoftware.com/index.php?title=Fast/Unload_SOUL_Interface&oldid=94666"