Fast/Unload SOUL Interface: Difference between revisions

From m204wiki
Jump to navigation Jump to search
 
(21 intermediate revisions by 3 users not shown)
Line 1: Line 1:
<!-- Page name: Fast/Unload SOUL Interface-->
<!-- Page name: Fast/Unload SOUL Interface-->
==Introduction==
==Introduction==
The <var class="product">Fast/Unload SOUL Interface</var>, purchased as a separate <var class="product">Fast/Unload</var> option,
The <var class="product">Fast/Unload SOUL Interface (FUSI)</var>, purchased as a separate <var class="product">Fast/Unload</var> option,
allows one to invoke <var class="product">Fast/Unload</var> from a <var class="product">[[SOUL]]</var> program.
allows one to invoke <var class="product">Fast/Unload</var> from a <var class="product">[[SOUL]]</var> program.
With this approach, an application builds a set of records to be unloaded,
With this approach, an application builds a set of records to be unloaded,
Line 18: Line 18:
The full power of the FUEL language is available, including sorts, <var>UAI</var>, etc.
The full power of the FUEL language is available, including sorts, <var>UAI</var>, etc.


When invoked via <var>$Funload</var> or the <var>FastUnload</var> or <var>FastUnloadTask</var> methods, <var class="product">Fast/Unload</var>
When invoked via <var>$Funload</var> or the <var>FastUnload</var> or <var>FastUnloadTask</var> methods, <var class="product">Fast/Unload</var> only accesses the records in the passed record set, and it only issues reads
only accesses the records in the passed record set, and only issues reads
for disk tracks containing those records.
for disk tracks containing those records.
Whether or not the <var class="product">Fast/Unload SOUL Interface</var> is employed, <var class="product">Fast/Unload</var> <b>always</b> issues reads
Whether or not the <var class="product">FUSI</var> is employed, <var class="product">Fast/Unload</var> <b>always</b> issues reads
only for disk tracks containing the records accessed, but, except for
only for disk tracks containing the records accessed, but, except for records skipped by the <var>EVERY</var> and <var>SKIPREC</var> parameters, a standalone <var class="product">Fast/Unload</var> accesses all Table B records in a file until
records skipped by the <var>EVERY</var> and <var>SKIPREC</var> parameters,
a standalone <var class="product">Fast/Unload</var> accesses all Table B records in a file until
<var class="product">Fast/Unload</var> processing ends.
<var class="product">Fast/Unload</var> processing ends.


The <var class="product">Fast/Unload SOUL Interface</var> also
The <var class="product">FUSI</var> also provides the ability to unload data from a <var class="product">Model 204</var> group; standalone <var class="product">Fast/Unload</var> provides that functionality by allowing multiple <var class="product">Model 204</var> file names on the <var>[[FUEL#openStmt|OPEN]]</var> statement.
provides the ability to unload data from a <var class="product">Model 204</var> group; standalone <var class="product">Fast/Unload</var> provides that functionality by allowing
multiple <var class="product">Model 204</var> file names on the <var>[[FUEL#openStmt|OPEN]]</var> statement.


Note that the <var class="product">Fast/Unload SOUL Interface</var> allows unloading to be performed (at the thread level) either of these ways:
Note that the <var class="product">FUSI</var> allows unloading to be performed (at the thread level) either of these ways:
<ul>
<ul>
<li><b>Synchronously</b>, so that all processing is
<li><b>Synchronously</b>, so that all processing is completed before control is returned from <var>$Funload</var> or the
completed before control is returned from <var>$Funload</var> or the
<var>FastUnload</var> and <var>FastUnloadTask</var> methods.</li>
<var>FastUnload</var> and <var>FastUnloadTask</var> methods.</li>


Line 42: Line 36:


<p class="note"><b>Note:</b> <var class="product">Fast/Unload</var> processing is <b>never</b> synchronous at the Online level:
<p class="note"><b>Note:</b> <var class="product">Fast/Unload</var> processing is <b>never</b> synchronous at the Online level:
threads not waiting for the completion of a <var class="product">Fast/Unload SOUL Interface</var> requests will continue
threads not waiting for the completion of a <var class="product">FUSI</var> request will continue to run while one or more unloads are progressing.
to run while one or more unloads are progressing.
When processing is asynchronous (at the thread level), it is up to the user to verify successful completion of the unload, much as a user would verify successful completion of a batch job.
When processing is asynchronous (at the thread level), it is up to the
user to verify successful completion of the unload much as a user would
verify successful completion of a batch job.
</p>
</p>


A SOUL procedure is provided as part of the installation
A SOUL procedure is provided as part of the installation process to monitor and control <var class="product">Fast/Unload</var> requests from the <code>ONLINE</code> region.
process to monitor and control <var class="product">Fast/Unload</var> requests from the <code>ONLINE</code> region.
Specifically, a system manager can cancel or examine <var class="product">Fast/Unload</var> requests.
Specifically, a system manager can
In addition, each user can perform the same functions on asynchronous requests that he or she initiated.
cancel or examine <var class="product">Fast/Unload</var> requests.
In addition, each user
can perform the same functions on asynchronous requests that he or she initiated.


To make it easier to diagnose problems where <var class="product">Fast/Unload SOUL Interface</var> is involved, messages are also sent to the <var class="product">Model 204</var> journal/audit trail at each
To make it easier to diagnose problems where the <var class="product">FUSI</var> is involved, messages are also sent to the <var class="product">Model 204</var> journal/audit trail at each <var class="product">Fast/Unload</var> request made and at each completed by the <var class="product">FUSI</var>.
<var class="product">Fast/Unload</var> request made and at each completed by the <var class="product">Fast/Unload SOUL Interface</var>.
   
   
The message when the requests are started looks like:
The message when the requests are started looks like:
Line 67: Line 54:
</p>
</p>


==When to use the Fast/Unload SOUL Interface==
===Name changed from Fast/Unload User Language Interface===
If you have the <var class="product">Fast/Unload SOUL Interface</var>, you can use it,
The <var class="product">Fast/Unload Soul Interface</var> was previously called the The <var class="product">Fast/Unload User Language Interface</var>.  The product has remained the same.
 
==When to use the FUSI==
If you have the <var class="product">FUSI</var>, you can use it,
rather then a standalone <var class="product">Fast/Unload</var>, to perform a given unload. In fact,
rather then a standalone <var class="product">Fast/Unload</var>, to perform a given unload. In fact,
you can usually use the <var class="product">Fast/Unload SOUL Interface</var> and expect it will run as well
you can usually use the <var class="product">FUSI</var> and expect it will run as well
or better than the comparable standalone <var class="product">Fast/Unload</var>.
or better than the comparable standalone <var class="product">Fast/Unload</var>.
Many factors make use of the <var class="product">Fast/Unload SOUL Interface</var> clearly
Many factors make use of the <var class="product">FUSI</var> clearly
advantageous over a standalone <var class="product">Fast/Unload</var>, such as:
advantageous over a standalone <var class="product">Fast/Unload</var>, such as:
<table class="thJustBold">
<table class="thJustBold">
Line 84: Line 74:
<td>When a file is open for update in an ONLINE environment, the only way to access that file in a standalone job is to bypass file enqueueing (with the <var>NOENQ</var> parameter, as described in [[Fast/Unload program parameters#noenq|NOEnq]]). A standalone <var class="product">Fast/Unload</var> with <var>NOENQ</var> will only examine pages that have been written to disk, and so will not see any modified pages still in the buffer pool.  
<td>When a file is open for update in an ONLINE environment, the only way to access that file in a standalone job is to bypass file enqueueing (with the <var>NOENQ</var> parameter, as described in [[Fast/Unload program parameters#noenq|NOEnq]]). A standalone <var class="product">Fast/Unload</var> with <var>NOENQ</var> will only examine pages that have been written to disk, and so will not see any modified pages still in the buffer pool.  
<p>
<p>
Performing the unload with the <var class="product">Fast/Unload SOUL Interface</var> allows you to use normal <var class="product">Model 204</var> record-level locking, so a file can be unloaded while it is open for update, obtaining a current image of all pages accessed. There is a tradeoff, of course, involved in holding record locks over the duration of an unload job. You can use an unlocked set of records (record lists, or the SOUL <var>FIND WITHOUT LOCKS</var> statement) with the <var class="product">Fast/Unload SOUL Interface</var>, and the current disk buffer page image is still used. Note, however, that unloading without a locked record set while the file is open for update is suitable for creating some kind of report or other file extract that can tolerate a degree of inconsistent data, but it is generally not advisable if you are unloading in order to reorganize the file.</p></td></tr>
Performing the unload with the <var class="product">FUSI</var> allows you to use normal <var class="product">Model 204</var> record-level locking, so a file can be unloaded while it is open for update, obtaining a current image of all pages accessed. There is a tradeoff, of course, involved in holding record locks over the duration of an unload job. You can use an unlocked set of records (record lists, or the SOUL <var>FIND WITHOUT LOCKS</var> statement) with the <var class="product">FUSI</var>, and the current disk buffer page image is still used. Note, however, that unloading without a locked record set while the file is open for update is suitable for creating some kind of report or other file extract that can tolerate a degree of inconsistent data, but it is generally not advisable if you are unloading in order to reorganize the file.</p></td></tr>


<tr><th>Groups (prior to version 4.4)</th>
<tr><th>Groups (prior to version 4.4)</th>
<td>The <var class="product">Fast/Unload SOUL Interface</var> allows you to process a set of records derived from one or more files in a permanent, temporary, or ad hoc <var class="product">Model 204</var> group. Prior to version 4.4, a standalone <var class="product">Fast/Unload</var> can only access the single file designated on the <var class="product">Fast/Unload</var> <var>OPEN</var> directive.</td></tr>
<td>The <var class="product">FUSI</var> allows you to process a set of records derived from one or more files in a permanent, temporary, or ad hoc <var class="product">Model 204</var> group. Prior to version 4.4, a standalone <var class="product">Fast/Unload</var> can only access the single file designated on the <var class="product">Fast/Unload</var> <var>OPEN</var> directive.</td></tr>


<tr><th>SOUL pre-/post-processing, scheduling</th>
<tr><th>SOUL pre-/post-processing, scheduling</th>
<td>You can use the features of SOUL to prepare the <var class="product">Fast/Unload</var> input program. The program can be dynamically generated, or static and stored in an <var>[[Text and Html statements|Html]]</var> or <var>[[Text and Html statements|Text]]</var> statement block or in a separate procedure that is read with <var>[[$ProcOpn]]</var> and <var>[[$ProcDat]]</var>. The <var>Html</var> or <var>Text</var> block is particularly useful for generating <var class="product">Fast/Unload</var> input programs that are largely static but have some dynamic parts.  
<td>You can use the features of SOUL to prepare the <var class="product">Fast/Unload</var> input program. The program can be dynamically generated, or static and stored in an <var>[[Text and Html statements|Html]]</var> or <var>[[Text and Html statements|Text]]</var> statement block or in a separate procedure that is read with <var>[[$ProcOpn]]</var> and <var>[[$ProcDat]]</var>. The <var>Html</var> or <var>Text</var> block is particularly useful for generating <var class="product">Fast/Unload</var> input programs that are largely static but have some dynamic parts.  
<p>
<p>
The <var class="product">Fast/Unload SOUL Interface</var> can be used to manipulate and combine the results of one or more <var>$Funload</var> or <var>FastUnload</var> or <var>FastUnloadTask</var> calls. Besides offering the convenience of post-processing with SOUL, this can be useful for merging, matching, or cross-checking applications. The results can be processed conveniently and efficiently with [[$lists|$lists]], with the <var>$FunImg</var> function, with <var>Stringlist</var> objects, or with a <var>FastUnloadTask</var> object. </p>
The <var class="product">FUSI</var> can be used to manipulate and combine the results of one or more <var>$Funload</var> or <var>FastUnload</var> or <var>FastUnloadTask</var> calls. Besides offering the convenience of post-processing with SOUL, this can be useful for merging, matching, or cross-checking applications. The results can be processed conveniently and efficiently with [[$lists|$lists]], with the <var>$FunImg</var> function, with <var>Stringlist</var> objects, or with a <var>FastUnloadTask</var> object. </p>
<p>
<p>
You can combine scheduling and control of all "background" processing in your <var class="product">Model 204</var> Online with initiating and controlling <var class="product">Fast/Unload SOUL Interface</var> processing.</p></td></tr>
You can combine scheduling and control of all "background" processing in your <var class="product">Model 204</var> Online with initiating and controlling <var class="product">FUSI</var> processing.</p></td></tr>
</table>
</table>


The CPU time consumed by the <var class="product">Fast/Unload</var> task running under <var class="product">Model 204</var> will not
The CPU time consumed by the <var class="product">Fast/Unload</var> task running under <var class="product">Model 204</var> will not
interfere with online <var class="product">Model 204</var> users: the z/OS dispatching priority of
interfere with online <var class="product">Model 204</var> users: the z/OS dispatching priority of
the <var class="product">Fast/Unload</var> task is 4 less than that of the <var class="product">Model 204</var> main task.
the <var class="product">Fast/Unload</var> task is 4 less than that of the <var class="product">Model 204</var> maintask.


In some cases, the only appropriate environment to run <var class="product">Fast/Unload</var> is
In some cases, the only appropriate environment to run <var class="product">Fast/Unload</var> is
a batch environment, but you can still obtain the relevant advantages
a batch environment, but you can still obtain the relevant advantages of the <var class="product">FUSI</var> by invoking it in a single-user <var class="product">Model 204</var> job ("BATCH204").
of the <var class="product">Fast/Unload SOUL Interface</var> by invoking it in a single-user <var class="product">Model 204</var> job ("BATCH204").
Some of the reasons for running in a batch environment are:
Some of the reasons for running in a batch environment are:
<table class="thJustBold">
<table class="thJustBold">
Line 113: Line 102:


<tr><th>Communication for current buffers</th>
<tr><th>Communication for current buffers</th>
<td>As mentioned in the table above under "File enqueue", the <var class="product">Fast/Unload SOUL Interface</var> provides the current copy of any modified pages that the <var class="product">Fast/Unload</var> subtask accesses. If a very large number of modified pages are in the buffer pool when <var>$Funload</var> or the <var>FastUnload</var> or <var>FastUnloadTask</var> method is called, this could, in some cases, lead to a noticable amount of overhead.  
<td>As mentioned in the table above under "File enqueue", the <var class="product">FUSI</var> provides the current copy of any modified pages that the <var class="product">Fast/Unload</var> subtask accesses. If a very large number of modified pages are in the buffer pool when <var>$Funload</var> or the <var>FastUnload</var> or <var>FastUnloadTask</var> method is called, this could, in some cases, lead to a noticable amount of overhead.  
<p>
<p>
One measure of this would be the <b>PST wait time</b>, described in [[Fast/Unload job statistics]]. If this overhead is significant, you might evaluate the tradeoffs in either unloading the file in a standalone unload with the <var>NOENQ</var> parameter, or unloading it when there are fewer dirty pages in the buffer pool, for example, when updating activity is lower.</p></td></tr>
One measure of this would be the <b>PST wait time</b>, described in [[Fast/Unload job statistics]]. If this overhead is significant, you might evaluate the tradeoffs in either unloading the file in a standalone unload with the <var>NOENQ</var> parameter, or unloading it when there are fewer dirty pages in the buffer pool, for example, when updating activity is lower.</p></td></tr>
</table>
</table>


==Setting up the Fast/Unload SOUL Interface environment==
==Setting up the FUSI environment==
<p class="note"><b>Note:</b> Using the <var class="product">Fast/Unload SOUL Interface</var> with Model&nbsp;204 version 7.4 requires <var class="product">Fast/Unload</var> version 4.6 or higher. </p>
<p class="note"><b>Note:</b> Using the <var class="product">FUSI</var> with Model&nbsp;204 version 7.4 requires <var class="product">Fast/Unload</var> version 4.6 or higher. Version 4.7 or higher is required for full exploitation of Model&nbsp;204 version 7.5.</p>


To use the <var class="product">Fast/Unload SOUL Interface</var>, the system
To use the <var class="product">FUSI</var>, the system manager must perform the following steps:
manager must perform the following steps:
<ol>
<ol>
<li>If using a version of Model 204 prior to 7.5, include the [[Sirius Mods]] in your <code>ONLINE</code> or <code>BATCH204</code> load module.
<li>If using a version of Model 204 prior to 7.5:
</li>
<ul>
 
<li>Include the [[Sirius Mods]] in your <code>ONLINE</code> or <code>BATCH204</code> load module; a load module with the Sirius Mods is necessary to invoke the FUSI.</li>
<li>Concatenate the load library containing the <var class="product">Fast/Unload</var>
<li>Concatenate the load library containing the <var class="product">Fast/Unload</var>
load module to the load library containing the <code>ONLINE</code> or <code>BATCH204</code> load module on the <code>STEPLIB</code> DD, or copy the <var class="product">Fast/Unload</var> load module into the load library containing the <code>ONLINE</code> or <code>BATCH204</code> load module.
load module to the load library containing the <code>ONLINE</code> or <code>BATCH204</code> load module on the <code>STEPLIB</code> DD, or copy the <var class="product">Fast/Unload</var> load module into the load library containing the <code>ONLINE</code> or <code>BATCH204</code> load module.
Line 137: Line 125:
Under CMS, the <var class="product">Fast/Unload</var> TEXT file must be on a disk accessed by the
Under CMS, the <var class="product">Fast/Unload</var> TEXT file must be on a disk accessed by the
<var class="product">Model 204</var> <code>ONLINE</code> service machine.</p></li>
<var class="product">Model 204</var> <code>ONLINE</code> service machine.</p></li>
</ul>
</li>


<li>Modify the JCL (or EXEC) used to run the <var class="product">Model 204</var> <code>ONLINE</code> or <code>BATCH204</code> so that it invokes the custom load module from step one, and so that it contains a
<li>Modify the JCL (or EXEC) used to run the <var class="product">Model&nbsp;204</var> <code>ONLINE</code> or <code>BATCH204</code> so that it contains a
DD card (or FILEDEF statement) for the <var class="product">Fast/Unload</var> audit trail.
DD card (or FILEDEF statement) for the <var class="product">Fast/Unload</var> audit trail.
This audit trail has DDNAME <code>FUNAUDIT</code>.
This audit trail has DDNAME <code>FUNAUDIT</code>.
For initial testing purposes, this DD could simply specify <code>SYSOUT=*</code>.</li>
For initial testing purposes, this DD could simply specify <code>SYSOUT=*</code>.
<p>See the [[#audFmt|FUSI audit format below]].</p></li>


<li>Modify the User 0 parm card to indicate the maximum number
<li>Modify the User 0 parm card to indicate the maximum number
Line 166: Line 157:
This causes the single user
This causes the single user
to perform the work of the <var class="product">Fast/Unload</var> PST whenever a <var>$FunWait</var>, <var>$FunImg</var>, <var>$FunsStr</var>, or <var>$FunSkip</var> is executed.</p></li>
to perform the work of the <var class="product">Fast/Unload</var> PST whenever a <var>$FunWait</var>, <var>$FunImg</var>, <var>$FunsStr</var>, or <var>$FunSkip</var> is executed.</p></li>
<li>If you want to exploit Model 204 zIIP support, see [[Fast/Unload overview#fuhpo|Fast/Unload HPO]].</li>
</ol>
</ol>


Line 171: Line 164:
ready for use by the programmer.
ready for use by the programmer.
Communication with <var class="product">Fast/Unload</var> is achieved by the <var class="product">SOUL</var> $functions or methods
Communication with <var class="product">Fast/Unload</var> is achieved by the <var class="product">SOUL</var> $functions or methods
that are provided when a site purchases the <var class="product">Fast/Unload SOUL Interface</var>.
that are provided when your site purchases the <var class="product">FUSI</var> product.
 
===<b id="audFmt"></b>FUSI audit format===
If <var>FUNAUDIT</var> is allocated to the Online, it will contain messages
(for example, starting a Fast/Unload task for a request, or completing
a request) in the following format:
<p class="code"><nowiki>Col  Len  Data
---  ---  --------------------------
1    1  Carriage control character
2    11  Date/time (YYDDDHHMMSS)
13    1  Space
14    5  User number
19    1  Space
20    5  Request number
25    1  Space
26    5  Message number (within request)
31    1  Space
32  101  Output data
</nowiki></p>
If <var>FUNAUDIT</var> is not allocated to the Online, the messages are instead
produced in the Model&nbsp;204 audit trail as RK messages with the
following format:
<p class="code"><nowiki>Col  Len  Data
---  ---  --------------------------
1    4  'FUN:'
5    1  Space
6    5  User number
11    1  Space
12    5  Request number
17    1  Space
18    5  Message number (within request)
23    1  Space
24  101  Output data
</nowiki></p>
In the above a "request" is a Fast/Unload request, which corresponds
to a single invocation of <var>$Funload</var> or of the
<var>FastUnload</var> or the <var>FastUnloadTask</var> method.
 
==<b id="sprmuli"></b>System parameters for the FUSI==
The parameters described in the following subsections provide useful controls for <var class="product">FUSI</var> requests.
   
   
<div id="sprmuli"></div>
===<b id="funmtask"></b>FUNMTASK===
The <var>FUNMTASK</var> parameter specifies whether Fast/Unload Soul Interface (FUSI) requests are to be executed on a z/OS subtask or on one of these: the Model 204 maintask, an MP/204 subtask, or the zIIP support SRB.


==System parameters for the Fast/Unload SOUL Interface==
For usage details, see the <var>[[FUNMTASK parameter|FUNMTASK]]</var> page.
<!--Caution: <div> above-->


The parameters described in the following subsections provide
===<b id="funparm"></b>FUNPARM===
useful controls for <var class="product">Fast/Unload SOUL Interface</var> requests.
The <var>FUNPARM</var> system parameter is a standard <var class="product">Model 204</var> bitmask-style parameter.
<p>
<div id="funparm"></div>
===FUNPARM===
<!--Caution: <div> above-->
The <var>[[FUNPARM parameter|FUNPARM]]</var> system parameter
is a standard <var class="product">Model 204</var> bitmask-style parameter.
Setting the X'01' bit specifies that a synchronous
Setting the X'01' bit specifies that a synchronous
<var class="product">Fast/Unload</var> request is <i>not</i> to be allowed while an updating transaction is active.
<var class="product">Fast/Unload</var> request is <i>not</i> to be allowed while an updating transaction is active.
This is to prevent a <var class="product">Fast/Unload</var> request that might take a long time to
This is to prevent a <var class="product">Fast/Unload</var> request that might take a long time to complete from being run while a user has resources enqueued for an updating transaction.
complete from being run while a user has resources enqueued for an updating
transaction.
These resources would, of course, include the blocking of checkpoints.
 
If the <var>FUNPARM</var> X'01' bit is set, and a thread attempts a synchronous <var class="product">Fast/Unload SOUL Interface</var>
request (with <var>$Funload</var> or the <var>FastUnload</var> method of the <var>Recordset</var> class)
in the middle of an updating transaction, the transaction is cancelled
with a message like the following:
<p class="code">CANCELLING REQUEST: MSIR.0561: $FUNLOAD: Synchronous request during
                    update transaction in line 43, procedure
                    FUNTEST, file ALEXPROC
</p>
</p>
<p>
<p>
The X'02', X'04', and X'08' bits of <var>FUNPARM</var> are supported as of version 4.7 of <var class="product">Fast/Unload</var>. These settings control the <var class="product">Model&nbsp;204</var> auditing of <var class="product">Fast/Unload</var> messages.
The X'02', X'04', and X'08' bits (and the <var>[[FUNPRMU parameter|FUNPRMU]]</var> user parameter) allow the auditing of some of the messages from the <var class="product">FUSI</var> job report to be captured on <code>FUNAUDIT</code> and/or the <var class="product">Model&nbsp;204</var> audit trail, regardless of the report destination specified in the SOUL invocation of the <var class="product">FUSI</var>.</p>
</p>
<p>
 
For more detailed usage information, see the <var>[[FUNPARM parameter|FUNPARM]]</var> page. </p>
<div id="funprmu"></div>
 
===FUNPRMU===
<!--Caution: <div> above-->
The <var>[[FUNPRMU parameter|FUNPRMU]]</var> system parameter is a standard <var class="product">Model 204</var> bitmask-style parameter, supported as of version 4.7 of <var class="product">Fast/Unload</var>.
 
The X'02', X'04', and X'08' bits of <var>FUNPRMU</var> control the <var class="product">Model&nbsp;204</var> auditing of <var class="product">Fast/Unload</var> messages. As described in [[FUNPRMU_parameter#Fast/Unload report processing|Fast/Unload report processing]], <var>FUNPRMU</var> (and <var>FUNPARM</var>) allow the auditing of some of the messages from the <var class="product">Fast/Unload SOUL Interface</var> job report to be captured on <var>FUNAUDIT</var> and/or the <var class="product">Model 204</var> audit trail, regardless of the report destination specified in the SOUL invocation of the <var class="product">Fast/Unload SOUL Interface</var>.
 
<div id="funmaxt"></div>
 
===FUNMAXT===
<!--Caution: <div> above-->


The <var>[[FUNMAXT parameter|FUNMAXT]]</var> system parameter specifies
===<b id="funmaxt"></b>FUNMAXT===
the maximum amount of time, in seconds, a <var class="product">Fast/Unload SOUL Interface</var> request is to be given to complete.
The <var>FUNMAXT</var> system parameter specifies the maximum amount of time, in seconds, a <var class="product">FUSI</var> request is to be given to complete.
The timer begins when the <var class="product">Fast/Unload SOUL Interface</var> is requested,  
The timer begins when the <var class="product">FUSI</var> is requested,  
by <var>$Funload</var> or by the <var>Recordset</var> class <var>FastUnload</var> or <var>FastUnloadTask</var> method.
by <var>[[$Funload]]</var> or by the <var>[[FastUnload (Recordset function)|FastUnload]]</var> and <var>[[FastUnloadTask (Recordset function)|FastUnloadTask]]</var> methods the <var>Recordset</var> class.


<var>FUNMAXT</var> is a numeric parameter with valid values from 0 to 36000.
<var>FUNMAXT</var> is a numeric parameter with valid values from 0 to 36000.
The default value, 0, means no time limit is placed on <var class="product">Fast/Unload SOUL Interface</var> requests.
The default value, 0, means no time limit is placed on <var class="product">FUSI</var> requests.


The purpose of <var>FUNMAXT</var> is to prevent user requests from being "hung up"
The purpose of <var>FUNMAXT</var> is to prevent user requests from being "hung up"
indefinitely while queuing for busy <var class="product">Fast/Unload</var> tasks or for unintentionally long-running requests.
indefinitely while queuing for busy <var class="product">Fast/Unload</var> tasks or for unintentionally long-running requests.


To override <var>FUNMAXT</var> for specific requests, you can use either of these:
For more usage information, see the <var>[[FUNMAXT parameter|FUNMAXT]]</var> page.  
<ul>
<li>The <var>MaxTime</var>
named parameter on the <var>FastUnload</var> and <var>FastUnloadTask</var> methods in the
<var>Recordset</var> class. </li>
 
<li>The sixth parameter on <var>$Funload</var>:
<p class="code"><nowiki>* Make sure request completes in one minute
%rc = %rs:fastUnload(%listi, %listo, -
                    parameters='NEBUFF=10', maxTime=60)
...
* Make sure request completes in one minute
%rc = $funload('LABEL', %iList, %oList, , 'NEBUFF=10', 60)
</nowiki></p></li>
</ul>
 
It is a reasonable strategy to set <var>FUNMAXT</var> to a fairly low value, then to
selectively set it higher for requests that need more time.
Of course, it can be very difficult to ensure that short-running requests
complete quickly if the Online also has long-running requests that
might tie up all the <var class="product">Fast/Unload</var> tasks.


You improve the odds of quick completion if you specify more <var class="product">Fast/Unload</var> tasks
==See also==
(<var>FUNTSKN</var> bigger), but this may still not be enough:
<ul>
<li>All tasks might be tied up anyway, if there are many long-running requests.</li>
 
<li>Some of the <var class="product">Fast/Unload</var> tasks might have trouble getting
dispatched, because there are more of them than CPUs to run them. </li>
</ul>


==See also==
===Fast/Unload SOUL Interface topics===
===Fast/Unload SOUL interface topics===
{{Template:Fast/Unload SOUL Interface topic list}}
{{Template:Fast/Unload SOUL Interface topic list}}



Latest revision as of 21:48, 21 November 2016

Introduction

The Fast/Unload SOUL Interface (FUSI), purchased as a separate Fast/Unload option, allows one to invoke Fast/Unload from a SOUL program. With this approach, an application builds a set of records to be unloaded, using standard SOUL statements. For example, you can reduce unload time by restricting the set of records using indexed Model 204 fields.

There are two ways to invoke Fast/Unload from a SOUL program:

$Funload or the FastUnload and FastUnloadTask methods send the record set to the Fast/Unload PST, which invokes the Fast/Unload load module in a subtask (or PST, under CMS) of Model 204. The full power of the FUEL language is available, including sorts, UAI, etc.

When invoked via $Funload or the FastUnload or FastUnloadTask methods, Fast/Unload only accesses the records in the passed record set, and it only issues reads for disk tracks containing those records. Whether or not the FUSI is employed, Fast/Unload always issues reads only for disk tracks containing the records accessed, but, except for records skipped by the EVERY and SKIPREC parameters, a standalone Fast/Unload accesses all Table B records in a file until Fast/Unload processing ends.

The FUSI also provides the ability to unload data from a Model 204 group; standalone Fast/Unload provides that functionality by allowing multiple Model 204 file names on the OPEN statement.

Note that the FUSI allows unloading to be performed (at the thread level) either of these ways:

  • Synchronously, so that all processing is completed before control is returned from $Funload or the FastUnload and FastUnloadTask methods.
  • Asynchronously, so that control is returned to the SOUL program as soon as the $function or method parameters are verified and accepted.

Note: Fast/Unload processing is never synchronous at the Online level: threads not waiting for the completion of a FUSI request will continue to run while one or more unloads are progressing. When processing is asynchronous (at the thread level), it is up to the user to verify successful completion of the unload, much as a user would verify successful completion of a batch job.

A SOUL procedure is provided as part of the installation process to monitor and control Fast/Unload requests from the ONLINE region. Specifically, a system manager can cancel or examine Fast/Unload requests. In addition, each user can perform the same functions on asynchronous requests that he or she initiated.

To make it easier to diagnose problems where the FUSI is involved, messages are also sent to the Model 204 journal/audit trail at each Fast/Unload request made and at each completed by the FUSI.

The message when the requests are started looks like:

MSIR.0890: Asynchronous request 2 made by $funload

And the message when the requests are completed looks like:

MSIR.0891: Asynchronous request 2 completed, RC = 0

Name changed from Fast/Unload User Language Interface

The Fast/Unload Soul Interface was previously called the The Fast/Unload User Language Interface. The product has remained the same.

When to use the FUSI

If you have the FUSI, you can use it, rather then a standalone Fast/Unload, to perform a given unload. In fact, you can usually use the FUSI and expect it will run as well or better than the comparable standalone Fast/Unload. Many factors make use of the FUSI clearly advantageous over a standalone Fast/Unload, such as:

Sparse index-driven record set If you can use the FIND statement with indexed fields (that is, no Table B scan) to determine a small fraction of the file which then can be processed by Fast/Unload, this will be a significant performance advantage.
Model 204 Boolean processing The capabilities of Model 204 record list processing, and the ability to combine sets of records in the FIND statement, provide not only efficient index-driven record restriction, but also very powerful application capabilities.
File enqueue When a file is open for update in an ONLINE environment, the only way to access that file in a standalone job is to bypass file enqueueing (with the NOENQ parameter, as described in NOEnq). A standalone Fast/Unload with NOENQ will only examine pages that have been written to disk, and so will not see any modified pages still in the buffer pool.

Performing the unload with the FUSI allows you to use normal Model 204 record-level locking, so a file can be unloaded while it is open for update, obtaining a current image of all pages accessed. There is a tradeoff, of course, involved in holding record locks over the duration of an unload job. You can use an unlocked set of records (record lists, or the SOUL FIND WITHOUT LOCKS statement) with the FUSI, and the current disk buffer page image is still used. Note, however, that unloading without a locked record set while the file is open for update is suitable for creating some kind of report or other file extract that can tolerate a degree of inconsistent data, but it is generally not advisable if you are unloading in order to reorganize the file.

Groups (prior to version 4.4) The FUSI allows you to process a set of records derived from one or more files in a permanent, temporary, or ad hoc Model 204 group. Prior to version 4.4, a standalone Fast/Unload can only access the single file designated on the Fast/Unload OPEN directive.
SOUL pre-/post-processing, scheduling You can use the features of SOUL to prepare the Fast/Unload input program. The program can be dynamically generated, or static and stored in an Html or Text statement block or in a separate procedure that is read with $ProcOpn and $ProcDat. The Html or Text block is particularly useful for generating Fast/Unload input programs that are largely static but have some dynamic parts.

The FUSI can be used to manipulate and combine the results of one or more $Funload or FastUnload or FastUnloadTask calls. Besides offering the convenience of post-processing with SOUL, this can be useful for merging, matching, or cross-checking applications. The results can be processed conveniently and efficiently with $lists, with the $FunImg function, with Stringlist objects, or with a FastUnloadTask object.

You can combine scheduling and control of all "background" processing in your Model 204 Online with initiating and controlling FUSI processing.

The CPU time consumed by the Fast/Unload task running under Model 204 will not interfere with online Model 204 users: the z/OS dispatching priority of the Fast/Unload task is 4 less than that of the Model 204 maintask.

In some cases, the only appropriate environment to run Fast/Unload is a batch environment, but you can still obtain the relevant advantages of the FUSI by invoking it in a single-user Model 204 job ("BATCH204"). Some of the reasons for running in a batch environment are:

Virtual storage use The virtual storage requirements of a Fast/Unload task running under Model 204 could significantly affect the virtual storage requirements of the whole region, especially if an external sort is being invoked by Fast/Unload as part of its processing. Care should be taken to ensure that these requirements do not cause paging in the Model 204 region, paging having potentially disastrous performance impact on a multi-user Model 204 region.
Below-the-line storage If the below-the-line storage requirements of Fast/Unload are large, it may need to run in a separate address space from a Model 204 Online job, and if they are extremely demanding, this might not even permit a BATCH204 environment.
Communication for current buffers As mentioned in the table above under "File enqueue", the FUSI provides the current copy of any modified pages that the Fast/Unload subtask accesses. If a very large number of modified pages are in the buffer pool when $Funload or the FastUnload or FastUnloadTask method is called, this could, in some cases, lead to a noticable amount of overhead.

One measure of this would be the PST wait time, described in Fast/Unload job statistics. If this overhead is significant, you might evaluate the tradeoffs in either unloading the file in a standalone unload with the NOENQ parameter, or unloading it when there are fewer dirty pages in the buffer pool, for example, when updating activity is lower.

Setting up the FUSI environment

Note: Using the FUSI with Model 204 version 7.4 requires Fast/Unload version 4.6 or higher. Version 4.7 or higher is required for full exploitation of Model 204 version 7.5.

To use the FUSI, the system manager must perform the following steps:

  1. If using a version of Model 204 prior to 7.5:
    • Include the Sirius Mods in your ONLINE or BATCH204 load module; a load module with the Sirius Mods is necessary to invoke the FUSI.
    • Concatenate the load library containing the Fast/Unload load module to the load library containing the ONLINE or BATCH204 load module on the STEPLIB DD, or copy the Fast/Unload load module into the load library containing the ONLINE or BATCH204 load module.

      For example, if the Model 204 ONLINE load module resides in M204.LOADLIB, and the Fast/Unload load module resides in SIRIUS.LOAD, then this is an appropriate STEPLIB DD for the ONLINE JCL:

      //STEPLIB DD DSN=M204.LOADLIB,DISP=SHR // DD DSN=SIRIUS.LOAD,DISP=SHR

      Under CMS, the Fast/Unload TEXT file must be on a disk accessed by the Model 204 ONLINE service machine.

  2. Modify the JCL (or EXEC) used to run the Model 204 ONLINE or BATCH204 so that it contains a DD card (or FILEDEF statement) for the Fast/Unload audit trail. This audit trail has DDNAME FUNAUDIT. For initial testing purposes, this DD could simply specify SYSOUT=*.

    See the FUSI audit format below.

  3. Modify the User 0 parm card to indicate the maximum number of concurrent Fast/Unload tasks to be allowed. This is specified with the FUNTSKN parameter. This parameter has a default of 0 and a maximum of 64. If this parameter is set to 0, no user will be able to access Fast/Unload via $Funload or the FastUnload or FastUnloadtask method.

    In addition, it might be necessary to specify the name of the Fast/Unload load module (or TEXT file, in CMS). This is done using the FUNPGM parameter. The default value of FUNPGM is FUNLOAD.

    For example, the following would be valid parameters on User 0's parm card:

    FUNTSKN=8,FUNPGM='MYFUN'

  4. The system manager might also need to increase the value of NSUBTKS on User 0's parm card. Fast/Unload requires a PST, and when running under CMS an additional PST, for every active Fast/Unload task. Thus, the NSUBTKS requirement is increased by 1 for Fast/Unload under z/OS, and by 1+FUNTSKN under CMS.

    The one exception to this rule is that when Fast/Unload is to be invoked in a single-user run, you can avoid the overhead of using the multi-user scheduler by setting NSUBTKS to 0. This causes the single user to perform the work of the Fast/Unload PST whenever a $FunWait, $FunImg, $FunsStr, or $FunSkip is executed.

  5. If you want to exploit Model 204 zIIP support, see Fast/Unload HPO.

Once these tasks have been performed by the system manager, Fast/Unload is ready for use by the programmer. Communication with Fast/Unload is achieved by the SOUL $functions or methods that are provided when your site purchases the FUSI product.

FUSI audit format

If FUNAUDIT is allocated to the Online, it will contain messages (for example, starting a Fast/Unload task for a request, or completing a request) in the following format:

Col Len Data --- --- -------------------------- 1 1 Carriage control character 2 11 Date/time (YYDDDHHMMSS) 13 1 Space 14 5 User number 19 1 Space 20 5 Request number 25 1 Space 26 5 Message number (within request) 31 1 Space 32 101 Output data

If FUNAUDIT is not allocated to the Online, the messages are instead produced in the Model 204 audit trail as RK messages with the following format:

Col Len Data --- --- -------------------------- 1 4 'FUN:' 5 1 Space 6 5 User number 11 1 Space 12 5 Request number 17 1 Space 18 5 Message number (within request) 23 1 Space 24 101 Output data

In the above a "request" is a Fast/Unload request, which corresponds to a single invocation of $Funload or of the FastUnload or the FastUnloadTask method.

System parameters for the FUSI

The parameters described in the following subsections provide useful controls for FUSI requests.

FUNMTASK

The FUNMTASK parameter specifies whether Fast/Unload Soul Interface (FUSI) requests are to be executed on a z/OS subtask or on one of these: the Model 204 maintask, an MP/204 subtask, or the zIIP support SRB.

For usage details, see the FUNMTASK page.

FUNPARM

The FUNPARM system parameter is a standard Model 204 bitmask-style parameter.

Setting the X'01' bit specifies that a synchronous Fast/Unload request is not to be allowed while an updating transaction is active. This is to prevent a Fast/Unload request that might take a long time to complete from being run while a user has resources enqueued for an updating transaction.

The X'02', X'04', and X'08' bits (and the FUNPRMU user parameter) allow the auditing of some of the messages from the FUSI job report to be captured on FUNAUDIT and/or the Model 204 audit trail, regardless of the report destination specified in the SOUL invocation of the FUSI.

For more detailed usage information, see the FUNPARM page.

FUNMAXT

The FUNMAXT system parameter specifies the maximum amount of time, in seconds, a FUSI request is to be given to complete. The timer begins when the FUSI is requested, by $Funload or by the FastUnload and FastUnloadTask methods the Recordset class.

FUNMAXT is a numeric parameter with valid values from 0 to 36000. The default value, 0, means no time limit is placed on FUSI requests.

The purpose of FUNMAXT is to prevent user requests from being "hung up" indefinitely while queuing for busy Fast/Unload tasks or for unintentionally long-running requests.

For more usage information, see the FUNMAXT page.

See also

Fast/Unload SOUL Interface topics

Fast/Unload topics