Fast/Unload job statistics: Difference between revisions

Latest revision as of 14:35, 11 December 2015

Because Fast/Unload is essentially a performance product, it is important to maintain statistics that indicate the cost of performing a Fast/Unload and provide information that might be useful in tuning future unloads. These statistics are reported on the report data set. These statistics are reported for the compile step, the unload step, and Ordered Index unload step separately.

One of these stats, "Number of extension pages in base buffer," along with some of the Table B statistics that are reported as part of FSTATS processing, can also help you detect a need for a reorganization for improving your file's performance with Model 204.

If you use Fast/Unload in a z/OS environment, you can choose to generate SMF records in addition to the Fast/Unload report. See Fast/Unload SMF record format for more information about Fast/Unload SMF records. All of the statistics, of course, are reported on SMF records; the printing of statistics on the FUNPRINT data set, however, is restricted to those statistics that have a value greater than zero. You must apply a zap with the SMF record number to enable SMF record generation.

Many of the statistics refer to I/O processing of the input Model 204 file (or files, in the case of a group). There are two groups of these I/O statistics:

Base buffer statistics These occur only during the Unload phase, and are for accessing Table B pages in page number order, that is, accessing the the base records processed during the unload.

Extension page statistics

These can apply during any phase. They are for directly reading pages from the Model 204 file in no predetermined order. These are called extension page statistics, because one type of direct page access is for extension record access during the Unload phase.

There are other types of direct page access, such as reading the Table A field and coded value information during the Compile phase, reading the Existence Bit Map, and reading B-tree, list, and index bitmap pages during the Index unload phase.

Note that the extension pages actually accessed in the unload phase depend on the type of processing in your FUEL program:

A UAI operation accesses all extensions of all records being unloaded.
FSTATS processing accesses all extensions of all records processed.
Some FUEL constructs, such as DELETE or using # or * for a field occurrence, cause all extensions of a record to be processed.

Otherwise, only as much of a logical Model 204 record is processed as needed to reference the fields used in the FUEL program.

The following is a description of statistics that are reported:

Base buffer reads	Total number of EXCPs issued to read a track or group of tracks into a base buffer. It is directly affected by the `SEBUFF` parameter and the number of `Model 204` input records (that is, base records) processed. This statistic was new in `Fast/Unload` version 4.0.
Base buffer waits	Total number of WAITs issued after initiating reads into the base buffers. The ratio of EXCPs to WAITs indicates, to some extent, the degree of overlap of CPU with I/O processing. This statistic was new in `Fast/Unload` version 4.0.
Extension pg in base buffs	Total number of `Model 204` file pages accessed "directly" which were currently available in pages read into base buffers. This indicates how many extension records were "physically close" to their base record during the Unload phase; specifically, within the `SBBUFF` times `NBBUFF` tracks starting with the group of `SBBUFF` tracks containing the base record. A large number for this statistic indicates that `Model 204` database processing can efficiently access the extension records accompanying a base record, with little physical disk manipulation (or even none, with caching devices). As mentioned in Description of Table B statistics, this information can be used in addition to the `FSTATS` information about extension records, to help you determine possible benefits obtained by reorganizing a file. If extension records are frequently accessed in `Model 204` processing and a large number of them are not physically close to the base record, performance may suffer to some degree. This statistic was new in `Fast/Unload` version 4.0.
Extension pg in exten pool	Total number of `Model 204` file pages accessed "directly" which were not currently available in pages read into base buffers but which were available in the pool of extension pages. The larger this number is, the fewer EXCPs are needed to re-read pages, and this statistic will increase (up to the limit of the number of pages needed) as `NEBUFF` is increased. You can make the tradeoff of costs for real memory due to increased `NEBUFF` versus the the costs for I/O. This statistic was new in `Fast/Unload` version 4.0.
Extension buffer reads	Total number of EXCPs to read pages accessed "directly." It is the number of such pages that could not be found either in the base buffers or in the extension buffer pool. This statistic was new in `Fast/Unload` version 4.0.
CPU time	This is the total CPU time used. If data is going out to a sort package, this CPU time value does not include the CPU used by the sort task. If this value is large, there is probably not much you can do about speeding up the unload without making changes to your FUEL program.
Waiting for CPU time	Total real time spent waiting for a CPU to run on. This figure is calculated by subtracting the total of all the wait times and the total CPU time from the total real time for the job. This value could actually include page wait time, or waits for synchronous BSAM I/O under CMS. If you have a high value for waiting for CPU time, it probably indicates severe system-wide contention for CPU or real memory resources.
Report buffer wait time	Total real time spent waiting for a report buffer to be written. This value should always be relatively small, unless an unload produces a huge number of reported errors (conversion, hard or missing value). `Fast/Unload` is optimized for placing data into the output data set (`FUNOUT`). A large value for report buffer wait time indicates a misuse of the report data set for outputting large amounts of data. One can reduce report buffer wait time by using a relatively large block size on the report data sets (`FUNPRINT`) DD card or FILEDEF statement.
Input wait time	Total real time spent waiting for a program input record to be read in.
Open wait time	Total real time spent waiting for OPEN's of input or output data sets to complete. This would ordinarily be a relatively small value, unless one is using a data set that requires mounting or staging from a removable storage medium such as a magnetic tape.
Output buffer wait time	Total real time spent waiting for an output buffer to be written. If this value is a relatively large chunk of total real time, you might be able to get better performance by increasing the blocksize on the output data set (on the output data set DD card or FILEDEF statement), or by using more output buffers (by setting `NOBUFF`). If the output data set is on a disk device, better performance might be achieved by using a tape device. If you are going to an external sort routine and this value is large, your efforts might be best spent tuning the sort portion of the job.
Base buffer wait time	Total real time spent waiting for the next base record to be read in. If this value is a relatively large chunk of total real time, one might be able to get better performance by increasing the base record buffer size (by setting `SBBUFF`) or by increasing the number of base record buffers (by setting `NBBUFF`). In general, however, one is not likely to see a significant improvement from having more than two base record buffers.
Extension buffer wait time	Total real time spent waiting for an extension record to be read in. If this value is a relatively large chunk of total real time, you might be able to get better performance by increasing the number of extension record buffers (by setting (`NEBUFF`) or their size (by setting `SEBUFF`). If this value is extremely high (more than half of total run time), better performance might be achieved by actually setting the number of base buffers to 1 (setting `NBBUFF` to 1), preventing base buffer read ahead from blocking extension record retrieval.
PST wait time	Total real time spent waiting for the `Fast/Unload SOUL Interface` PST. This can be either to access the next file in the `Model 204` `GROUP` being unloaded, or to access a `Model 204` page that was marked as being modified when the `Fast/Unload` job started.
Total time	Total time spent in the compile or unload step. This is both figuratively and literally the bottom line for `Fast/Unload` statistics. Since the object of `Fast/Unload` is to unload data as quickly as possible, this figure is a measure of your success. If you get an unexpectedly high total time value, you should examine the individual components of total time and attempt to correct the dominating components.
Storage for occurrence pointer arrays	Storage usage for pointers to field and fieldgroup occurrences that are referenced in the FUEL program. This statistic was new in Fast/Unload version 4.7.

@@ Line 1: / Line 1: @@
 <!-- Page name: Fast/Unload job statistics-->
-<p></p>
+Because <var class="product">Fast/Unload</var> is essentially a performance product, it is important to maintain statistics that indicate the cost of performing a <var class="product">Fast/Unload</var> and provide information that might be useful in tuning future unloads.
-Because <var class="product">Fast/Unload</var> is essentially a performance product, it is important to
-maintain
-statistics that indicate the cost of performing a <var class="product">Fast/Unload</var> and provide
-information that might be useful in tuning future unloads.
 These statistics are reported on the report data set.
-These statistics are
+These statistics are reported for the compile step, the unload step, and Ordered Index unload step separately.
-reported for
-the compile step, the unload step, and Ordered Index unload step separately.
+One of these stats, "Number of extension pages in base buffer," along
-<p></p>
+with some of the [[Fast/Unload Extraction Language (FUEL)#fstb|Table B statistics]] that are reported as part of [[Fast/Unload program parameters#fst|FSTATS processing]], can also help you detect a need for a reorganization for improving your file's performance with <var class="product">Model 204</var>.
-One of these stats, "Number of extension pages in base buffer", along
-with some of the Table B statistics that are reported as part of FSTATS
+If you use <var class="product">Fast/Unload</var> in a z/OS environment, you can choose to generate SMF records in addition to the <var class="product">Fast/Unload</var> report.
-processing, can also help you detect a need for a reorganization for
-improving your file's performance with <var class="product">Model 204</var>.
-(See [[#fstb|Description of Table B statistics]] for an explanation of the Table B statistics generated
-by FSTATS processing.)
-<p></p>
-If you use <var class="product">Fast/Unload</var> in a z/OS environment, you can choose to generate SMF
-records in addition to the <var class="product">Fast/Unload</var> report.
 See [[Fast/Unload SMF record format]] for more
 information about <var class="product">Fast/Unload</var> SMF records.
 All of the statistics, of course, are reported on SMF records; the printing of
-statistics on the FUNPRINT dataset, however, is restricted to those statistics
+statistics on the <code>FUNPRINT</code> data set, however, is restricted to those statistics
-which have a value greater tha zero.
+that have a value greater than zero.  You must apply a [[Fast/Unload custom zap#smfRecn|zap with the SMF record number]] to enable SMF record generation.
-<p></p>
-Many of the statistics refer to I/O processing of the input <var class="product">Model 204</var> file
+Many of the statistics refer to I/O processing of the input <var class="product">Model 204</var> file (or files, in the case of a group).
-(or files, in the case of a group).
 There are two groups of these I/O statistics:
-<table>
+<table class="thJustBold">
-<tr><th>Base buffer statistics</th><td>These occur only during the Unload phase, and are for accessing Table B pages in page number order, that is, accessing the the base records processed during the unload.</td></tr>
+<tr><th>Base buffer statistics</th>
-<tr><th>Extension page statistics</th><td>These can apply during any phase, and are for directly reading pages from the <var class="product">Model 204</var> file in no pre-determined order. These are called extension page statistics because one type of direct page access is for extension record access during the Unload phase. There are other types of direct page access, such as reading the Table A field and coded value information during the Compile phase, reading the Existence Bit Map, and reading B-tree, list, and index bitmap pages during the Index unload phase. <p></p> Note that the extension pages actually accessed in the unload phase depend on the type of processing in your FUEL program. A UAI operation will access all extensions of all records being unloaded; FSTATS processing will access all extensions of all records processed; some FUEL constructs, such as DELETE or using "#" or "*" for a field occurrence, cause all extensions of a record to be processed; otherwise, only as much of a logical <var class="product">Model 204</var> record is processed as needed to reference the fields used in the FUEL program.</td></tr>
+<td>These occur only during the Unload phase, and are for accessing Table B pages in page number order, that is, accessing the the base records processed during the unload.</td></tr>
+<tr><th nowrap>Extension page statistics</th>
+<td>These can apply during any phase. They are for directly reading pages from the <var class="product">Model 204</var> file in no predetermined order. These are called extension page statistics, because one type of direct page access is for extension record access during the Unload phase.
+<p>
+There are other types of direct page access, such as reading the Table A field and coded value information during the Compile phase, reading the Existence Bit Map, and reading B-tree, list, and index bitmap pages during the Index unload phase. </p>
+<p>
+Note that the extension pages actually accessed in the unload phase depend on the type of processing in your FUEL program: </p>
+<ul>
+<li>A <var>UAI</var> operation accesses all extensions of all records being unloaded. </li>
+<li><var>FSTATS</var> processing accesses all extensions of all records processed.</li>
+<li>Some FUEL constructs, such as <var>DELETE</var> or using <code>#</code> or <code>*</code> for a field occurrence, cause all extensions of a record to be processed.</li>
+</ul>
+<p>
+Otherwise, only as much of a logical <var class="product">Model 204</var> record is processed as needed to reference the fields used in the FUEL program.</p></td></tr>
 </table>
-<p></p>
-The following is
+The following is a description of statistics that are reported:
-a description of statistics that are reported:
+<table class="thJustBold">
-<table>
+<tr><th>Base buffer reads</th>
-<tr><th>Base buffer reads</th><td>Total number of EXCPs issued to read a track or group of tracks into a base buffer. It is directly affected by the SEBUFF parameter and the number of <var class="product">Model 204</var> input records (that is, base records) processed. <p></p> This statistic is new in <var class="product">Fast/Unload</var> version 4.0.</td></tr>
+<td>Total number of EXCPs issued to read a track or group of tracks into a base buffer. It is directly affected by the <var>[[Fast/Unload program parameters#SEbuff=n|SEBUFF]]</var> parameter and the number of <var class="product">Model 204</var> input records (that is, base records) processed.
-<tr><th>Base buffer waits</th><td>Total number of WAITs issued after initiating reads into the base buffers; the ratio of EXCPs to WAITs indicates, to some extent, the degree of overlap of CPU with I/O processing. <p></p> This statistic is new in <var class="product">Fast/Unload</var> version 4.0.</td></tr>
+<p>
-<tr><th>Extension pg in base buffs</th><td>Total number of <var class="product">Model 204</var> file pages accessed "directly" which were currently available in pages read into base buffers. This indicates how many extension records were "physically close" to their base record during the Unload phase; specifically, within the SBBUFF times NBBUFF tracks starting with the group of SBBUFF tracks containing the base record. A large number for this indicates that <var class="product">Model 204</var> database processing can efficiently access the extension records accompanying a base record, with little physical disk manipulation (or even none, with caching devices). As mentioned in [[#fstb|Description of Table B statistics]], this information can be used in addition to the FSTATS information about extension records, to help you determine possible benefits obtained by reorganizing a file; if extension records are frequently accessed in <var class="product">Model 204</var> processing and a large number of them are not physically close to the base record, performance may suffer to some degree. <p></p> This statistic is new in <var class="product">Fast/Unload</var> version 4.0.</td></tr>
+This statistic was new in <var class="product">Fast/Unload</var> version 4.0.</p></td></tr>
-<tr><th>Extension pg in exten pool</th><td>Total number of <var class="product">Model 204</var> file pages accessed "directly" which were not currently available in pages read into base buffers but which were available in the pool of extension pages. The larger this number is, the fewer EXCPs are needed to re-read pages, and this statistic will increase (up to the limit of the number of pages needed) as NEBUFF is increased; you can make the tradeoff of costs for real memory due to increased NEBUFF versus the the costs for I/O. <p></p> This statistic is new in <var class="product">Fast/Unload</var> version 4.0.</td></tr>
-<tr><th>Extension buffer reads</th><td>Total number of EXCPs to read pages accessed "directly". It is the number of such pages that could not be found either in the base buffers or in the extension buffer pool. <p></p> This statistic is new in <var class="product">Fast/Unload</var> version 4.0.</td></tr>
+<tr><th>Base buffer waits</th>
-<tr><th>CPU time</th><td>This is the total CPU time used. If data is going out to a sort package, this CPU time value does not include the CPU used by the sort task. If this value is large, there is probably not much you can do about speeding up the unload without making changes to your FUEL program.</td></tr>
+<td>Total number of WAITs issued after initiating reads into the base buffers. The ratio of EXCPs to WAITs indicates, to some extent, the degree of overlap of CPU with I/O processing.
-<tr><th>Waiting for CPU time</th><td>Total real time spent waiting for a CPU to run on. This figure is calculated by subtracting the total of all the wait times and the total CPU time from the total real time for the job. This value could actually include page wait time, or waits for synchronous BSAM I/O under CMS. If you have a high value for waiting for CPU time, it probably indicates severe system-wide contention for CPU or real memory resources.</td></tr>
+<p>
-<tr><th>Report buffer wait time</th><td>Total real time spent waiting for a report buffer to be written. This value should always be relatively small unless an unload produces a huge number of reported errors (conversion, hard or missing value). <var class="product">Fast/Unload</var> is optimized for placing data into the output data set (FUNOUT). A large value for report buffer wait time indicates a misuse of the report data set for outputting large amounts of data. One can reduce report buffer wait time by using a relatively large block size on the report data sets (FUNPRINT) DD card or FILEDEF statement.</td></tr>
+This statistic was new in <var class="product">Fast/Unload</var> version 4.0.</p></td></tr>
-<tr><th>Input wait time</th><td>Total real time spent waiting for a program input record to be read in.</td></tr>
-<tr><th>Open wait time</th><td>Total real time spent waiting for OPEN's of input or output data sets to complete. This would ordinarily be a relatively small value, unless one is using a data set that requires mounting or staging from a removable storage medium such as a magnetic tape.</td></tr>
+<tr><th>Extension pg in base buffs</th>
-<tr><th>Output buffer wait time</th><td>Total real time spent waiting for an output buffer to be written. If this value is a relatively large chunk of total real time one might be able to get better performance by increasing the blocksize on the output data set (on the output data set DD card or FILEDEF statement) or by using more output buffers (by setting the NOBUFF). If the output data set is on a disk device, better performance might be achieved by using a tape device. If you are going to an external sort routine and this value is large, your efforts might be best spent tuning the sort portion of the job.</td></tr>
+<td>Total number of <var class="product">Model 204</var> file pages accessed "directly" which were currently available in pages read into base buffers. This indicates how many extension records were "physically close" to their base record during the Unload phase; specifically, within the <var>[[Fast/Unload program parameters#Sbbuff=n|SBBUFF]]</var> times <var>[[Fast/Unload program parameters#Sbbuff=n|NBBUFF]]</var> tracks starting with the group of <var>SBBUFF</var> tracks containing the base record.
-<tr><th>Base buffer wait time</th><td>Total real time spent waiting for the next base record to be read in. If this value is a relatively large chunk of total real time, one might be able to get better performance by increasing the base record buffer size (by setting SBBUFF) or by increasing the number of base record buffers (by setting NBBUFF). In general, however, one is not likely to see a significant improvement from having more than 2 base record buffers.</td></tr>
+<p>
-<tr><th>Extension buffer wait time</th><td>Total real time spent waiting for an extension record to be read in. If this value is a relatively large chunk of total real time one might be able to get better performance by increasing the number of extension record buffers (by setting (NEBUFF) or their size (by setting SEBUFF). If this value is extremely high (more than half of total run time) better performance might be achieved by actually setting the number of base buffers to 1 (setting NBBUFF to 1), preventing base buffer read ahead from blocking extension record retrieval.</td></tr>
+A large number for this statistic indicates that <var class="product">Model 204</var> database processing can efficiently access the extension records accompanying a base record, with little physical disk manipulation (or even none, with caching devices). As mentioned in [[Fast/Unload Extraction Language (FUEL)#fstb|Description of Table B statistics]], this information can be used in addition to the <var>FSTATS</var> information about extension records, to help you determine possible benefits obtained by reorganizing a file. If extension records are frequently accessed in <var class="product">Model 204</var> processing and a large number of them are not physically close to the base record, performance may suffer to some degree. </p>
-<tr><th>PST wait time</th><td>Total real time spent waiting for the <var class="product">Fast/Unload SOUL Interface</var> PST. This can be either to access the next file in the <var class="product">Model 204</var> GROUP being unloaded, or to access a <var class="product">Model 204</var> page that was marked as being modified when the <var class="product">Fast/Unload</var> job started.</td></tr>
+<p>
-<tr><th>Total time</th><td>Total time spent in the compile or unload step. This is both figuratively and literally the bottom line for <var class="product">Fast/Unload</var> statistics. Since the object of <var class="product">Fast/Unload</var> is to unload data as quickly as possible, this figure is a measure of your success. If you get an unexpectedly high total time value,  you should examine the individual components of total time and attempt to correct the dominating components.</td></tr>
+This statistic was new in <var class="product">Fast/Unload</var> version 4.0.</p></td></tr>
+<tr><th nowrap>Extension pg in exten pool</th>
+<td>Total number of <var class="product">Model 204</var> file pages accessed "directly" which were not currently available in pages read into base buffers but which were available in the pool of extension pages. The larger this number is, the fewer EXCPs are needed to re-read pages, and this statistic will increase (up to the limit of the number of pages needed) as <var>NEBUFF</var> is increased. You can make the tradeoff of costs for real memory due to increased <var>NEBUFF</var> versus the the costs for I/O.
+<p>
+This statistic was new in <var class="product">Fast/Unload</var> version 4.0.</p></td></tr>
+<tr><th>Extension buffer reads</th>
+<td>Total number of EXCPs to read pages accessed "directly." It is the number of such pages that could not be found either in the base buffers or in the extension buffer pool.
+<p>
+This statistic was new in <var class="product">Fast/Unload</var> version 4.0.</p></td></tr>
+<tr><th>CPU time</th>
+<td>This is the total CPU time used. If data is going out to a sort package, this CPU time value does not include the CPU used by the sort task. If this value is large, there is probably not much you can do about speeding up the unload without making changes to your FUEL program.</td></tr>
+<tr><th>Waiting for CPU time</th>
+<td>Total real time spent waiting for a CPU to run on. This figure is calculated by subtracting the total of all the wait times and the total CPU time from the total real time for the job. This value could actually include page wait time, or waits for synchronous BSAM I/O under CMS.
+<p>
+If you have a high value for waiting for CPU time, it probably indicates severe system-wide contention for CPU or real memory resources.</p></td></tr>
+<tr><th>Report buffer wait time</th>
+<td>Total real time spent waiting for a report buffer to be written. This value should always be relatively small, unless an unload produces a huge number of reported errors (conversion, hard or missing value). <var class="product">Fast/Unload</var> is optimized for placing data into the output data set (<code>FUNOUT</code>).
+<p>
+A large value for report buffer wait time indicates a misuse of the report data set for outputting large amounts of data. One can reduce report buffer wait time by using a relatively large block size on the report data sets (<code>FUNPRINT</code>) DD card or FILEDEF statement.</p></td></tr>
+<tr><th>Input wait time</th>
+<td>Total real time spent waiting for a program input record to be read in.</td></tr>
+<tr><th>Open wait time</th>
+<td>Total real time spent waiting for OPEN's of input or output data sets to complete. This would ordinarily be a relatively small value, unless one is using a data set that requires mounting or staging from a removable storage medium such as a magnetic tape.</td></tr>
+<tr><th>Output buffer wait time</th>
+<td>Total real time spent waiting for an output buffer to be written. If this value is a relatively large chunk of total real time, you might be able to get better performance by increasing the blocksize on the output data set (on the output data set DD card or FILEDEF statement), or by using more output buffers (by setting <var>[[Fast/Unload program parameters#NObuff=n|NOBUFF]]</var>). If the output data set is on a disk device, better performance might be achieved by using a tape device.
+<p>
+If you are going to an external sort routine and this value is large, your efforts might be best spent tuning the sort portion of the job.</p></td></tr>
+<tr><th>Base buffer wait time</th>
+<td>Total real time spent waiting for the next base record to be read in. If this value is a relatively large chunk of total real time, one might be able to get better performance by increasing the base record buffer size (by setting <var>SBBUFF</var>) or by increasing the number of base record buffers (by setting <var>NBBUFF</var>). In general, however, one is not likely to see a significant improvement from having more than two base record buffers.</td></tr>
+<tr><th nowrap>Extension buffer wait time</th>
+<td>Total real time spent waiting for an extension record to be read in. If this value is a relatively large chunk of total real time, you might be able to get better performance by increasing the number of extension record buffers (by setting (<var>NEBUFF</var>) or their size (by setting <var>SEBUFF</var>).
+<p>
+If this value is extremely high (more than half of total run time), better performance might be achieved by actually setting the number of base buffers to 1 (setting <var>NBBUFF</var> to 1), preventing base buffer read ahead from blocking extension record retrieval.</p></td></tr>
+<tr><th>PST wait time</th>
+<td>Total real time spent waiting for the <var class="product">Fast/Unload SOUL Interface</var> PST. This can be either to access the next file in the <var class="product">Model 204</var> <var>GROUP</var> being unloaded, or to access a <var class="product">Model 204</var> page that was marked as being modified when the <var class="product">Fast/Unload</var> job started.</td></tr>
+<tr><th>Total time</th>
+<td>Total time spent in the compile or unload step. This is both figuratively and literally the bottom line for <var class="product">Fast/Unload</var> statistics. Since the object of <var class="product">Fast/Unload</var> is to unload data as quickly as possible, this figure is a measure of your success. If you get an unexpectedly high total time value,  you should examine the individual components of total time and attempt to correct the dominating components.</td></tr>
+<tr><th>Storage for occurrence pointer arrays</th>
+<td>Storage usage for pointers to field and fieldgroup occurrences that are referenced in the FUEL program.
+<p>
+This statistic was new in Fast/Unload version 4.7.</p>
 </table>
 ==See also==
-[[Fast/Unload overview#WIKFUN$$topics|Fast/Unload topics]]
+{{Template:Fast/Unload topic list}}

Fast/Unload job statistics: Difference between revisions

Latest revision as of 14:35, 11 December 2015

See also

Navigation menu