Fast/Reload statements

Overview

All Fast/Reload programs must begin with a FILELOAD or FLOD command and end with an END statement. All statements between the FILELOAD or FLOD and the END are converted into machine code by the Fast/Reload compiler or are used to set environmental parameters by the Fast/Reload compiler. An example of a very simple Fast/Reload program is:

FILELOAD -1,-1,0,1000000,10000,10000,,68 G DUMMY=1,0,X'8000' FIELD1=1,10 END

Fast/Reload supports all the statements supported by FLOD and FILELOAD except the following:

The L statement.
The ENDL statement.
The F statement.
The ENDF statement.
The UP statement.
The DOWN statement.
The DELETE statement.

In addition, the X'4000' mode bit used to delete a field occurrence or occurrences is not currently supported. Finally, FLOD exits are not currently supported.

If you have FILELOAD or FLOD programs that contain statements not supported by Fast/Reload, you can still run the programs with the same load module you use for Fast/Reload programs. The FLODX and FILELOADX commands invoke the original Model 204 FLOD and FILELOAD programs instead of Fast/Reload.

Fast/Reload attempts to copy the FLOD/FILELOAD program to the temporary procedure indicated by the FRELPREV system parameter (which defaults to -1). If Fast/Reload determines that the FLOD or FILELOAD program cannot be processed by Fast/Reload but can be processed by standard FLOD or FILELOAD, the FRELPREV temporary procedure is opened for input, and the program is passed to standard FLOD or FILELOAD to be read again.

For more information on both supported and unsupported statements, see Category:File loading and reorganization. In addition to the standard statement set, Fast/Reload supports the extra statements described in the following subsections: The OPTIONS statement and The LAI statement.

Note: Each statement must begin in column one, since a blank in column one of a FLOD statement indicates the "read-and-load-a-field" statement.

The OPTIONS statement

This statement can appear anywhere inside a Fast/Reload program and is used to set environmental parameters that affect the way the load will be performed. The OPTIONS statement can be abbreviated to its first three or more characters. For example, the following are all valid OPTIONS statements:

OPTIONS TAPEI FOOFOO OPTION TAPEI FOOFOO OPTI TAPEI FOOFOO OPT TAPEI FOOFOO

Multiple parameters can be set via the OPTIONS statement. They are described in the following subsections. The minimum required abbreviation for each is indicated in uppercase.

ANYorder

This option only has an effect when data is being loaded into a hash or sort key file. When data is being loaded into one of these files, performance will generally be good if the data is being loaded in hash or sort key order. If the data being loaded is not in this order, load performance could be disastrously affected. Fast/Reload normally does not continue in such a case. The ANYORDER option overrides this behavior and lets the file load continue.

In general, it is better to determine the reason records are out of order and to correct the problem, than to use the ANYORDER option to ignore it.

If you are using LAI, the ordering can be done by the HASH or SORT option of the UAI statement in Fast/Unload.

Otherwise, to obtain the full track I/O benefits of Fast/Reload, you should ensure that your input records are in the correct order. You must sort the TAPEI dataset in order by the SORT key, or by the hash code of the HASH key. This can be achieved using the M204HASH utility, as described in Hash key files.

BAVail [page_type] size

This option has the following two beneficial effects:

If size (default -1) is non-negative (and the file is not SORTED), the use of the BRESERVE (= bres) parameter is changed during the file load. A record will start within remain <= size + bres bytes at the end of the page if and only if the record fits in its entirety on that page. That is, the total record size is less than or equal to remain. This ensures that no base record is split with BRESERVE bytes or fewer in the base record, while using more available space on a Table B page than the default algorithm.
If size is greater than zero, Fast/Reload ensures, during the file load, that at least size bytes are unused on a Table B page. This provides for some record growth by subsequent updates, for example, during an Online run, without shifting fields to an extension record.

BAVAIL is ignored if Fast/Reload is not operating using full track I/O. A base record is the first part of a Table B record, as opposed to the subsequent parts of a Table B record, which are called extension records.

Parameters

size is -1 (the default) or a non-negative number. The smallest value of size for any BAVAIL option is -1. The largest value is 6130 minus the length of all preallocated fields, minus 3 if FILEORG is SORTED, minus 1 if FILEORG is HASHED, minus 4 if FILEORG is UNORDERED.

page_type can be any of the following:

ALL	This applies to all Table B pages. If `size` is non-negative, then `Fast/Reload` guarantees that every table B page contains at least `size` unused bytes.
BASE	This applies to any page that contains one or more base records. If `size` is non-negative, then `Fast/Reload` guarantees that any page with a base record contains at least `size` unused bytes.
EXT	This applies to any page that contains an extension record. If `size` is non-negative, then `Fast/Reload` guarantees that any page with an extension record contains at least `size` unused bytes.

If page_type is omitted, size applies to all Table B pages.

Usage

Multiple OPTIONS BAVAIL specifications will override the value(s) previously specified. For example:

OPTIONS BAVAIL 100 BAVAIL BASE 150

will set aside 150 bytes on each page with a Table B base record, and 100 bytes on all other Table B pages, but

OPTIONS BAVAIL BASE 150 BAVAIL 100

will set aside 100 bytes on all Table B pages.

The default BAVAIL for each page type is -1. If the value of all of them is -1, Fast/Reload does not specifically reserve any Table B space, but uses the normal Model 204 space algorithm determined by BRESERVE, ensuring that a record does not start within the last BRESERVE bytes of a page.

If the file being loaded is not a SORTED file (FILEORG X'01'), a non-negative setting for any page type will modify the effect of the BRESERVE parameter for all Table B pages. Let BA be the maximum of all BAVAIL values; if it is non-negative, then if an entire Table B record is less than BRESERVE bytes long, it can be stored within the last BA + BRESERVE bytes on the page, but otherwise a Table B record is not started in that area.

Therefore, you may want to set a temporary value for BRESERVE during the file load which differs from the value used during online updating. For example:

OPEN FILE mumble RESET BRESERVE 300 FILELOAD -1,-1,0,1000000,1000000 OPTIONS BAVAIL 400 LAI END RESET BRESERVE 800

Note: The correct values for BRESERVE and BAVAIL are file-specific, and depend on characteristics such as record sizes, the cost to your applications of extension records, the importance of conserving Table B space, and updating characteristics. Like any tuning exercise, this requires analysis and experimentation. The numbers in the above example could be wrong for your file, in fact, you may do better with an online BRESERVE that is smaller than the file load BRESERVE, and BAVAIL may be smaller than both, larger than both, or in between.

Examples

In the following examples:

The file is a simple ENTRY ORDER file (FILEORG X'00').
There are no preallocated fields.
Each page contains 6140 usable bytes. (A page has 6144 bytes prior to the trailer, but there are 2 bytes used to determine the number of records on a page, and 2 used for the amount of unused space.)
Dataln means the total length of data in a record, basln means the space used for a base record, and extln means the space used for an extension record.
Each base record or extension record requires a 2-byte page locator, and contains a 3-byte area which is zero or has the record number of a subsequent extension record. So, for example, if a record is completely stored in a base record, basln = dataln + 5.
Each example illustrates a case starting with a base record on an empty page.
The resulting space allocation shows each page as the letter "P" and a page number, followed by extension and/or base records and free space enclosed in square brackets. For example, the following shows two records stored on one page:
Rec0 dataln=100 Rec1 dataln=300 P0 [Rec0 basln=105] [Rec1 basln=305] [free=5730]
Extension records are shown with the base record ID followed by a period and extension number. For example, this shows one record stored on three pages:
Rec0 dataln=12570 P0 [Rec0 basln=6140] P1 [Rec0.1 extln=6140] P2 [Rec0.2 extln=305] [free=5835]
Results are shown for different values for BAVAIL. With the default value of BAVAIL=-1, Fast/Reload uses the standard Model 204 algorithm for allocating Table B space during a file load. This standard algorithm is always used when Fast/Reload is not using full track I/O.

The following example shows that BAVAIL=0 can be used for maximum packing on a page, and BAVAIL>0 can be used to ensure free space on a page; each of them ensure a base record is not split in the first BRESERVE bytes:
Rec0 dataln = 6000 Rec1 dataln = 130 BRESERVE = 300 Using BAVAIL = -1: P0 [Rec0 basln=6005] [free=135] P1 [Rec1 basln=135] [free=6005] Using BAVAIL = 0: P0 [Rec0 basln=6005] [Rec1 basln=135] Using BAVAIL = 135: P0 [Rec0 basln=6005] [free=135] P1 [Rec1 basln=135] [free=6005]
The following example shows that without BAVAIL, you cannot guarantee free space on a page:
Rec0 dataln=130 Rec1 dataln=6000 (60 100-byte fields) BRESERVE = 300 Using BAVAIL = -1: P0 [Rec0 basln=135] [Rec1 basln=6005] Using BAVAIL = 0: P0 [Rec0 basln=135] [Rec1 basln=6005] Using BAVAIL = 135: P0 [Rec0 basln=135] [Rec1 basln=5805] [free=200] P1 [Rec1.1 extln=205] [free=5935]
The following example shows tighter packing and fewer extension records:
Rec0 dataln=6000 (60 100-byte fields) Rec1 dataln=100 Rec2 dataln=6100 (61 100-byte fields) Rec3 dataln=100 Rec4 dataln=6000 (60 100-byte fields) BRESERVE = 200 Using BAVAIL = -1: P0 [Rec0 basln=6005] [free=135] P1 [Rec1 basln=105] [Rec2 basln=6005] [free=30] P2 [Rec2.1 extln=105] [Rec3 basln=105] [Rec4 basln=5905] [free=25] P3 [Rec4.1 extln=105] [free=6035] Using BAVAIL = 0: P0 [Rec0 basln=6005] [Rec1 basln=105] [free=30] P1 [Rec2 basln=6105] [free=35] P2 [Rec3 extln=105] [Rec4 basln=6005] [free=30]
The following example shows use of BAVAIL that increases the extension records at file load time, but by leaving space on all pages, records can grow without producing extension records which are stored on pages "far away" from the base page:
Rec0 .. Rec5 dataln=4000 (40 100-byte fields) BRESERVE = 300 Using BAVAIL = -1: P0 [Rec0 basln=4005] [Rec1 basln=2105] [free=30] P1 [Rec1.1 extln=1905] [Rec2 basln=4005] [free=230] P2 [Rec3 basln=4005] [Rec4 basln=2105] [free=30] P3 [Rec4.1 extln=1905] [Rec5 basln=4005] [free=230] Using BAVAIL = 300: P0 [Rec0 basln=4005] [Rec1 basln=1805] [free=330] P1 [Rec1.1 extln=2205] [Rec2 basln=3605] [free=330] P2 [Rec2.1 extln=405] [Rec3 basln=4005] [Rec4 basln=1405] [free=325] P3 [Rec4.1 extln=2605] [Rec5 basln=3205] [free=330] P4 [Rec5.1 extln=805] [free=5335]

DKOnly

In many cases, Fast/Reload bypasses the Model 204 disk buffer monitor for Table B pages and instead uses its own full track buffers for output. This can, in certain instances, result in data being loaded into different Table B pages than standard FLOD or FILELOAD would use. An example of this is non-empty reuse record number files. You can force Fast/Reload to use the Model 204 disk buffer monitor by specifying the DKONLY option. This option can result in a significant increase in the real and CPU time required to perform a file load.

ERRCont

This parameter lets you complete a reload despite errors that would normally cause a MSIR.0316 user restart. This can be useful, for example, in a case where you know you have some input data that will cause an error, but you want to use a reload for reformatting purposes despite the problematic fields.

If you specify ERRCONT to "continue on error" a Fast/Reload program, you probably need to increase the Model 204 ERMX parameter setting to prevent a session termination because of excessive errors.

Note: Using ERRCONT may place your data at risk, so use this parameter only in cases where you know your data and have taken due precautions and backups beforehand.

The following FILELOAD code uses the ERRCONT parameter:

FILELOAD -1,-1,0,1000000,10000,10000,10000,50 OPTIONS ERRCONT LAI END

FTOnly

In many cases, Fast/Reload bypasses the Model 204 disk buffer monitor for Table B pages and instead uses its own full track buffers for output. In certain cases, though, Fast/Reload will automatically drop back to using the Model 204 disk buffer monitor. When it does this, a file load might take considerably longer than expected. If you do not wish Fast/Reload to perform this automatic dropback, simply specify the FTONLY option. FTONLY will result in the file load being terminated when an automatic dropback would ordinarily be done.

NObuff n_out_buffs

NOBUFF must be followed by a number indicating the number of full track output buffers Fast/Reload is to use. When Fast/Reload is bypassing the Model 204 disk buffer monitor and performing full track I/O to Table B, it uses a certain number of full track buffers. These buffers must each be large enough to hold the data that will fit on a single track of the disk devices containing the database file. This means that the buffers will each be 43,288 bytes long if you are using 3380s, 49,472 bytes long if you are using 3390s, and 49,472 bytes long if you are using both.

Ordinarily Fast/Reload uses three full track output buffers. This is probably sufficient for most purposes. On rare occasions, it might be possible to get some speed improvements by using more than three output buffers. In these cases, simply specify the NOBUFF option followed by a number greater than 3.

NOBUFF must always be greater than or equal to 3.

NXBuff n_tableX_buffs

NXBUFF must be followed by a number indicating the number of full track Table X buffers Fast/Reload is to use.

NXBUFF must always be greater than or equal to 3.

NOVAlidate

NOVALIDATE indicates that no buffer pointer validation is to be performed. For example, in the following statement, buffer pointer validation would make sure that the input record was at least 107 bytes long:

FIELDA=99,9

Without buffer pointer validation, the above statement would simply load data off the end of the input buffer, which would result in FIELDA being loaded with garbage, or perhaps in certain cases, would result in an addressing exception because the area past the end of the buffer is not allocated. Buffer pointer validation is probably more important with index register based references to the input buffer, because these types of references are more likely to have bad values in the index register and so go outside the current input buffer:

FIELDC=4|1,20

With buffer pointer validation, if a buffer reference is determined to be invalid, an error message is issued indicating the input record number as well as the invalid buffer offset and length, as in the following example:

MSIR.0892: Input error: input record number: 1, start position/length: 99/9

Standard FILELOAD started performing buffer pointer validation in Model 204 V5R1.

While buffer pointer validation is probably a good idea, it does have a slight performance cost. If there appears to be little chance of a buffer pointer error in a FILELOAD program, and performance is at an absolute premium, you can use the NOVALIDATE option to eliminate the cost of the buffer pointer validity checks.

Since LAI programs have no explicit buffer references, the NOVALIDATE option has no effect on LAI programs.

STAtc stats_intvl

STATC must be followed by a number indicating the number of database file records that Fast/Reload will create before updating Model 204 statistics. To speed up processing, Fast/Reload does not ordinarily update Model 204 statistics (since last, system, etc.) until the end of a file load.

If you want these statistics to be updated more frequently, you can specify a STATC value. If you specify OPTION STATC 10, Model 204 statistics are updated for every 10th record created in the database file. OPTION STATC 0 and OPTION STATC 1 are equivalent.

TAPei ddname

TAPEI must be followed by string indicating the DDNAME to be used instead of TAPEI for the input sequential file containing the data to be loaded. For example if you wish to load data from DDNAME HOMER, simply specify OPTION TAPEI HOMER. This allows more than one input stream to be used to load data in a single BATCH204 or Online run.

The LAI statement

The LAI statement allows data created with the Fast/Unload UAI (Unload All Information) statement to be loaded into a database file. LAI stands for Load All Information and has several optional parameters.

Syntax

LAI [DElfield] [FAMsplit] [NEWfgid] [NOfdef] [NOIndex] [NOProcs]

where the parameters can be abbreviated with only the part shown in uppercase, and where:

DElfield	Means that if a field that is not defined in the target file is found in the input dataset, that is, was unloaded by `UAI`, that field should simply be deleted. If this parameter is not specified, the presence of an undefined field would terminate the `LAI`with an error. This statement provides an efficient way of deleting fields during a reorg. Note: This parameter has no effect if the `NOFDEF` parameter is notalso specified. This is because if `NOFDEF` is not specified, any unloaded field will be automatically defined in the target file.
FAMsplit	Means that if any two names (two fields, a field and an alias, or two aliases) were part of a single `Sir2000 Field Migration Facility` family in the unloaded file but are now either separate fields or part of separate `Sir2000 Field Migration Facility` families, the values should be loaded into each field or family. Effectively, this "splits" the original family into multiple components. If `FAMSPLIT` is not specified, an attempt to do such a split causes the `FLOD` or `FILELOAD` to be terminated. The most likely use of this parameter is to convert a file that is being controlled by `Sir2000 Field Migration Facility` into one that is not, but where any fields or aliases that were in the original file could still be referred to with SOUL programs. This might be useful for sending a non-`Sir2000 Field Migration Facility` version of a file to a site that does not have `Sir2000 Field Migration Facility`. Since `Sir2000 Field Migration Facility` makes sure that related fields and aliases automatically stay in synch, it is generally not a good idea to update the non-`Sir2000 Field Migration Facility` version of the file. See the "File Reorganizations" chapter in the `Sir2000 Field Migration Facility Reference Manual` for considerations about preserving SIRFIELD information using `Fast/Reorg`.
NEWfgid	Means that fieldgroup IDs from the TAPEI input file are not copied to created fieldgroups, but rather that fieldgroup IDs within each record are created starting with 1.
NOfdef	Means that field definitions unloaded by `UAI` are not to be used to automatically define a field if it does not exist in the new file. If this parameter is not specified, all fields that have not been explicitly defined in the new database file but were defined in the old database file (unloaded with `UAI`) will be defined with exactly the same attributes that they had in the old database file.
NOIndex	Means that Ordered Index data unloaded by UAI is not to be loaded by `LAI`. When this optionis used, visible Ordered Index data will be regenerated and sorted by `LAI` and invisible Ordered Index data will be lost. This parameter has no effect if the `OINDEX` parameter was not specified on the `UAI` command for the unload.
NOProcs	Means that any procedures and procedure aliases unloaded by `UAI` are not to be loaded by `LAI`. Procedure and alias records in the input dataset will be skipped. For more information about loading procedures, see Fast/Reload procedure handling.

Fast/Reload programs using LAI are quite simple. For example, the following is a valid Fast/Reload program:

FILELOAD -1,-1,0,1000000 LAI END

Note: The LAI statement has this limitation: it must be the only statement that appears in a Fast/Reload program other than the OPTION statement.

LAI compared to FILELOAD/FLOD

When using LAI, some of the parameters on the FILELOAD or FLOD statement have a slightly different meaning than when doing a standard FLOD. These parameters are:

Parameter 1
This parameter has the same meaning as in standard FILELOAD/FLOD. That is, this is the maximum number of records that will be loaded into the database file.
Parameter 2
This parameter is ignored when doing an LAI.
Parameter 3
In standard FILELOAD/FLOD programs this indicates the number of physical input records in TAPEI to be skipped. When doing an LAI, this indicates the number of unloaded Table B records to be skipped.

This parameter, along with parameter 1, lets you split a reload into pieces. For example, the following statements load the first million records unloaded via UAI, then the next million records and finally the remaining records.

FILELOAD 1000000,-1,0,100000,100000 LAI END FILELOAD 1000000,-1,1000000,100000,100000 LAI END FILELOAD -1,-1,2000000,100000,100000 LAI END

All other parameters for the FILELOAD statement have the same meaning as when doing a non-LAI FILELOAD.

Using LAI with UAI

LAI is closely related to the UAI statement in Fast/Unload. Ordinarily, the interaction between these two is quite simple. However, in cases where sorting of the data unloaded by Fast/Unload is desired and/or if the data is being loaded into a hash or sort key file, the interaction can become more complex.

When loading data into a hash or sort key file with the LAI statement, the data must have been unloaded with a UAI statement that specified the HASH or SORT parameter and specified the new file's hash or sort key as the first key on the UAI statement. If this is not the case, the LAI will not load any data. If the UAI statement is coded correctly, the data unloaded by UAI will be in the order that the data will eventually be in the database file being loaded. In these cases, Fast/Reload will never have to make more than one pass through Table B and will actually be able to use full track I/O to write the Table B pages if the data is being loaded into an empty hash or sort key file.

If the UAI statement is coded incorrectly, however, the unloaded data will be in an order different from that which the data will loaded into the destination database file. The impact of this on performance could be disastrous. Consequently, Fast/Reload will not ordinarily continue in the case when data is detected out of order. You can override this behavior with the ANYORDER option, but at least in the case of reorganizations, it is generally faster to unload the data again with the correct parameters rather than trying to load the data in incorrect order.

The types of errors on the UAI statement that can cause data to be in incorrect order are:

Specifying SORT on the UAI and then attempting to load the data into a hash key file.
Specifying HASH on the UAI and then attempting to load the data into a sort key file.
Specifying a BSIZE on the UAI HASH statement different from the BSIZE in the hashed file being loaded.

Note: If you are loading the data into a sorted file, there are several cases of the UAI SORT statement that render the sort key unusable. See UAI SORT or HASH and field unload order.

If you are loading the data into a non-hash key, non-sort key file, there is nothing wrong with loading data unloaded with a UAI command that had a SORT or HASH key specified. In fact, if you want to maintain good locality for data with similar values for a particular field without incurring the overhead of maintaining a sort key file, you can specify that field as a sort key on a UAI and simply load the data into a non-sort key file.

Loading null data

Standard Model 204 FLOD and FILELOAD do not allow null data (stringdata with a length of 0) to be loaded into the database file. That is, if after stripping blanks and zeros the resulting field has length zero, no data will be loaded into the database file. This is the case, even if you are trying to load null data that had been dumped using the SOUL PAI statement. Thus, it is extremely difficult if not impossible to reorganize a database file containing null data using FLOD or FILELOAD.

Fast/Reload allows you to do this in a couple of ways:

Any null data unloaded via Fast/Unload's UAI statement will be loaded as null data by Fast/Reload's LAI statement.
The X'0001' mode bit lets you load null data as a zero length string into the database file.

For example, the following FILELOAD program will load null data dumped via the PAI statement back into the new database file as null data:

FILELOAD -1,-1,0,1000000,10000,10000,,68 I 3 #1 G =4,5,* I 1 #2 =3,6|1,= I 1,,,1|1 =2 #3 I 2,1,2,-7,-1|1 D 5,0|1=8|1,0|2,X'0801' =1 #4 PROC.ID=1,0,X'8000' END

The key part of the above example is the X'0001' mode bit on the D statement. Note also, that for PAI type reloads, it is generally a good idea to turn off blanks stripping. This is controlled by the X'0800' mode bit on the D statement.