Deferred update feature

Overview

Whenever you change or add a large number of records to a Model 204 file, the process can be made more efficient, in terms of execution time, disk I/O, and storage utilization, by deferring the changes or additions that update Tables C and D: updates involving KEY, NUMERIC RANGE, ORDERED, and FRV fields. In the Model 204 deferred update feature, the deferred updates to Tables C and D are stored in deferred update data sets while Tables A and B are updated. The deferred updates are then sorted and applied to Tables C and D.

This article describes

• Basic Model 204 deferred update feature
• How the feature is implemented
• Several different job forms-number of job steps-that the feature can take.

When a file is opened in deferred update mode, Model 204 protects the file by making it impossible to free the file in that Model 204 run. This limitation is necessary to protect the file from being updated before the deferred updates have been applied. All files are closed completely (free) at the end of the Model 204 job.

Deferred updates and large (FILEORG X'200') files

The FILEORG parameter X'200' setting supports large files, allowing files to hold up to 48M records.

The deferred index update feature was implemented well before the existence of FILEORG X'200' files, when the maximum number of records possible in a file was 16M. At that time, only three bytes were required to represent record numbers up to 16M - 1, so the format of the deferred index update records optimized record numbers to three bytes. However, if using FILEORG X'200' files, an extra byte is required to represent the record numbers for deferred index updates.

This means that when converting a file to FILEORG X'200', you might need to modify reorg jobs to accommodate the longer record numbers. Fortunately, most such changes should work with non-FILEORG X'200' files if common infrastructure is used for files that are a mix of FILEORG X'200' and non-FILEORG X'200'. The rest of this page indicates where values need to be adjusted for FILEORG X'200' files.

Deferred update phases

The two phases of Table updating in a Model 204 file during the deferred update process are:

• Loading Tables A and B and loading the index (Tables C and D) updates into the deferred update data sets
• Loading the deferred index updates into Tables C and D

First phase of deferred update

The first phase of the deferred update process involves executing whatever configuration of Model 204 would be run if updates to the index were not being deferred: BATCH204, ONLINE, IFAM1, or IFAM4. Storing new records or changing existing records proceeds normally until the Model 204 file is put into deferred update mode by a special form of the OPEN command or the IFOPEN call. Thereafter, only Table A, the internal file dictionary, and Table B, the VISIBLE data, of the file are updated.

The index information pertaining to FOR EACH VALUE (FRV), KEY, NUMERIC RANGE, and ORDERED retrieval fields is not added directly to Table C and Table D. Instead, it is written to one or two deferred update data sets for processing in later job steps.

Note: The index update generated by a File Records Under statement is never deferred.

Query results between deferred update phases

For example, suppose that you add the field name = value pair NAME=MURRAY to an existing record in file PEOPLE. PEOPLE is in deferred update mode, and the NAME field is KEY and VISIBLE.

If the record is retrieved by some other (preexisting) KEY field, a Print NAME statement prints MURRAY, because the field is VISIBLE and is, therefore, stored in Table B. However, the statement FIND ALL RECORDS FOR WHICH NAME = MURRAY does not find the record, because the index information NAME=MURRAY has not yet been added to Tables C and D.

Second phase of deferred update

The second phase of the deferred update process is executed only after the first phase of deferred update process is complete. The second phase of the deferred update process sorts each of the deferred update data sets created in the first phase, then applies the updates to Tables C and D.

Applying the sorted updates from the deferred update data sets adds the information necessary to perform retrievals based on the new KEY, NUMERIC RANGE, and ORDERED retrieval field values. For instance, after applying the deferred updates, the retrieval statement FIND ALL RECORDS FOR WHICH NAME = MURRAY finds the record in the previous example.

Additional job steps in a deferred update

If the file contains FRV (FOR EACH VALUE) fields, applying the deferred index updates to Tables C and D generates an additional deferred update data set. This additional deferred update data set contains more deferred index updates for fields with the FRV attribute, which must also be sorted, and the sorted deferred updates applied to Tables C and D.

For example, suppose that the NAME field described previously also has the FRV attribute and that MURRAY is a new value of NAME that has never occurred in any record. The statement For Each Value of NAME does not find the value MURRAY until certain information is added to Tables C and D. This information is contained in the additional deferred update data set.

The procedure for deferring updates to the index normally requires multiple job steps instead of the one job step required if the updates are not deferred. The deferred update feature can use three, five, or seven job steps, depending on the attributes of the fields being deferred and the number of deferred update data sets specified on the OPEN command or the IFOPEN call. The specific job steps required for deferred update processing are described in detail in the section Job steps.

Opening deferred update files

Types of deferred update data sets

There are two forms of deferred update data sets. The OPEN command that places the file in deferred update mode and the attributes of the fields being updated determine whether a fixed-length data set, a variable-length data set, or both are used for the deferred updates.

Deferred update OPEN syntax

The OPEN command and the IFOPEN Host Language Interface function have special forms that place the file being opened into deferred update mode. In these forms, you can specify one or two deferred update data sets.

The OPEN command syntax in this case is:

OPEN [FILE] filename,{[dsname1][,dsname2]}

Where:

• filename is the name of the Model 204 file to be opened.
• dsname1 is the name of the fixed-length deferred update data set. If dsname1 is supplied in the OPEN command, deferred updates to KEY and NUMERIC RANGE fields are written to this data set.
• dsname2 is the name of the variable-length deferred update data set. If dsname2 is supplied in the OPEN command, deferred updates to ORDERED fields are written to this data set. If only dsname2 is supplied in the OPEN command, all index updates are written to this data set. If dsname2 is not supplied in the OPEN command, updates to ORDERED fields are not deferred.

Naming conventions

A deferred update data set must have a name starting with the letters TAPE. It can be placed on magnetic tape or a direct access device.

For example:

OPEN PEOPLE,TAPEPEOP

To which Model 204 responds:

*** M204.0656: UPDATES DEFERRED TO TAPEPEOP

Usage

Use the special form of OPEN only once per main step for a particular file. Subsequent ordinary OPEN commands treat the file as though the already declared deferred update data set(s) were specified.

The maximum number of deferred update files you can have open in a run is 192. If you try to open any more than 192 files, the OPEN fails and Model 204 displays the following message:

*** M204.1529: MAXIMUM DEFERRED UPDATE FILES ALREADY OPEN

Space requirements

The space requirements for the deferred update data sets depend on which type of data set is used and the number of KEY, NUMERIC RANGE, or ORDERED retrieval fields that have their index updates deferred during the first phase of the deferred update process.

Example 1: Opening a deferred update file with the first data set specified

In the following example, the CARS file is placed in deferred update mode by using the OPEN command with only the first update data set name, dsname1, specified. The updates to the Ordered Index-the ORDERED fields-are not deferred. The Ordered Index updates are applied during the first phase of the deferred update process:

OPEN CARS,TAPECARF

Example 2: Opening a deferred update file with the second data set specified

In the following example, the CARS file is placed in deferred update mode by using the OPEN command with only the second update data set name, dsname2, specified. Note the two leading commas before the data set name):

OPEN CARS,,TAPECARV

The updates to KEY, NUMERIC RANGE, and ORDERED fields are all deferred to TAPECARV, a variable-length data set.

Example 3: Opening a deferred update file with both data sets specified

Deferring all the updates to the variable-length data set is somewhat less efficient than splitting the updates by providing both deferred update data sets, as in the following example:

OPEN CARS,TAPECARF,TAPECARV

The updates to KEY and NUMERIC RANGE fields are written to the fixed-length data set, TAPECARF, and the updates to ORDERED fields are written to the variable-length data set, TAPECARV.

Active-defer in deferred update files

When you open a file in deferred update mode, Model 204 places the file in active-defer mode, and the file stays in this mode for the life of the run. The file stays in this mode until the first Z step is successfully completed.

If the file is in active-defer mode, Model 204 displays the following error message whenever the file is opened:

*** M204.0624: FILE IS IN DEFERRED UPDATE MODE

However, after the first successful Z step has been run, this message is turned off, whether or not there are still updates to be completed.

Deferred updates: Phase 1

The system parameter DUFILES gives the number of files that have been opened in deferred update mode in the current run.)

After the file is placed in deferred update mode, updates to the file that normally update the index structures are deferred to the deferred update data set(s). These updates can be the result of User Language requests, Host Language Interface functions, or file updating commands.

Number of deferred update data sets required

You must specify at least one deferred update data set in the JCL or FILEDEFs for each file in deferred update mode. If updates are deferred for more than one file in the first phase step, a separate sequence of the remaining second phase steps must be run for each file opened in deferred update mode. (See the seven-, five-, and three-step process descriptions Job steps.)

Fixed-length deferred update data sets

The fixed-length data set name corresponds to dsname1 in the OPEN command described in Deferred update OPEN syntax.

DCB parameters for fixed-length data sets

The default DCB parameters for the fixed-length data sets are:

RECFM=FB . LRECL=24 BLKSIZE=6000 .

For fixed-length deferred update CMS files, the FILEDEF statement must specify BLOCK 24 for files stored on disk. This results in a blocking factor of one, and it prevents errors from occurring during checkpointing.

For FILEORG X'200' files, the LRECL should be 25 and, in fact, that is the default. The default BLKSIZE works for both FILEORG X'200' and non-FILEORG X'200' files, so it is used for both. Therefore, to eliminate worries about jobs working on files with or without FILEORG X'200', the best approach is to not specify LRECL and BLKSIZE, allowing them to default.

Deferred KEY and NUMERIC RANGE field space requirements

When the updates to KEY and NUMERIC RANGE fields are deferred to the fixed-length data set, the space requirements for the fixed-length deferred update data set are as follows:

• For each occurrence of a KEY field that is added or deleted, one 24-byte record is written to the data set. A changed field counts twice; the old value is deleted and the new one is added.
• For each occurrence of a NUMERIC RANGE retrieval field that is added or deleted, the average number of 24-byte records written is equal to:

  4 + (4.5 * the number of digits in the occurrence)

Variable-length deferred update data sets

The variable-length data set name corresponds to dsname2 discussed in Deferred update OPEN syntax.

DCB parameters for variable-length data sets

The default DCB parameters for the variable-length data sets for non-FILEORG X'200' files are:

RECFM=VB . LRECL=270 BLKSIZE=6000 .

For FILEORG X'200' files, the parameters are:

RECFM=VB . LRECL=271 BLKSIZE=6000 .

For z/VSE, set the BLKSIZE parameter with the DEFINE DATASET command.

You can override the BLKSIZE on the data set or file definition statement. If enough space is available, set BLKSIZE to a full track for most efficient processing. See the description of SPCORE on Setting the SPCORE parameter.

The length of the variable deferred index update entries is 270 for non-FILEORG X'200' files, and it is 271 for FILEORG X'200' files. If using a mix of FILEORG X'200' and non-FILEORG X'200' files, the best strategy is either of these:

• Do not explicitly specify LRECL, letting it default.
• Specify LRECL 271, which will work for both FILEORG X'200' and non-FILEORG X'200' files.

Deferring updates to ORDERED fields

When the updates to ORDERED fields are deferred to the variable-length data set, the space requirements for the variable-length deferred update data set are one record for each occurrence of an ORDERED field that is added or deleted of a length equal to the following is written to the data set:

15 + the length of the ORDERED field value being added or deleted

Note: The length of an ORDERED NUMERIC deferred numeric value is always 8. The numeric values of an ORDERED NUMERIC field are always stored as FLOAT 8 numbers in the Ordered Index, even if the numbers are stored as FLOAT 16 in Table B. The length of an ORDERED NUMERIC deferred non-numeric value is the length of the value.

A changed field writes two records: one record to delete the old value, and one record to add the new value.

Deferring updates to KEY and NUMERIC fields

When the updates to KEY and NUMERIC RANGE fields, as well as ORDERED fields, are deferred to the variable-length data set, the additional space required in the variable-length deferred update data set is as follows:

For each occurrence of...	That is added or deleted...
KEY field	One 28-byte (non-FILEORG X'200') or 29-byte (FILEORG X'200') record is written to the data set. A changed field writes two records: one record to delete the old value and one record to add the new value.
NUMERIC RANGE field	Average number of 28-byte or 29-byte records written is equal to: 4 + (4.5 * the number of digits in the occurrence)

Setting the SPCORE parameter

SPCORE determines how much spare core or unallocated main memory is available after the Model 204 program has finished initializing itself. When a Model 204 file is first placed into deferred update mode, there must be enough spare core to open the Model 204 file and to accommodate the output buffers for each deferred update data set. Each data set requires a spare core of BUFNO*BLKSIZE bytes.

If more than one deferred update data set is used and/or more than one file goes into deferred update mode, spare core must be large enough to accommodate all the output buffers and associated structures.

• Under z/OS, the parameter SPCORE is normally set on User 0's parameter line — the first input line, containing parameter assignments that further define system options and work area sizes.
• Under z/VSE, the SIZE parameter is set.
• Under z/VM, SPCORE is usually set in the CCAIN input. See User 0 parameters| for further explanation.

Phase 1 sample JCL

This section presents z/OS and z/VSE JCL examples for the first phase of the deferred update process. The section that include a single z/VM example begins in z/VM deferred update example.

The following sequence is an example of the first phase in a deferred update procedure. The file PEOPLE is provided with one fixed-length deferred update data set. The file CARS is provided with two deferred update data sets: one fixed-length deferred update data set and one variable-length deferred update data set.

z/OS example, Phase 1

//JOB UPDATE FILES CARS AND PEOPLE // EXEC ... Include here the basic JCL or cataloged procedure invocation provided by the Model 204 system manager to run BATCH204. //PEOPLE DD DSN=M204.FILE.PEOPLE,DISP=SHR //TAPEPEOP DD DSN=M204.DEFER.PEOPLE,DISP=(NEW,CATLG),UNIT=TAPE //CARS DD DSN=M204.FILE.CARS,DISP=SHR //TAPECARF DD DSN=M204.DEFER.FIXED.CARS,DISP=(NEW,CATLG),UNIT=TAPE //TAPECARV DD DSN=M204.DEFER.VAR.CARS,DISP=(NEW,CATLG),UNIT=TAPE //CCAIN DD * user zero parameters LOGIN userid password OPEN PEOPLE,TAPEPEOP INCLUDE procedure Updating for Tables A and B; KEY . and NUMERIC RANGE deferred updates . written to TAPEPEOP. Any Ordered . Index updates are not deferred. OPEN CARS,TAPECARF,TAPECARV INCLUDE procedure Updating for Tables A and B; KEY . and NUMERIC RANGE deferred updates . written to TAPECARF. Ordered . Index updates written to TAPECARV. CLOSE ALL EOJ /*

z/VSE example, Phase 1

// JOB UPDATE FILES CARS AND PEOPLE // DLBL PEOPLE,'M204.FILE.PEOPLE',,DA // EXTENT SYSnnn,volser // TLBL SYS015,'M204.DEFER.PEOPLE' // ASSGN SYS015,TAPE // DLBL CARS,'M204.FILE.CARS' // EXTENT SYSnnn,volser // TLBL SYS016,'M204.DEFERF.CARS' // ASSGN SYS016,TAPE // TLBL SYS017,'M204.DEFERV.CARS' // ASSGN SYS017,TAPE // EXEC BATCH204,SIZE=AUTO user zero parameters DEFINE DATASET TAPEPEOP WITH SCOPE=SYSTEM DDNAME=SYS015 - RECFM=FB LRECL=24 BLKSIZE=6000 DEFINE DATASET TAPECARF LIKE TAPEPEOP WITH DDNAME=SYS016 DEFINE DATASET TAPECARV WITH SCOPE=SYSTEM DDNAME=SYS017 - RECFM=VB LRECL=270 BLKSIZE=6000 . . . LOGIN userid password OPEN PEOPLE,TAPEPEOP INCLUDE procedure Updating for Tables A and B; . KEY and NUMERIC RANGE deferred updates . written to TAPEPEOP. Any Ordered . Index updates are not deferred. OPEN CARS,TAPECARF,TAPECARV INCLUDE procedure Updating for Tables A and B; KEY . and NUMERIC RANGE deferred updates . written to TAPECARF. Ordered . Index updates written to TAPECARV. CLOSE ALL EOJ /* /&

Deferred updates: Phase 2

As many as three data sets might require sorting and application to Tables C and D:

• Fixed-length data set
• Variable-length data set
• FRV (fixed-length) data set

Descriptions of processing each of these data sets follow.

Sorting the fixed-length deferred update data set

If you use a fixed-length deferred update data set, it must be sorted by some job step after the first phase step.

• For non-FILEORG X'200' files, the fixed-length records are 24 bytes long and must be sorted in ascending order by the first 16 bytes, using the standard EBCDIC collating sequence.
• For FILEORG X'200' files, the fixed-length records are 25 bytes long and must be sorted in ascending order by the first 17 bytes, using the standard EBCDIC collating sequence.

Any sort utility can be used to perform the sort of the fixed-length deferred update data set under z/OS, z/VSE, or z/VM. The basic JCL for the IBM SORT Utility is described in the following paragraphs. Refer to the appropriate IBM manual for additional information.

Note: The z/VM SORT command is not adequate if large files are involved, because it lacks the required capacity.

SORTIN and SORTOUT data sets

SORTIN is the fixed-length deferred update data set.

Specify the SORTOUT DCB parameters as shown in DCB parameters for fixed-length data sets. In z/OS and z/VM, set DCB subparameters explicitly.

Recommended DCB subparameters

The following DCB subparameters are recommended for non-FILEORG X'200' files:

RECFM=FB LRECL=24 BLKSIZE=27998 (1/2 track on a 3390 disk)

The following are recommended for FILEORG X'200' files:

RECFM=FB LRECL=25 BLKSIZE=27975 (1/2 track on a 3390 disk)

In z/VSE, specify the physical record format for the SORTIN data set using the sort program's RECORD statement.

SORT control statements

The following SYSIN control statement sorts fixed-length deferred update records in z/OS for non-FILEORG X'200' files:

SORT FIELDS=(1,16,CH,A)

The effect of this statement is to sort the records in ascending order by the first 16 bytes.

The following statement sorts fixed-length deferred update records for FILEORG X'200' files:

SORT FIELDS=(1,17,CH,A)

Note: The FILEORG X'200' file SORT FIELD statement also works for non-FILEORG X'200' files.

For z/VSE, the SYSIPT control statement is as follows for non-FILEORG X'200' files:

SORT FIELDS=(1,16,CH,A) RECORD TYPE=F,LENGTH=24 INPFIL BLKSIZE=blocksize of SORTIN OUTFIL BLKSIZE=blocksize of SORTOUT

The statement for FILEORG X'200' files is:

SORT FIELDS=(1,17,CH,A) RECORD TYPE=F,LENGTH=25 INPFIL BLKSIZE=blocksize of SORTIN OUTFIL BLKSIZE=blocksize of SORTOUT

Sort work data sets for the fixed-length deferred update data set

The amount of space needed for the sort work data sets depends on the number of records in the fixed-length deferred update data set. In general, set the total amount of space in all the SORTWKxx data sets to about twice the number of bytes in the fixed-length deferred update data set that you are sorting.

For example, suppose that there are 100,000 records in the fixed-length deferred update data set. This is 2,400,000 bytes or about 4 cylinders (rounding up) of a 3380 disk. If you use three sort work data sets for sorting the fixed-length deferred update data set, space parameters of three cylinders apiece yield a total of nine cylinders, or slightly more than twice the size of the fixed-length deferred update data set.

Sorting the fixed-length deferred update data set

z/OS JCL

The following z/OS JCL sorts the fixed-length deferred update data set:

//SORT EXEC PGM=SORT,TIME=60, // REGION=6M, // PARM='MSG=AP,CORE=150000' //SORTLIB DD DSN=SYS1.SORTLIB,DISP=SHR //SORTMSGS DD SYSOUT=A //SYSOUT DD SYSOUT=A //SORTIN DD DSN=M204.DEFER.FIXED.CARS, // DISP=(OLD,DELETE,KEEP) //SORTOUT DD DSN=M204.SORTOUT.FIXED.CARS, // DISP=(NEW,CATLG,DELETE),UNIT=TAPE, // DCB=(RECFM=FB,LRECL=24,BLKSIZE=32760) //SORTWK01 DD SPACE=(CYL,3,,CONTIG), // UNIT=3390 //SORTWK02 DD SPACE=(CYL,3,,CONTIG), // UNIT=(3390,SEP=SORTWK01) //SORTWK03 DD SPACE=(CYL,3,,CONTIG), // UNIT=(3390,(SEP=SORTWK01,SORTWK02)) //SYSIN DD * SORT FIELDS=(1,16,CH,A) /*

Many installations provide cataloged procedures that contain most of the previous JCL.

z/VSE JCL

Following is an example of z/VSE JCL to sort the fixed-length deferred update data set:

// TLBL SORTIN1,'M204.DEFERF.CARS' // ASSGN SYS002,TAPE // TLBL SORTOUT,'M204.SORTFIX.CARS' // ASSGN SYS001,TAPE sort work statements Specify SORTWK1 . through SORTWKn for . sort work data sets . . // EXEC SORT,SIZE=100K SORT FIELDS=(1,16,CH,A),WORK=n n is number of RECORD TYPE=F,LENGTH=24 sort work data sets INPFIL BLKSIZE=1680 OUTFIL BLKSIZE=1680 /* /&

Applying the fixed-length deferred updates

To apply the fixed-length deferred updates, use the following Z command. The Z command processes the sorted output from the sort of the fixed-length deferred update data set and adds the KEY and NUMERIC RANGE retrieval index information to Tables C and D. If any of the KEY fields updated also have the FRV attribute, the Z command creates an additional deferred update data set, an FRV data set, that is processed in later job steps. For more information about the FRV deferred update data set, see Applying the FRV deferred updates.

Z command

The Z command has no arguments. It does the following:

• Reads records from the data set pointed to by the SORT5 data set or file definition statement
• Updates Tables C and D of the currently open Model 204 file.
• Writes FRV deferred update records to the data set pointed to by the TAPE5 data set or file definition statement.

When Z command processing is complete, the following statistics are printed, which reflect the changes that occurred during processing:

Statistic	Number of...
ADDED	Field name=value pairs added to the hashed index.
DISTINCT	New field name = value pairs added to hashed index fields; or, when processing the FRV deferred update data set, the number of FRV fields updated.
DELETED	Field name = value pairs deleted from the hashed index.
FRV	FRV deferred update records written out to the data set pointed to by the TAPE5 data set or file definition statement.
OI ADDED	Entries added to the Ordered Index.
OI DELETED	Entries deleted from the Ordered Index.

SPCORE parameter

The SPCORE parameter must be set large enough to open the Model 204 file and to accommodate both the input buffer for SORT5 and the output buffer for TAPE5, as well as other associated structures.

Fixed-length data set examples

z/OS JCL

Applying the fixed-length deferred index updates for z/OS follows:

// EXEC ... Include the basic JCL or cataloged procedure call provided by the Model 204 system manager to run BATCH204 //CARS DD DSN=M204.FILE.CARS,DISP=SHR //SORT5 DD DSN=M204.SORTOUT.FIXED.CARS, // DISP=(OLD,DELETE,KEEP) //TAPE5 DD DSN=M204.TAPE5.CARS, // DISP=(NEW,CATLG,DELETE), // UNIT=3390,SPACE=(TRK,(5,2)), // DCB=BLKSIZE=27998 //CCAIN DD * user zero parameters LOGIN userid password OPEN CARS Z CLOSE ALL EOJ /*

z/VSE JCL

Applying the fixed-length deferred index updates for z/VSE follows:

// DLBL CARS,'M204.FILE.CARS',,DA // EXTENT SYSnnn,volser // TLBL SYS015,'M204.SORTFIX.CARS' // ASSGN SYS015,TAPE // DLBL TAPE5,'M204.TAPE5.CARS',0 // EXTENT SYSnnn,volser // EXEC BATCH204,SIZE=AUTO PAGESZ=6184,MAXBUF=20 DEFINE DATASET TAPE5 WITH SCOPE=SYSTEM - RECFM=FB LRECL=24 BLKSIZE=1680 DEFINE DATASET SORT5 WITH SCOPE=SYSTEM - DDNAME=SYS015 RECFM=FB LRECL=24 BLKSIZE=1680 LOGIN userid password OPEN CARS Z CLOSE ALL EOJ /* /&

Sorting the variable-length deferred update data set

Variable-length sort keys

When you use a variable-length deferred update data set, it must be sorted by a job step after the deferred data set first phase. The variable-length records can vary in length from 16 to 270 (or 271, for FILEORG X'200' files), and they must be sorted into ascending order. Unlike fixed-length deferred update records, which contain a fixed-length sort key, variable-length deferred update records for ORDERED fields contain a variable-length sort key.

Because no standard sort program supports variable-length sort keys, two sort exits are provided to transform the variable-length sort keys into fixed-length sort keys. Transforming variable-length sort keys is done in the E15 sort exit. The E15 sort exit expands variable-length ORDERED deferred update records into fixed-length records.

This process produces longer sort records, which can increase the time and resources required to produce sorted output. Because sorting performance is quite sensitive to record length, be sure to set the expanded length as part of setting up SORT control statements.

Sort utilities

Any sort utility that supports sorting variable-length records and the use of the E15 and E35 sort exits can be used to perform the sort. The basic JCL for the IBM SORT Utility is described in the following paragraphs. Refer to the appropriate sort utility manual for additional information.

The z/VM SORT command cannot be used, because it cannot accommodate sort exits.

Required sort JCL and EXEC statements

When writing JCL or z/VM EXECs for sorting, use the following statements:

• SORTIN is the variable-length deferred update data set.
• Specify the SORTOUT DCB parameters as shown in the first phase step on DCB parameters for variable-length data sets. In z/OS and z/VM, set DCB subparameters explicitly.
• Use the following DCB subparameters for non-FILEORG X'200' files:

  RECFM=VB LRECL=270 BLKSIZE=27998 (1/2 track on a 3390 disk)

  Use the following for FILEORG X'200' files:

  RECFM=VB LRECL=271 BLKSIZE=27975 (1/2 track on a 3390 disk)

  Or better still, don't specify the DCB subparameters and let them default.

  In z/VSE, specify the physical record format for the SORTIN data set using the SORT program's RECORD statement.

• Specify CCAPRINT (SYSLST in z/VSE) in the step that sorts the variable-length deferred update data set. CCAPRINT is the ddname, FILEDEF, or DLBL that points to the sort error message output data set.
• Specify CCAIN in the step that sorts the variable-length deferred update data set. CCAIN must contain the sort control statements. SYSIN and CCAIN must point to copies of the same sort control statements.

  In z/VSE, the CCAIN DLBL is not needed when sorting the variable-length deferred update data set.

• CCAEXITS, the ddname of the library containing the sort exits, is used only in the z/OS environment. For information on installing the sort exits, see Link the Model 204 utility load modules.

SORT control statements

The syntax of SORT control statements is sort-package and operating-system specific, the following example of a SYSIN control statement is applicable for z/OS and z/VM:

Syntax for z/OS and z/VM

SORT FIELDS=(5,aa,CH,A) RECORD TYPE=V,LENGTH=(271,vl,271,vl) MODS E15=(OI15xxx,1800),E35=(OI35xxx,1300) END

Where:

• aa is the sort field length, the value of (vl - 5). It represents the length of the entire data portion of the expanded sort record.
• vl is a length parameter. See Setting the variable-length sort parameter (vl). The length of the variable deferred index update entries is 270 for non-FILEORG X'200' files and 271 for FILEORG X'200' files, but 271 can be used for either.
• xxx is OS for z/OS or CMS for z/VM, depending on the operating system.

The exact syntax for the MODS statement is operating-system and sort-package dependent.

Syntax for z/VSE

SORT FIELDS=(5,aa,CH,A) RECORD TYPE=V,LENGTH=(271,vl,271,vl) MODS PH1=(OI15,1800),PH3=(OI35,1300) END

Setting the variable-length sort parameter (vl)

Because deferred update records of unequal length cannot be properly compared, the sort exit adjusts the variable-length deferred update records to a specified length. That length is the variable-length sort parameter, vl.

The vl setting determines the number of records in the deferred update data set that are optimally ordered by the sort for loading ordered fields by the Z command. A record whose length is less than the specified setting for vl is sorted into an optimum loading order. Records whose length is greater than or equal to vl are placed in the sorted deferred update data set but not in an optimum loading order.

Finding the optimum loading order

It is usually best to place all deferred update records in optimum loading order. If ORDERED fields and either KEY or NUMERIC RANGE fields exist on the same deferred update data set, specify vl as:

vl = A or B, whichever is greater.

where:

This value	Equals...
A	Length of the longest Ordered Index deferred update record + 1. Estimate this as the length of the longest ORDERED field + 16 or 17 (non-FILEORG X'200' or FILEORG X'200', respectively) bytes overhead. If you are using a mix of FILEORG X'200' and non-FILEORG X'200' files, you do not have to concern yourself with the type of the file in calculations if you use the 17 as a more conservative value — it will work fine for FILEORG X'200' and non-FILEORG X'200' files.
B	28 or 29 (non-FILEORG X'200' or FILEORG X'200', respectively) bytes for Hashed Index (KEY or NUMERIC RANGE fields) deferred update records.

Measuring the longest record

Determining the exact length of the longest deferred update record requires examining the contents of the variable-length deferred update data sets. This examination is done for you if the File Load utility is being used, as the FILELOAD and FLOD commands print statistics that list the lengths of the deferred update records in the form of a histogram. Record lengths can also be obtained through a user-written program or a utility such as SYNCSORT's HISTOGRAM to read the records in the deferred update data set and determine the longest record.

Setting vl for extreme cases

Specifying vl as described in the previous section results in a sorted deferred update data set that is completely ordered for the most efficient loading by the Z command. In some extreme cases, sorting to completely order all deferred update records places excessive demands on the sorting process because of the way the length is increased for the records to be sorted.

Typically these cases involve very large files with unusual distributions of record lengths in the deferred update data set as shown in Setting vl example. In these cases, a smaller setting of the vl parameter than previously recommended might be desired, even though this results in a less than complete ordering of the data set. The consequent slowing of the Z command processing must be weighed against the benefit to the sort performance.

Note: In all cases, if a KEY or NUMERIC RANGE field is deferred in the variable-length deferred update data set, vl must be greater than 28 for non-FILEORG X'200' files, and greater than 29 for FILEORG X'200' files. Using the minimum allowed values is a bad idea, in any case, and you probably want to use a value significantly greater than the minimum allowed (for example, 40 or more).

Setting vl to optimize performance

Sort performance is sensitive to the effective length (approximately, the product of the number of records and the vl parameter) of the data set to be sorted, falling as the effective length increases. The vl setting that optimizes sort performance minimizes the data set's effective length. Simply lowering vl to minimize effective length might not succeed, however.

Setting vl greater than or equal to the longest deferred update record length causes each update record to be lengthened to this setting. Sort performance is hampered, because the effective length of the sort data set is increased.

Setting vl less than the longest deferred update record length causes records whose length is greater than or equal to vl to be spanned: broken into multiple sort records of length vl by the E15 sort exit. These additional sort records are added to the "number of records" factor of the effective length of the sort data set, and might offset the reduction to the effective length from a lower vl. Furthermore, after a record is spanned, it sorts to the end of the sorted data set, reducing the efficiency of the subsequent index loading by the Z command.

Setting vl example

The goal in setting vl, therefore, is to make vl as small as possible while spanning as few records as possible. For example, suppose that the variable-length deferred update data set containing ORDERED field updates consists of the following distribution of record lengths and records by the histogram produced by the FLOD command:

LONGEST DEFERRED UPDATE LRECL=190 RANGE OF LRECL NUMBER OF RECORDS 256-270 0 240-255 0 224-239 0 208-223 0 192-207 0 176-191 5 160-175 0 144-159 0 128-143 0 112-127 0 96-111 15 80- 95 80 64- 79 0 48- 63 2400000 32- 47 400000 1- 31 0

The size of the longest record is 190, but this is not a very satisfactory choice for the vl parameter. The distribution is such that choosing a lower vl produces a significant decrease in the data set's effective length without adding a significant number of spanned records. The most efficient choice for vl here is 64. A large gain in sort performance is purchased at the expense of the relatively few records in the data set that do not get ordered.

Sort work data sets

The space needed for the sort work data sets depends on:

• Sort program used
• Number of records in the variable-length deferred update data set
• Value of vl.

For example, for IBM's DFSORT the recommended sum of the sizes of all the SORTWKxx data sets is:

2 * (number of records to be sorted) * vl

or:

2 * (effective length of sort data set)

The number of records to be sorted is approximately the number of records in the variable-length deferred update data set plus the number of spanned records.

In the example above, with 2,800,100 records in the variable-length deferred update data set, if vl is set to 64, the effective length of the data set to be sorted is:

(2800000 + (2.∗80) + (2∗15) + (3∗5)).∗ 64 = 123,215,040

This is about 174 cylinders (rounding up) of a 3380 disk. If you use 3 sort work data sets for sorting the variable-length deferred update data set, space parameters of 116 cylinders apiece yield a total of 348 cylinders, two times the effective length of the sort data set.

Installing sort exits

For a detailed discussion of the installation and use of sort exits, consult the sort manuals for the sort program to be used, and also see the M204wiki installation page for your operating system.

Variable-length data set examples

The following examples assume that the sort exit modules or z/VM TEXT files have been given the names OI15xxx and OI35xxx, where xxx is either OS for z/OS or CMS for z/VM, depending on the operating system.

z/OS JCL

The following z/OS JCL sorts the variable-length deferred update data set:

//SORT EXEC PGM=SORT,TIME=60, // REGION=7M, // PARM='MSG=AP,CORE=150000' //SORTLIB DD DSN=SYS1.SORTLIB,DISP=SHR //SORTMSGS DD SYSOUT=A //SYSOUT DD SYSOUT=A //CCAPRINT DD SYSOUT=A //CCAEXITS DD DSN=M204.LINKLIB,DISP=SHR //SORTIN DD DSN=M204.DEFER.VAR.CARS, // DISP=(OLD,DELETE,KEEP) //SORTOUT DD DSN=M204.SORTOUT.VAR.CARS, // DISP=(NEW,CATLG,DELETE),UNIT=TAPE, // DCB=(RECFM=VB,LRECL=270,BLKSIZE=15476) //SORTWK01 DD SPACE=(CYL,3,,CONTIG), // UNIT=3380 //SORTWK02 DD SPACE=(CYL,3,,CONTIG), // UNIT=(3380,SEP=SORTWK01) //SORTWK03 DD SPACE=(CYL,3,,CONTIG), // UNIT=(3380,(SEP=SORTWK01,SORTWK02)) //CCAIN DD * SORT FIELDS=(5,75,CH,A) RECORD TYPE=V,LENGTH=(270,80,270,80) MODS E15=(OI15OS,1800,CCAEXITS,N),E35=(OI35OS,1300,CCAEXITS,N) END /* //SYSIN DD * SORT FIELDS=(5,75,CH,A) RECORD TYPE=V,LENGTH=(270,80,270,80) MODS E15=(OI15OS,1800,CCAEXITS,N),E35=(OI35OS,1300,CCAEXITS,N) END /*

Many installations provide cataloged procedures that contain most of the above JCL. Check the sort package documentation for the definitive forms of the sort control statements necessary at your installation.

z/VSE JCL

The following z/VSE JCL sorts the variable-length deferred update data set:

// TLBL SORTIN1,'M204.DEFERV.CARS' // ASSGN SYS002,TAPE // TLBL SORTOUT,'M204.SORTVAR.CARS' // ASSGN SYS001,TAPE sort work statements Specify SORTWK1 through . SORTWKn for sort work . data sets . // DLBL M204CL,'M204.CORE.IMAGE.LIBRARY' // LIBDEF CL,SEARCH=M204CL // EXEC SORT,SIZE=(AUTO,100K) SORT FIELDS=(5,75,CH,A),WORK=n n is number of sort work data sets RECORD TYPE=V,LENGTH=(270,80,270,80) MODS PH1=(OI15,L2000,E15),PH3=(OI35,L1000,E35) INPFIL BLKSIZE=6000 OUTFIL BLKSIZE=6000 /* /&

Applying the variable-length deferred updates

Apply the variable-length deferred updates, using the Z command. The Z command:

• Processes the sorted output from the sort of the variable-length deferred update data set.
• Adds the Ordered Index information to Table D.
• If any KEY and/or NUMERIC RANGE deferred update records are also in the variable-length deferred update data set, the Z command adds this KEY and/or NUMERIC RANGE retrieval index information to Tables C and D.
• If there are KEY field updates in the variable-length deferred update data set, and any of the KEY fields updated also have the FRV attribute, the Z command creates an additional deferred update data set, an FRV data set, that is processed in later job steps.

For more information on the FRV deferred update data set, see Applying the FRV deferred updates.

z/OS JCL

The JCL to apply the variable-length deferred index updates for z/OS follows:

// EXEC ... Include here the basic JCL or cataloged procedure invocation provided by the Model 204 system manager to run BATCH204. //CARS DD DSN=M204.FILE.CARS,DISP=SHR //SORT5 DD DSN=M204.SORTOUT.VAR.CARS, // DISP=(OLD,DELETE,KEEP) //TAPE5 DD DUMMY,DCB=BLKSIZE=24 //CCAIN DD * user zero parameters LOGIN userid password OPEN CARS Z CLOSE ALL EOJ /*

If both a fixed- and a variable-length data set are supplied for the deferred updates, then TAPE5 is not used when applying the variable-length updates and should point to a dummy data set when applying the variable-length updates.

If a fixed-length deferred update data set exists, the FRV deferred updates are generated when the fixed-length deferred updates are applied.

If only a variable-length data set is supplied for the deferred updates, and any FRV fields are updated during the first phase step, then an FRV deferred update data set must be provided with the TAPE5 ddname so that Model 204 can write out the FRV deferred updates to this data set.

z/VSE JCL

The JCL to apply the variable-length deferred index updates for z/VSE follows:

// DLBL CARS,'M204.FILE.CARS',,DA // EXTENT SYSnnn,volser // TLBL SYS015,'M204.SORTVAR.CARS' // ASSGN SYS015,TAPE // DLBL TAPE5,'M204.TAPE5.DUMMY' // EXTENT SYSnnn,volser // EXEC BATCH204,SIZE=AUTO PAGESZ=6184,MAXBUF=20 DEFINE DATASET TAPE5 WITH SCOPE=SYSTEM - RECFM=FB LRECL=24 BLKSIZE=1680 DEFINE DATASET SORT5 WITH SCOPE=SYSTEM DDNAME=SYS015- RECFM=VB LRECL=270 BLKSIZE=6000 LOGIN userid password OPEN CARS Z CLOSE ALL EOJ /* /&

FRV deferred update data set

The FRV (For Each Value) deferred update data set is very similar to a fixed-length deferred update data set. One 24-byte record is written to the FRV deferred update data set for every new value of an FRV field that is stored in Tables C and D during the Z command processing. Unless the Model 204 file being updated has many FRV fields or an FRV field is also MANY-VALUED, the FRV deferred update data set is fairly small.

Determining the FRV step

The number and type of deferred update data sets specified in the OPEN command or IFOPEN call in the first phase of deferred data set processing determines when the FRV defined data set is generated.

If a fixed-length deferred update data set was specified on the OPEN command or IFOPEN call in the first phase step, and an FRV field was updated, the FRV deferred update data set is generated when the fixed-length deferred updates are applied (in other words, during the execution of the Z command that applies the fixed-length deferred updates).

If no fixed-length deferred update data set was specified in the OPEN command or IFOPEN call, but a variable-length deferred update data set was specified, and an FRV field was updated, the FRV deferred update data set is generated when the variable-length deferred updates are applied-in other words, during the execution of the Z command that applies the variable-length deferred updates.

Creating the FRV deferred update data set

The FRV deferred update data set is always named TAPE5. It can be placed either on magnetic tape or on a direct access device.

DCB parameters

The default parameters are:

RECFM=FB LRECL=24 BLKSIZE=6000

For z/VSE installations, TAPE5 must be allocated. Under z/OS, no additional action regarding TAPE5 need be taken.

You can specify the BLKSIZE subparameter explicitly. A full disk track is recommended. z/VSE users can specify TAPE5's block size using the DEFINE DATASET command.

Sorting the FRV deferred update data set

The FRV deferred update data set is sorted after the step that created the FRV deferred update data set. The FRV deferred update records are 24 bytes long and must be sorted in ascending order by the first 16 bytes using the standard EBCDIC collating sequence.

Determine the number of records in the FRV deferred update data set by checking statistics of the Z command that created the FRV deferred update data set. The FRV statistic records the number of FRV deferred update records written to the FRV deferred update data set.

Any sort utility can be used to perform the sort of the FRV deferred update data set. The basic JCL for the IBM SORT Utility follows. Refer to the appropriate IBM manual for additional information.

SORTIN and SORTOUT

SORTIN is the FRV deferred update data set.

Specify SORTOUT in the same way as in the step in which it was created. In z/OS and z/VM, set DCB subparameters explicitly. The following DCB subparameters are recommended:

RECFM=FB LRECL=24 BLKSIZE=27998 (1/2 track on a 3390 disk)

In z/VSE, the physical record format for the SORTIN data set is specified using the SORT program's RECORD statement.

SORT control statement examples

The following SYSIN control statement example sorts FRV deferred update records in z/OS:

z/OS

SORT FIELDS=(1,16,CH,A),SIZE=En

Where:

n is the value of the FRV statistic taken from the output of the Z command that created the FRV deferred update data set.

z/VSE

For z/VSE, the SYSIPT control statement is as follows:

SORT FIELDS=(1,16,CH,A),SIZE=En RECORD TYPE=F,LENGTH=24 INPFIL BLKSIZE=blocksize of SORTIN OUTFIL BLKSIZE=blocksize of SORTOUT

Sort work data sets for the FRV deferred update data set

The amount of space needed for the sort work data sets for the FRV deferred update data set is calculated in the same way as for the sort work data sets for the fixed-length deferred update data set. See Sort work data sets for the fixed-length deferred update data set.

z/OS JCL

The following z/OS JCL sorts the FRV deferred update data set:

//SORT EXEC PGM=SORT,TIME=60, // REGION=7M, // PARM='MSG=AP,CORE=150000' //SORTLIB DD DSN=SYS1.SORTLIB,DISP=SHR //SORTMSGS DD SYSOUT=A //SYSOUT DD SYSOUT=A //SORTIN DD DSN=M204.TAPE5.CARS, // DISP=(OLD,DELETE,KEEP) //SORTOUT DD DSN=M204.SORTOUT.FRV.CARS, // DISP=(NEW,CATLG,DELETE),UNIT=TAPE, // DCB=(RECFM=FB,LRECL=24,BLKSIZE=15476) //SORTWK01 DD SPACE=(CYL,1), // UNIT=3380 //SORTWK02 DD SPACE=(CYL,1), // UNIT=(3380,SEP=SORTWK01) //SORTWK03 DD SPACE=(CYL,1), // UNIT=(3380,(SEP=SORTWK01,SORTWK02)) //SYSIN DD * SORT FIELDS=(1,16,CH,A),SIZE=E100 /*

Many sites provide cataloged procedures that contain most of the above JCL.

z/VSE JCL

The following z/VSE JCL sorts the FRV deferred update data set:

// TLBL SORTIN1,'M204.TAPE5.CARS' // ASSGN SYS002,TAPE // TLBL SORTOUT,'M204.SORTFRV.CARS' // ASSGN SYS001,TAPE sort work statements Specify SORTWK1 through . SORTWKn for sort work . data sets . // EXEC SORT,SIZE=1M SORT FIELDS=(1,16,CH,A),WORK=n n is number of sort work data sets RECORD TYPE=F,LENGTH=24 INPFIL BLKSIZE=1680 OUTFIL BLKSIZE=1680 /*

Applying the FRV deferred updates

To apply the FRV deferred updates, use the Z command. The Z command:

• Processes the sorted output from the sort of the FRV deferred update data set
• Adds the FRV retrieval index information to Tables C and D.

Applying the FRV deferred updates is much like applying the fixed-length deferred updates, except that:

• SORT5 points to the sorted FRV deferred update data set.
• TAPE5, under z/OS, points to a dummy data set that has DCB=BLKSIZE=24. Under z/VM, TAPE5 is a dummy file; under z/VSE, it must not be a dummy file.

SPCORE

Set the SPCORE parameter large enough to open the Model 204 file and to accommodate both the input buffer for SORT5 and the output buffer for TAPE5 along with other structures. See Setting the SPCORE parameter.

z/OS JCL

The z/OS JCL to apply the FRV deferred updates for z/OS follows:

// EXEC ... Include here the basic JCL or cataloged procedure invocation provided by the Model 204 system manager to run BATCH204. //CARS DD DSN=M204.FILE.CARS,DISP=SHR //SORT5 DD DSN=M204.SORTOUT.FRV.CARS, // DISP=(OLD,DELETE,KEEP) //TAPE5 DD DUMMY,DCB=BLKSIZE=24 TAPE5 points to a dummy data set because no new FRV deferred updates will be created. //CCAIN DD * user zero parameters LOGIN userid password OPEN CARS Z CLOSE ALL EOJ /*

z/VSE JCL

The z/VSE JCL to apply the FRV deferred updates for z/VSE follows:

// DLBL CARS,'M204.FILE.CARS',,DA // EXTENT SYSnnn,volser // TLBL SYS015,'M204.SORTFRV.CARS' // ASSGN SYS015,TAPE // DLBL TAPE5,'M204.TAPE5.DUMMY' // EXTENT SYSnnn,volser TAPE5 is not used because no new FRV deferred updates will be created. // EXEC BATCH204,SIZE=AUTO PAGESZ=6184,MAXBUF=20 DEFINE DATASET TAPE5 WITH SCOPE=SYSTEM - RECFM=FB LRECL=24 BLKSIZE=1680 DEFINE DATASET SORT5 WITH SCOPE=SYSTEM - DDNAME=SYS015 RECFM=FB LRECL=24 BLKSIZE=1680 LOGIN userid password OPEN CARS Z CLOSE ALL EOJ /* /&

z/VM deferred update example

Using deferred updates with FASTLOAD to load files

The most common use of the deferred update feature is to load files. The z/VM FASTLOAD driver EXEC, distributed with the Model 204 installation, incorporates the deferred update procedure into the file loading process. The FASTLOAD EXEC can initiate a multistep deferred update procedure that is as many job-steps (seven, five, three, or one) as required.

The FASTLOAD EXEC accepts as parameters the name of another EXEC, such as DEFUPD in the following example, and the type (number of deferred update procedure job steps) of FASTLOAD to execute. DEFUPD EXEC is used by the driver to define the files to be used for each FASTLOAD step and to invoke Model 204 when necessary.

FASTLOAD EXEC format

The format of the FASTLOAD command is:

Syntax

FASTLOAD [TEST] [NODCSS] EXECname [NUM] [(option[)]]

If no option is specified, 5STEP is the default.

where:

• TEST specifies a test version of the Model 204 ONLINE module or shared segment-that is, T204. If TEST is not specified, the production version-that is, M204 is invoked.
• NODCSS specifies that shared segments are not to be used, even though they exist.
• EXECname specifies the name of an EXEC procedure that provides the file definitions and other logic required to prepare a deferred update or File Load utility job. The EXEC procedure is invoked once before each step of the deferred update process, and during FASTLOAD command initialization.
• NUM specifies the number, from 1 through 7, of the step to begin with in case of a restart.
• option is one of the following:

  Option	Indicates...
  7STEP	Seven-step job
  5STEP	Five-step job
  3STEP	Three-step job
  1STEP	One-step job

Setting parameters in the FASTLOAD EXEC

Parameters required to invoke Model 204 are set in the EXEC procedure called by the FASTLOAD EXEC. These parameters must be placed in the stack using the &STACK command as keyword-value pairs, separated by blanks, during Steps 1, 3, 5, and 7. The only operand provided to the EXEC called by the FASTLOAD EXEC is the step number.

FASTLOAD EXEC example

In the following example, the FASTLOAD EXEC calls DEFUPD and passes it the parameter for the step number to be executed. The DEFUPD EXEC concatenates the passed parameter (&1, for example) with the step name (FLOAD) in a &GOTO statement. This numbering and consequent program flow simulate the job steps in the previous z/OS and z/VSE examples.

z/VM EXEC

Enter the following command to run the example:

FASTLOAD DEFUPD (7STEP

The DEFUPD EXEC appears below, followed by a step-by-step explanation of the EXEC and the files defined in each step.

&CONTROL OFF &ERROR &EXIT &RETCODE &IF &1 EQ 0 &GOTO -ACCESS FILEDEF * CLEAR &GOTO -FLOAD&1 -ACCESS RELEASE M ACCESS 201 M &EXIT -FLOAD1 FILEDEF TAPECARF TAP1 SL ( BLOCK 2400 LRECL 24 RECFM FB LABELDEF TAPECARF VOLID 123456 FILEDEF TAPECARV DISK CARS DEFV W (BLOCK 6000 LRECL 270 RECFM VB FILEDEF CARS M DSN M204DB CARS FILEDEF CCAIN DISK DEFUPD CCAIN1 * &GOTO -COMMON -FLOAD2 LABELDEF SORTIN1 VOLID 123456 SSORT TAP1 CARS DEFSORT W DEFUPDF SORTCTRL B &EXIT &RC -FLOAD3 FILEDEF SORT5 DISK CARS DEFSORT W FILEDEF TAPE5 DISK CARS FRV W ( BLOCK 24 LRECL 24 RECFM FB FILEDEF CCAIN DISK DEFUPD CCAIN3 * FILEDEF CARS M DSN M204DB CARS &GOTO -COMMON -FLOAD4 FILEDEF CCAIN DISK DEFUPDV SORTCTRL B FILEDEF CCAPRINT DISK DEFUPD&1 CCAPRINT A SSORT CARS DEFV W CARS SORTVAR W DEFUPDV SORTCTRL B &EXIT &RC -FLOAD5 FILEDEF SORT5 DISK CARS SORTVAR W FILEDEF TAPE5 DUMMY ( BLOCK 24 LRECL 24 RECFM FB) FILEDEF CCAIN DISK DEFUPD CCAIN5 * FILEDEF CARS M DSN M204DB CARS &GOTO -COMMON -FLOAD6 SSORT CARS FRV W CARS FRVSORT W FASTLOAD SORTCTRL C &EXIT &RC -FLOAD7 FILEDEF SORT5 DISK CARS FRVSORT W FILEDEF TAPE5 DUMMY ( BLOCK 24 LRECL 24 RECFM FB FILEDEF CCAIN DISK DEFUPD CCAIN7 * FILEDEF CARS M DSN M204DB CARS &GOTO -COMMON &EXIT -COMMON FILEDEF CCAPRINT DISK DEFUPD&1 CCAPRINT A FILEDEF CCAAUDIT DISK DEFUPD&1 CCAAUDIT A FILEDEF CCATEMP N DSN WORK CCATEMP FILEDEF CCASTAT N DSN WORK CCASTAT FILEDEF CCASNAP PRINTER &STACK SYSOPT 128 &EXIT

Description of steps

The task performed and the files defined in each step are as follows.

Step 1

Update database file CARS in deferred update mode. The files defined are:

TAPECARF	Fixed-length deferred update data set, defined to go to a standard label tape. A LABELDEF command defines the volume serial number. When Model 204 uses a tape, it executes the M204MOUN EXEC to tell an operator to mount the volume label specified in the LABELDEF. The M204UNLD EXEC is executed at the end of tape processing to issue the appropriate z/VM TAPE command function. The version of M204UNLD distributed has a TAPE RUN command. Change this to TAPE REW to run this example.
TAPECARV	Variable-length deferred update data set.
CARS	Model 204 database file being updated. The file is defined on a z/OS minidisk.
CCAIN	z/VM disk file, DEFUPD CCAIN1, that contains the commands to open and update the file CARS. A file listing follows the EXEC.

Step 2

Sort the fixed-length deferred update file that has been written in Step 1. This example uses SYNCSORT, which is able to read tape input. A LABELDEF is needed to define the volume serial number.

The SYSIN file to the sort is DEFUPDF SORTCTRL. See the DEFUPD EXEC for the control statement format.

Step 3

Execute the first Z command to apply the fixed-length deferred updates. The files defined are:

SORT5	Sorted fixed-length deferred update data set.
TAPE5	FRV deferred update data set.
CCAIN	z/VM disk file, DEFUPD CCAIN3,containing the commands to open and update the file CARS. A file listing follows the EXEC.
CARS	Model 204 database file being updated. The file is defined on a z/OS minidisk.

Step 4

Sort the variable-length deferred update data set. This example uses SYNCSORT and the sort exits OI15CMS and OI35CMS. These sort exits are provided with the installation. Typically, the sort exits are either placed in a TXTLIB or reside on an accessed disk with a filetype of TEXT.

The SYSIN file to the sort is DEFUPDV SORTCTRL. See the DEFUPD EXEC for the control statement format.:

CCAIN	Sort control statements (same as SYSIN).
CCAPRINT	Sort error message file for the variable-length deferred update data set sort.

Step 5

Execute the second Z command to apply the variable-length deferred updates:

SORT5	Sorted data set of variable-length deferred update entries.
TAPE5	Dummy file. If both a fixed-length deferred update data set and a variable-length deferred update data set are used, the FRV deferred update data set is generated during the processing of the fixed-length deferred update data set (Step 3).
CCAIN	z/VM disk file, DEFUPD CCAIN5, containing the commands to open and update the file CARS. A file listing follows the EXEC.
CARS	Model 204 database file being updated. The file is defined on a z/OS minidisk.

Step 6

Sort the FRV index file that has been written in Step 3. This example uses SYNCSORT.

The SYSIN file to the sort is FASTLOAD SORTCTRL. This file is provided with the installation. See the DEFUPD EXEC for the control statement format.

Step 7

Apply the FRV deferred updates using the Z command:

SORT5	Sorted data set of FRV deferred updates.
TAPE5	Dummy file.
CCAIN	z/VM disk file, DEFUPD CCAIN7, that contains the commands to open and update the file CARS. A file listing follows the EXEC.
CARS	Model 204 database file being updated. The file is defined on a z/OS minidisk.

Step COMMON

Define files common to each step.

COMMON is executed with Steps 1, 3, 5, and 7. The files defined are:

CCAAUDIT	Step number is concatenated to the filename to have a separate file for each step.
CCAPRINT	Step number is concatenated to the filename to have a separate file for each step.
CCASTAT	Previously defined file on a z/OS minidisk.
CCATEMP	Previously defined file on a z/OS minidisk.
CCASNAP	Set to PRINTER to save disk space.
&STACK	Passes execution parameter SYSOPT 128 in the program stack to FASTLOAD.

DEFUPD EXEC

The DEFUPD EXEC input files are:

DEFUPD CCAIN1 SPCORE=50000 OPEN CARS,TAPECARF,TAPECARV I STORE CLOSE CARS EOJ DEFUPD CCAIN3 SPCORE=30000 OPEN CARS Z EOJ DEFUPD CCAIN5 SPCORE=30000 OPEN CARS Z EOJ DEFUPD CCAIN7 SPCORE=27000 OPEN CARS Z EOJ DEFUPDF SORTCTRL SORT FIELDS=(1,16,CH,A),SIZE=500 INPFIL BLKSIZE=2400 RECORD TYPE=F,LENGTH=(24,24) OPTION PRINT=ALL END DEFUPDV SORTCTRL SORT FIELDS=(5,75,CH,A) RECORD TYPE=V,LENGTH=(270,80,270,80) MODS E15 = (OI15CMS,1500),E35=(OI35CMS,800) END FASTLOAD SORTCTRL SORT FIELDS=(1,16,CH,A)

Job steps

The first phase of the deferred update process is performed in one job step. The second phase can involve many separate job steps: two job steps for each deferred update data set generated in the first phase and two job steps for the FRV deferred update data set, if one is generated.

Two basic job steps

The two job steps for processing each deferred update data set are:

• First job step executes a sort program, such as IBM SORT, to sort the deferred update data set for efficient processing in the next job step.
• Second job step executes the BATCH204 load module to process the sorted deferred index information and to update Tables C and D.

Total number of job steps

The number of job steps in the entire deferred update process depends on the attributes of the fields being updated and the number of deferred index update data sets specified on the OPEN command or IFOPEN call.

The deferred update process can have three, five, or seven steps. Each type of deferred update process is explained in detail in the following sections.

Determining deferred update process steps is a quick reference chart for determining the number of job steps needed, including the first phase job step, for each file placed in deferred update mode.

Determining deferred update process steps

File attributes for all deferred fields in file	No. of deferred update data sets in OPEN command	No. of sort files required	No. of steps
KEY	1	1	3
KEY and FRV	1	2	5
KEY and NUMERIC RANGE	1	1	3
KEY and ORDERED	1 2	1 2	3 5
KEY, FRV, and NUMERIC RANGE	1	2	5
KEY, FRV, and ORDERED	1 2	2 3	5 7
KEY, FRV, NUMERIC RANGE, and ORDERED	1 2	2 3	5 7
KEY, NUMERIC RANGE, and ORDERED	1 2	1 2	3 5
NUMERIC RANGE	1	1	3
NUMERIC RANGE and ORDERED	1 2	1 2	3 5
ORDERED	1	1	3

Seven-step deferred update process

A seven-step deferred update process is needed when the following conditions occur:

Condition	Description
Deferred update mode	File was put in deferred update mode by specifying two deferred update data sets on the OPEN command or IFOPEN call: One fixed-length deferred update data set One variable-length deferred update data set
Hashed index fields	Fields with the KEY and/or NUMERIC RANGE attribute are updated and the index updates are deferred to the fixed-length data set.
Ordered Index fields	ORDERED CHARACTER and/or ORDERED NUMERIC are updated and the index updates are deferred to the variable-length data set.
Key fields	One or more of the KEY fields updated and deferred also has the FRV attribute.

Seven-step job steps

Follow these steps in the seven-step deferred update process:

1. The first phase step, generate two deferred update data sets: one fixed-length and one variable-length.
2. Sort the variable-length deferred update data set. (It is not necessary to sort the variable-length data set before the fixed-length data set. Steps 4 and 5 can be switched with steps 2 and 3.)
3. Apply the updates from the sorted variable-length deferred update data set.
4. Sort the fixed-length deferred update data set
5. Apply the updates from the sorted fixed-length deferred update data set, which generates an additional fixed-length deferred update data set (for FRV fields).
6. Sort the additional fixed-length deferred update data set.
7. Apply the updates for the additional sorted fixed-length deferred update data set.

Five-step deferred update process

Three combinations of field attributes and specified deferred update data sets require five job steps to complete the deferred update process:

This type...	Does...
Fixed-length data set with FRV field	Fixed-length deferred update processing for KEY and/or NUMERIC RANGE and FRV fields.
Fixed-length and Variable-length data set	Fixed- and variable-length deferred update processing for KEY and/or NUMERIC RANGE and ORDERED fields.
Variable-length data set and FRV field	Variable-length deferred update processing for KEY, NUMERIC RANGE, and ORDERED fields, and fixed-length deferred update processing for FRV fields.

When to use Fixed/FRV

The Fixed/FRV type of five-step process is needed when the following conditions occur:

• File was put in deferred update mode by specifying one fixed-length deferred update data set on the OPEN command or IFOPEN call.
• Hashed index fields (KEY and/or NUMERIC RANGE) are updated and the index updates are deferred to the fixed-length data set.
• One or more of the KEY fields updated and deferred also has the FRV attribute.

In this case, either no ORDERED fields are updated or the variable-length deferred update data set is intentionally omitted. This omission forces updates to the Ordered Index (ORDERED fields) to be updated immediately, and no Ordered Index updates are deferred.

Fixed/FRV five-step process

1. The first-phase step, generate one fixed-length deferred update data set.
2. Sort the fixed-length deferred update data set.
3. Apply the updates from the sorted fixed-length deferred update data set, which generates an additional fixed-length deferred update data set for FRV fields.
4. Sort the additional fixed-length deferred update data set.
5. Apply the updates for the additional sorted fixed-length deferred update data set.

When to use Fixed/Variable

The Fixed/Variable type of five-step process is needed when the following conditions occur:

• File was put in deferred update mode by specifying one fixed-length and one variable-length deferred update data sets on the OPEN command or IFOPEN call.
• Hashed index fields-KEY and/or NUMERIC RANGE-are updated and the index updates are deferred to the fixed-length data set.
• Ordered Index fields-ORDERED CHARACTER and/or ORDERED NUMERIC-are updated and the index updates are deferred to the variable-length data set.
• None of the KEY fields updated and deferred has the FRV attribute.

Fixed/Variable five-step process

1. The first-phase step, generates one fixed-length and one variable-length deferred update data set.
2. Sort the variable-length deferred update data set. It is not necessary to sort the variable-length data set before the fixed-length data set. Steps 4 and 5 can be switched with steps 2 and 3.
3. Apply the updates from the sorted variable-length deferred update data set.
4. Sort the fixed-length deferred update data set.
5. Apply the updates from the sorted fixed-length deferred update data set. This does not generate an additional fixed-length deferred update data set, because there are no FRV fields involved.

When to use Variable/FRV

The Variable/FRV type of five-step deferred update process is used when the following conditions occur:

• File was put in deferred update mode by specifying one variable-length deferred update data set on the OPEN command or IFOPEN call.
• Hashed index fields-KEY and/or NUMERIC RANGE) and/or Ordered Index fields (ORDERED CHARACTER and/or ORDERED NUMERIC-are updated and the index updates are deferred to the variable-length data set.
• One or more of the KEY fields updated and deferred also has the FRV attribute.

In other words, no fixed-length deferred update data set is supplied, so all the deferred updates generated by the first phase are sent to the variable-length data set.

Variable/FRV five-step process

1. The first-phase step, generate one variable-length deferred update data set.
2. Sort the variable-length deferred update data set.
3. Apply the updates from the sorted variable-length deferred update data set, which generates an additional fixed-length deferred update data set (for FRV fields).
4. Sort the additional fixed-length deferred update data set.
5. Apply the updates for the additional sorted fixed-length deferred update data set.

Three-step deferred update process

The two types of three-step deferred update processes are:

• Fixed type
• Variable type

When to use the Fixed three-step process

The Fixed type of three-step process is used when the following conditions occur:

• File was put in deferred update mode by specifying one fixed-length deferred update data set on the OPEN command or IFOPEN call.
• Hashed index fields-KEY and/or NUMERIC RANGE-are updated and the index updates are deferred to the fixed-length data set.
• None of the KEY fields updated and deferred has the FRV attribute.

Either no ORDERED fields are updated or the variable-length deferred update data set is intentionally omitted. This omission forces immediate updates to the Ordered Index (ORDERED fields) and no Ordered Index updates are deferred.

Fixed three-step process

1. The first-phase step, generate one fixed-length deferred update data set.
2. Sort the fixed-length deferred update data set.
3. Apply the updates from the sorted fixed-length deferred update data set. (This does not generate an additional fixed-length deferred update data set, because there are no FRV fields involved.)

When to use the Variable three-step process

The Variable type of three-step deferred update process is used when the following conditions occur:

• File was put in deferred update mode by specifying one variable-length deferred update data set on the OPEN command or IFOPEN call.
• Hashed index fields-KEY and/or NUMERIC RANGE-and/or Ordered Index fields-ORDERED CHARACTER and/or ORDERED NUMERIC-are updated and the index updates are deferred to the variable-length data set.
• None of the KEY fields updated and deferred has the FRV attribute.

In other words, no fixed-length deferred update data set is supplied, so all the deferred updates generated by the first phase are sent to the variable-length data set.

Variable three-step process

In this case the following steps are necessary in the deferred update process:

1. First-phase step, generate one variable-length deferred update data set.
2. Sort the variable-length deferred update data set.
3. Apply the updates from the sorted variable-length deferred update data set.

Merging deferred updates

It is possible to merge the fixed-length deferred update data sets of many first-phase steps into a single sorted data set. This can be useful when many jobs are used to update one file, and each job produces its own fixed-length deferred update data set. The following restrictions apply to merging deferred update data sets:

• No KEY or NUMERIC RANGE deferred update records can be deferred to the variable-length deferred update data set.
• No variable-length deferred update data sets can be merged.

The merged fixed-length deferred update sort output is used as input for a single index step. A single index step can handle a maximum of 2,147,483,648 (or 2³¹) deferred update records.

The variable-length deferred update data set cannot be merged. If the fixed-length deferred update data sets produced in the same first-phase steps as the variable-length data sets are merged, the variable-length deferred update data sets must still be processed separately and in chronological order.

Although merging several deferred update data sets can produce a cost reduction in the next step, other considerations, such as error recovery, can make merging undesirable.

Overriding deferred updates

Consider the following type of update situation: new records are to be added to an automobile file in which each record has a unique SERIAL NUMBER field, which is KEY. Before a record is added, the user issues a Find statement to see if any other record already exists with the same value of SERIAL NUMBER. If one is found, the new record is not added.

If updates to the index are deferred, the Find fails to locate duplicate records that already have been added in the same main step. In order to avoid this situation, define the field SERIAL NUMBER with the NON-DEFERRABLE attribute.

Deferring updates in single-user runs

In single-user runs, however, there is usually no reason not to defer updates. Consequently, in single-user runs, if a file is in deferred update mode, the NON-DEFERRABLE attribute is ignored by Model 204. If you must prevent updates to a field from being deferred in a single user environment, use the following technique to simulate a multiuser environment and thereby force the NON-DEFERRABLE fields to be updated immediately:

• On User 0's parameter line in the main job step, set NUSERS to 2.
• Follow User 0's parameter line with a second parameter line that contains IODEV=3, INPUT=T03U001I, and OUTPUT=T03U001O.
• Include these three statements in the JCL for the main step:

  //T03U001I DD * /* //T03U001O DD SYSOUT=A

For z/VSE, specify the IODEV=3 input and output file names using z/VSE system logical names that are assigned to "IGN" in the JCL.

Handling a deferred update override

If you override the deferred update feature, you must manually turn off the file load optimization feature. To turn this feature off, specify a read-and-load-a-field statement with the delete mode bit, and then include a branch before the statement so that it never executes.

For example:

DEFINE FIELD DUMMY Define a dummy field before . the FILELOAD statement. . . =5 Branch to 5 DUMMY=1,0.X'4000' Turn on delete mode bit #5 Always skip DUMMY statement. . . .

Deferred updates for NON-DEFERRABLE fields

There are times in Model 204 in which updates are deferred for NON-DEFERRABLE fields. Some NON-DEFERRABLE fields are deferred if the file is opened in deferred update mode and NUSERS=1 (that is, the file is opened in single user mode).

The following table describes the conditions under which the updates are deferred for NON-DEFERRABLE fields.

Deferring NON-DEFERRABLE fields

Condition	Fixed-length data set only	Variable-length* data set only	Fixed- and variable-length* data sets
UNIQUE through FLOD	N/A	KEY and ORD deferred	KEY deferred to Fixed ORD deferred to Variable
UNIQUE without FLOD	No updates deferred	No updates deferred	No updates deferred
NON-UNIQUE through FLOD	KEY and NR deferred	KEY, NR, and ORD deferred	KEY, NR deferred to Fixed ORD deferred to Variable
NON-UNIQUE without FLOD	KEY and NR deferred	KEY, NR, and ORD deferred	KEY, NR deferred to Fixed ORD deferred to Variable
*A variable-length data set is required for ORDERED fields loaded through FLOD.

Note: If new fieldname=value pairs are created (for example COLOR=BLUE, where BLUE is a new property entry) you can generate and defer FRV index updates during the Z-step of a KEY index.

Deferred updates and Reuse Record Number files

Do not use deferred update mode with Reuse Record Number (RRN) files except where you are performing simple record adds, such as reorganizing files. Using deferred update mode with RRN files might cause you to lose updates without any warning. For more information, see RRN files and deferred update mode.