Record (File architecture): Difference between revisions

From m204wiki
Jump to navigation Jump to search
m (mixed case for For Record Number)
 
(40 intermediate revisions by 7 users not shown)
Line 1: Line 1:
== Overview ==
==Overview==
<p>
A Model 204 <b>record</b> is a collection of fields (either individual, in field groups, or any mixture of these) containing a set of data about a thing. Admittedly, this is a very vague description, because the flexibility of Model 204 permits the File Manager to create files whose records are best suited to the required purpose, whatever it may be.</p> 


<p>A Model 204 Record is a collection of fields (either individually; in Repeating Field Groups (RFGs); or any mixture of these) containing a set of data about a thing. Admittedly, this is a very vague description, because the flexibility of Model 204 permits the File Manager to create files whose records are best suited to the required purpose, whatever it may be.</p>
<p>Each record is variable in length and need contain only the fields (some or all of those contained in the [[Table A (File architecture)#Internal file dictionary|internal file dictionary in Table A]]) that pertain to it. The limit of the number of field value pairs in a record is in the tens of millions.</p>


<p>Each record is variable in length and need contain only the fields (some or all of those contained in the [[#Table A (File Architecture) Internal File Dictionary|Internal File Dictionary in Table A]]) that pertain to it. The limit of the number of field value pairs in a record is in the tens of millions.</p>
<p>There is only a limited fixed format for data within a record (preallocated fields). Almost any number of fields can appear almost any number of times in almost any order. Each record is automatically assigned a unique internal record number (IRN) that is used by the system to build index entries for the record. The IRN is temporary and will likely change when the file is reorganized.</p>  


<p>There is only a limited fixed format for data within a record (pre-allocated fields). Almost any number of fields can appear almost any number of times in almost any order. Each record is automatically assigned a unique (but temporary... it will likely change when the file is reorganized) internal record number (IRN) that is used by the system to build index entries for the record.</p>  
<p>For example, the following code does, in fact, physically store a record:</p>


<p>For example, the following code:</p>
<p class="code">IN ''filename'' STORE RECORD
 
<p class="code">
IN ''filename'' STORE RECORD
END STORE
END STORE
</p>
</p>


<p>Does, in fact, physically store a record. It has been assigned an [[#Internal Record Number|Internal Record Number]] and would contain the pre-allocated field structure (if any), but no data.</p> 
<p>The record is assigned an [[#Internal Record Number|Internal Record Number]] and contains the preallocated field structure (if any) but no data.</p>
 
 
== Records. Base Records and Extension Records ==
 
<p>The term 'record' is actually used three ways: </p>
 
* the record as a full collection of all of the data about a 'thing'
* base records: the initial portion of that data
* extension records: a (possible) string of additional portions of the data   
 
<p>These are discussed below:</p>
 
=== the Record as a  container ===
 
<p>Is just that: a way to describe all of the data held against the record key (presuming there is one)...</p>
 
=== <div id="Base Records">Base Records</div> ===
 
<p>The base record is the first (and, often, only) part of the record. It is assigned a slot in Table B, and its internal record number is how Model 204 identifies the record for indexing peuporses as well as in record sets and lists.</p>
 
=== <div id="Extension Records">Extension Records</div> ===
 
<p>When a full record is not able to fit on a particular page (either, it itself may be more than 6140 bytes, or other records on the page do not leave sufficient space for the record) one or more extension records are stored.</p>
 
<p>If [[Table X (File Architecture)|Table X]] is enabled, all such extensions are stored there, otherwise the extension(s) are stored in Table B.</p> 
 
=== <div id="Record Growth">Record Growth</div> ===
 
<p>The [[Data Maintenance#ADD statement|ADD statement]] (or the file load process) always places new data at the end of a record, and so, the record growth is straightforward: add a field at the end of the record, if it doesn't fit, start a new extension.</p>
 
<p>[[Data Maintenance#INSERT statement|INSERT]]s are more complicated. Whether updating fields or repeating field groups, the correct location must be found within a record, and the data inserted at that point.</p>
 
<p>It is, of course, quite possible, that there will not be enough space on the page to insert it, in which case space is made by moving data from this record (after the insert point) down the extension chain. So, with large extension records, it is conceivable that a single field added to the base record (for example) might cause all of the extensions to need to change.</p>
 
=== <div id="Internal Record Number">The Internal Record Number (IRN)</div> ===
 
<p>The internal record number (IRN) is the base record number (or slot) assigned when the record is first stored. While, once assigned, it can be passed around inside a program as an easy way to refer back to the set of data (see the [[FOR RECORD NUMBER statement]]) it is not a permanent identifier: when the file is reorganized the position where a record is reloaded may be quite different than where it was before. </p>
 
<p>The IRN (represented by a single bit in a bit map) is used as a target for Model 204 indexing (both hashed and B-tree indexing). See [[Bit Maps (File Architecture)|Bit Maps]] for more information.</p>
 
 
== The Structure of a Model 204 Record ==
 
<p>As shown in [[Table B (File Architecture)|Table B]], Model 204 records grow from the end toward the beginning of the page it is stored upon. So, a proper representation of the record shows it as:</p> 
 
 
::[[File:Record Structure (File Architecture).jpg]]
 
 
<p>In any discussion of the record, the 'beginning' of the record is to the left, and the growth (represented by the blue arrow) is toward the right.</p>
 
 
=== Extension Pointer ===
 
<p>Every 'portion' of every Model 204 record (base or one of perhaps many extensions) begins with an extension record pointer.</p>  


<p>For files without a Table X, this pointer is 3 bytes in length.</p>
==Records, base records, and extension records==
<p>The term "record" is actually used three ways: </p>
<ul>
<li>The record as a full collection of all of the data about a "thing"
<li>Base record: the initial portion of the record data
<li>Extension record(s): a (possible) string of additional portions of the data   
</ul>


<p>For files with [[Table X (File Architecture)|Table X]] enabled, this pointer is 4 bytes in length (to handle the larger number of slots in Table X).</p>
<p>These "record" terms are discussed in the subsections that follow.</p>


<p>Regardless of the length, this pointer links this portion of the record to its next portion. Where that location is, of course, dependent on whether Table X is enabled.</p>  
===Record as container===
<p>
A record as a container is just that: a way to describe all of the data associated with the record key (if a record key exists).</p>


===Base records===
<p>
The base record is the first (and often, only) part of the record. It is assigned a slot in Table B, and its internal record number is how Model 204 identifies the record for indexing purposes as well as in record sets and lists.</p>


=== Data ===
===Extension records===
<p>
When a full record is not able to fit on a particular page (either, it itself may be more than 6140 bytes, or other records on the page do not leave sufficient space for the record), one or more extension records are stored.</p>


<p>If there are any preallocated fields in a file (see the [[Field Design (File Management)#Preallocated fields|OCCURS attribute]]) every record in that file will begin with the set of such fields, immediately after the extension pointer. The structure of the fields is held in [[#Table D (File Architecture) Record Map|the Record Map in Table D]].</p>  
<p>If [[Table X (File architecture)|Table X]] is enabled, all such extensions are stored there, otherwise the extension(s) are stored in Table B.</p>


<p class="note">'''Note:''' For the Sort or Hash key (for such files), if it is preallocated (and it should be unless it is wildly variable in size) it will be the first field in the record map. If it is not preallocated, it will be the first 'field value pair' in the record.</p>          
===Record growth===
<p>
The [[Data maintenance#ADD statement|ADD statement]] (or the file load process) always places new data at the end of a record, so the record growth is straightforward: add a field at the end of the record, if it doesn't fit, start a new extension.</p>


<p>After the preallocated fields, is the rest of the data; a series of: field value pairs; repeating field groups; and/or pointers to large objects. These will be physically positioned in: either the order they were [[ADD statement|ADD]]ed (which always adds the entity at the end of the record); or positioned by an [[INSERT statement]]. Hence, the application code controls the relative order of the data in the record, once past the preallocated fields.</p>  
<p>[[Data maintenance#INSERT statement|insertions]] are more complicated. Whether updating fields or field groups, the correct location must be found within a record, and the data inserted at that point.</p>


<p>Each of the four is briefly described below, but refer to the specific topic for each for further detail:</p>
<p>It is quite possible, that there will not be enough space on the page to insert a record, in which case space is made by moving data from this record (after the insert point) on to an extension record. So, with large extension records, it is conceivable that a single field added to the base record might cause all of the extensions for a record.</p>


==== [[Field Design (File Management)#Preallocated Fields|Preallocated Fields]] ====
===Internal record number===
<p>
The internal record number (IRN) is the base record number (or slot) assigned when the record is first stored. While, once assigned, it can be passed around inside a program as an easy way to refer back to the set of data (see the <var>[[Record loops#For Record Number processing|For Record Number]]</var> statement), it is not a permanent identifier: when the file is reorganized, the position where a record is reloaded may be quite different than where it was before. </p>


<p>All records stored in a Model204 file will contain all of the preallocated fields (those with the <var>OCCURS</var> attribute) (if any) for that file. These are stored based on the [[Table D (File Architecture)#Record Map of Preallocated Fields|Record Map of preallocated fields]] stored in Table D.</p>
<p>The IRN (represented by a single bit in a bitmap) is used as a target for Model&nbsp;204 indexing (both hashed and B-tree indexing). See [[Bitmaps (File architecture)|Bitmaps]] for more information.</p>  


==== [[Field Value Pairs (File Architecture)|Field Value Pairs]] ====
==The structure of a Model 204 record==
<p>
As shown in [[Table B (File architecture)|Table B]], Model&nbsp;204 records grow from the end toward the beginning of the page it is stored upon. So, a proper representation of the record shows it as:</p> 


Field Value Pairs, or, perhaps more precisely, ''Field Name 'Equals' Value Pairs'' are the most common way data is held in a Model 204 Record. (Even the Repeating Field Group structure is a set of Field Value Pairs which occur in tandem).</p>
:[[File:Record Structure (File Architecture).jpg|border|400px]]


<p>In any discussion of the record, the "beginning" of the record is to the left, and the growth (represented by the blue arrow) is toward the right.</p>


==== [[Repeating Field Group (File Architecture)|Repeating Field Groups]] (RFGs)====
===Extension pointer===
<p>
Every "portion" of every Model&nbsp;204 record (base or one of perhaps many extensions) begins with an extension record pointer.</p>


<p>Permits sets of data to be managed together. Inside of the RFG is a series of field value pairs (as above) but these fields are not independent of each other.</p>
<p>For files without a Table X, this pointer is three bytes in length.</p>
 


==== Large Objects ====
<p>For files with Table X enabled, this pointer is four bytes in length (to handle the larger number of slots in Table X).</p>


<p>There are two ways that large objects (data longer than 255 bytes) may be handled in a Model 204 record.</p>
<p>Regardless of the length, this pointer links this portion of the record to its next portion. Where that location is, of course, depends on whether Table X is enabled.</p>


===== [[Table E (File Architecture)|Native Large Objects in Table E]] =====
===Data===
<p>
If there are any preallocated fields in a file (see the [[Field design#Preallocated fields (OCCURS attribute)|OCCURS attribute]]) every record in that file will begin with the set of such fields, immediately after the extension pointer. The structure of the fields is held in [[#Table D (File architecture) Record Map|the record map in Table D]].</p>


<p>Large objects stored as part of a record, contain a pointer locally (within the record) to the starting point of the Large Object in Table E.</p>
<p class="note">'''Note:''' For the sort or hash key (for such files), if it is preallocated (and it should be unless it is wildly variable in size), it will be the first field in the record map. If it is not preallocated, it will be the first "field value pair" in the record.</p>          


<p>Starting in Model 204 V7R5, the handling and functionality of Large Objects can (by using the [[FILEORG parameter|FILEORG]] x'100' bit) be done rather differently. See [[Table E (File Architecture)|Table E]] for the full discussion.</p>
<p>After the preallocated fields comes the rest of the data: a series of field value pairs, field groups, and/or pointers to large objects. These will be physically positioned according to either of the following:
<ul>
<li>The order they were [[Data maintenance#ADD statement|ADD]]ed (which always adds the entity at the end of the record)


<li>An [[Processing multiply occurring fields and field groups#INSERT statement|INSERT statement]].
</ul>
Hence, once past the preallocated fields, the application code controls the relative order of the data in the record.</p>


===== Large Objects Without Table E =====
<p>The following subsections further describe each of the four record data components, but refer to the specific topic for each for further detail.</p> 


<p>If [[Table E (File Architecture)|Table E]] is not enabled, and the File Manager still wishes to store data which is more than 255 bytes long, it must be done by the application code. </p>
====Preallocated fields====
<p>
All records stored in a Model 204 file will contain all of the [[Field design#Preallocated fields (OCCURS attribute)|preallocated fields]] (those with the <var>OCCURS</var> attribute) (if any) for that file. These are stored based on the [[Table D (File architecture)#Record map of preallocated fields|Record map of preallocated fields]] stored in Table D.</p>


<p>The technique is to store the data as a series of repeating fields, each containing (up to) 255 bytes.</p>
====Field value pairs====


<p>The application would need to parse the input data into 255 byte chunks to store it, and then again, string it together to read it.</p>
[[Field value pairs (File architecture)|Field value pairs]], or, more precisely, ''field name "equals" value pairs'', are the most common way that data is held in a Model 204 record.  


<p>Rocket Software recommends that, where large objects need to be stored that, for efficiency reasons, Tabel E be enabled.</p> 
Even the [[Field group (File architecture)|field group]] structure is a set of field value pairs which occur in tandem.


====Field groups====
<p>
A field group allows sets of data to be managed together. Inside of the field group is a series of field value pairs (as above), but these fields are not independent of each other.</p>


====Large objects====
<p>
Large objects (data longer than 255 bytes) can be handled in a Model&nbsp;204 record in two ways: </p>
<ul>
<li>Native large objects in Table E
<p>
Large objects stored as part of a record contain a pointer locally (within the record) to the starting point of the large object in Table E.</p>
<p>
Starting in Model 204 V7.5, large objects can be handled by using the <var>[[FILEORG parameter|FILEORG]]</var> X'100' bit. See [[Table E (File architecture)|Table E]] for the full discussion.</p></li>


<li>Large objects without Table E
<p>
If Table E is not enabled, and the File Manager still wants to store data that is more than 255 bytes long, storage must be done by the application code. </p>
<p>
The technique is to store the data as a series of repeating fields, each containing (up to) 255 bytes. The application must: </p>
<ol>
<li>Parse the input data into 255-byte chunks to store it. </li>
<li>String the chunks together again to read it. </li>
</ol></li>
</ul>


<p>For efficient storage of large objects, Rocket Software recommends enabling Table E.</p> 


[[Category:File architecture]]
[[Category:File architecture]]

Latest revision as of 21:47, 26 April 2018

Overview

A Model 204 record is a collection of fields (either individual, in field groups, or any mixture of these) containing a set of data about a thing. Admittedly, this is a very vague description, because the flexibility of Model 204 permits the File Manager to create files whose records are best suited to the required purpose, whatever it may be.

Each record is variable in length and need contain only the fields (some or all of those contained in the internal file dictionary in Table A) that pertain to it. The limit of the number of field value pairs in a record is in the tens of millions.

There is only a limited fixed format for data within a record (preallocated fields). Almost any number of fields can appear almost any number of times in almost any order. Each record is automatically assigned a unique internal record number (IRN) that is used by the system to build index entries for the record. The IRN is temporary and will likely change when the file is reorganized.

For example, the following code does, in fact, physically store a record:

IN filename STORE RECORD END STORE

The record is assigned an Internal Record Number and contains the preallocated field structure (if any) but no data.

Records, base records, and extension records

The term "record" is actually used three ways:

  • The record as a full collection of all of the data about a "thing"
  • Base record: the initial portion of the record data
  • Extension record(s): a (possible) string of additional portions of the data

These "record" terms are discussed in the subsections that follow.

Record as container

A record as a container is just that: a way to describe all of the data associated with the record key (if a record key exists).

Base records

The base record is the first (and often, only) part of the record. It is assigned a slot in Table B, and its internal record number is how Model 204 identifies the record for indexing purposes as well as in record sets and lists.

Extension records

When a full record is not able to fit on a particular page (either, it itself may be more than 6140 bytes, or other records on the page do not leave sufficient space for the record), one or more extension records are stored.

If Table X is enabled, all such extensions are stored there, otherwise the extension(s) are stored in Table B.

Record growth

The ADD statement (or the file load process) always places new data at the end of a record, so the record growth is straightforward: add a field at the end of the record, if it doesn't fit, start a new extension.

insertions are more complicated. Whether updating fields or field groups, the correct location must be found within a record, and the data inserted at that point.

It is quite possible, that there will not be enough space on the page to insert a record, in which case space is made by moving data from this record (after the insert point) on to an extension record. So, with large extension records, it is conceivable that a single field added to the base record might cause all of the extensions for a record.

Internal record number

The internal record number (IRN) is the base record number (or slot) assigned when the record is first stored. While, once assigned, it can be passed around inside a program as an easy way to refer back to the set of data (see the For Record Number statement), it is not a permanent identifier: when the file is reorganized, the position where a record is reloaded may be quite different than where it was before.

The IRN (represented by a single bit in a bitmap) is used as a target for Model 204 indexing (both hashed and B-tree indexing). See Bitmaps for more information.

The structure of a Model 204 record

As shown in Table B, Model 204 records grow from the end toward the beginning of the page it is stored upon. So, a proper representation of the record shows it as:

In any discussion of the record, the "beginning" of the record is to the left, and the growth (represented by the blue arrow) is toward the right.

Extension pointer

Every "portion" of every Model 204 record (base or one of perhaps many extensions) begins with an extension record pointer.

For files without a Table X, this pointer is three bytes in length.

For files with Table X enabled, this pointer is four bytes in length (to handle the larger number of slots in Table X).

Regardless of the length, this pointer links this portion of the record to its next portion. Where that location is, of course, depends on whether Table X is enabled.

Data

If there are any preallocated fields in a file (see the OCCURS attribute) every record in that file will begin with the set of such fields, immediately after the extension pointer. The structure of the fields is held in the record map in Table D.

Note: For the sort or hash key (for such files), if it is preallocated (and it should be unless it is wildly variable in size), it will be the first field in the record map. If it is not preallocated, it will be the first "field value pair" in the record.

After the preallocated fields comes the rest of the data: a series of field value pairs, field groups, and/or pointers to large objects. These will be physically positioned according to either of the following:

Hence, once past the preallocated fields, the application code controls the relative order of the data in the record.

The following subsections further describe each of the four record data components, but refer to the specific topic for each for further detail.

Preallocated fields

All records stored in a Model 204 file will contain all of the preallocated fields (those with the OCCURS attribute) (if any) for that file. These are stored based on the Record map of preallocated fields stored in Table D.

Field value pairs

Field value pairs, or, more precisely, field name "equals" value pairs, are the most common way that data is held in a Model 204 record.

Even the field group structure is a set of field value pairs which occur in tandem.

Field groups

A field group allows sets of data to be managed together. Inside of the field group is a series of field value pairs (as above), but these fields are not independent of each other.

Large objects

Large objects (data longer than 255 bytes) can be handled in a Model 204 record in two ways:

  • Native large objects in Table E

    Large objects stored as part of a record contain a pointer locally (within the record) to the starting point of the large object in Table E.

    Starting in Model 204 V7.5, large objects can be handled by using the FILEORG X'100' bit. See Table E for the full discussion.

  • Large objects without Table E

    If Table E is not enabled, and the File Manager still wants to store data that is more than 255 bytes long, storage must be done by the application code.

    The technique is to store the data as a series of repeating fields, each containing (up to) 255 bytes. The application must:

    1. Parse the input data into 255-byte chunks to store it.
    2. String the chunks together again to read it.

For efficient storage of large objects, Rocket Software recommends enabling Table E.