Record (File architecture): Difference between revisions

From m204wiki
Jump to navigation Jump to search
(Automatically generated page update)
m (mixed case for For Record Number)
 
(52 intermediate revisions by 9 users not shown)
Line 1: Line 1:
== Overview ==
==Overview==
<p>
A Model 204 <b>record</b> is a collection of fields (either individual, in field groups, or any mixture of these) containing a set of data about a thing. Admittedly, this is a very vague description, because the flexibility of Model 204 permits the File Manager to create files whose records are best suited to the required purpose, whatever it may be.</p> 


<p>A Model 204 Record is a collection of fields (either individually or in Repeating Field Groups (RFGs)) containing a set of data about a thing. Admittedly, this is a very vague description, because the flexibility of Model 204 permits the File Manager to create files whose records are best suited to the required purpose, whatever it may be.</p>
<p>Each record is variable in length and need contain only the fields (some or all of those contained in the [[Table A (File architecture)#Internal file dictionary|internal file dictionary in Table A]]) that pertain to it. The limit of the number of field value pairs in a record is in the tens of millions.</p>


<p>Each record is variable in length and need contain only the fields (some or all of those contained in the [[#Table A (File Architecture) Internal File Dictionary|Internal File Dictionary in Table A]]) that pertain to it. The limit of the number of field value pairs in a record is in the tens of millions.</p>
<p>There is only a limited fixed format for data within a record (preallocated fields). Almost any number of fields can appear almost any number of times in almost any order. Each record is automatically assigned a unique internal record number (IRN) that is used by the system to build index entries for the record. The IRN is temporary and will likely change when the file is reorganized.</p>  


<p>There is only a limited fixed format for data within a record (pre-allocated fields). Almost any number of fields can appear almost any number of times in almost any order. Each record is automatically assigned a unique internal record number that is used by the system to build index entries for the record.</p>  
<p>For example, the following code does, in fact, physically store a record:</p>


<p>For example, the following code:</p>
<p class="code">IN ''filename'' STORE RECORD
 
<p class="code">
IN ''filename'' STORE RECORD
END STORE
END STORE
</p>
</p>


<p>Does, in fact, physically store a record. It has been assigned an [[#Internal Record Number|Internal Record Number]] and would contain the pre-allocated field structure (if any), but no data.</p>
<p>The record is assigned an [[#Internal Record Number|Internal Record Number]] and contains the preallocated field structure (if any) but no data.</p>
 
 
== The Concept of a Model 204 Record ==
 
<p>
 
<p>As noted above, a Model 204 record is highly variable in length.  </p>
 
 
 
 
 
 
=== <div id="Base Records">Base Records</div> ===
 
=== <div id="Extension Records">Extension Records</div> ===
 
 
=== <div id="Internal Record Number">The Internal Record Number (IRN)</div> ===
 
not permanent (changes during a reorganization)
 
used as target index (both hashed and B-tree indexing)
 
 
== The Structure of a Model 204 Record ==
 
<p>As shown in [[Table B (File Architecture)|Table B]], Model 204 records grow from the end toward the beginning of the page it is stored upon. So, a proper representation of the record shows it as:</p> 


==Records, base records, and extension records==
<p>The term "record" is actually used three ways: </p>
<ul>
<li>The record as a full collection of all of the data about a "thing"
<li>Base record: the initial portion of the record data
<li>Extension record(s): a (possible) string of additional portions of the data   
</ul>


::[[File:Record Structure (File Architecture).jpg]]
<p>These "record" terms are discussed in the subsections that follow.</p>


===Record as container===
<p>
A record as a container is just that: a way to describe all of the data associated with the record key (if a record key exists).</p>


<p>In any discussion of the record, the 'beginning' of the record is to the left, and the growth (represented by the blue arrow) is toward the right.</p>  
===Base records===
<p>
The base record is the first (and often, only) part of the record. It is assigned a slot in Table B, and its internal record number is how Model 204 identifies the record for indexing purposes as well as in record sets and lists.</p>


===Extension records===
<p>
When a full record is not able to fit on a particular page (either, it itself may be more than 6140 bytes, or other records on the page do not leave sufficient space for the record), one or more extension records are stored.</p>


=== Extension Pointer ===
<p>If [[Table X (File architecture)|Table X]] is enabled, all such extensions are stored there, otherwise the extension(s) are stored in Table B.</p>


<p>Every 'portion' of every Model 204 record (base or one of perhaps many extensions) begins with a 3 byte extension record pointer.</p>  
===Record growth===
<p>
The [[Data maintenance#ADD statement|ADD statement]] (or the file load process) always places new data at the end of a record, so the record growth is straightforward: add a field at the end of the record, if it doesn't fit, start a new extension.</p>


<p>This chains this part of the record to the next. Where that location is, is dependent on whether [[Table X (File Architecture)|Table X]] is enabled.</p>  
<p>[[Data maintenance#INSERT statement|insertions]] are more complicated. Whether updating fields or field groups, the correct location must be found within a record, and the data inserted at that point.</p>


<p>It is quite possible, that there will not be enough space on the page to insert a record, in which case space is made by moving data from this record (after the insert point) on to an extension record. So, with large extension records, it is conceivable that a single field added to the base record might cause all of the extensions for a record.</p>


=== Data ===
===Internal record number===
<p>
The internal record number (IRN) is the base record number (or slot) assigned when the record is first stored. While, once assigned, it can be passed around inside a program as an easy way to refer back to the set of data (see the <var>[[Record loops#For Record Number processing|For Record Number]]</var> statement), it is not a permanent identifier: when the file is reorganized, the position where a record is reloaded may be quite different than where it was before. </p>


<p>If there are any preallocated fields in a file (see the [[OCCURS attribute]]) every record in that file will begin with the set of such fields, immediately after the extension pointer. The structure of the fields is held in [[#Table D (File Architecture) Record Map|the Record Map in Table D]].</p>  
<p>The IRN (represented by a single bit in a bitmap) is used as a target for Model&nbsp;204 indexing (both hashed and B-tree indexing). See [[Bitmaps (File architecture)|Bitmaps]] for more information.</p>  


<p>Note: For the Sort or Hash key (for such files), if it is preallocated (and it should be unless it is wildly variable in size) it will be the first field in the record map. If it is not preallocated, it will be the first 'field value pair' in the record.</p>          
==The structure of a Model 204 record==
<p>
As shown in [[Table B (File architecture)|Table B]], Model&nbsp;204 records grow from the end toward the beginning of the page it is stored upon. So, a proper representation of the record shows it as:</p>


<p>After the preallocated fields, is the rest of the data; a series of: field value pairs; repeating field groups; and / or pointers to large objects. These will be physically positioned in: either the order they were [[ADD statement|ADD]]ed (which always adds the entity at the end of the record); or positioned by an [[INSERT statement]]. Hence, the application code controls the relative order of the data in the record, once past the preallocated fields.</p>
:[[File:Record Structure (File Architecture).jpg|border|400px]]


<p>Each of the four is briefly described below, but refer to the specific topic for each for further detail:</p>
<p>In any discussion of the record, the "beginning" of the record is to the left, and the growth (represented by the blue arrow) is toward the right.</p>  


==== [[Preallocated Fields (File Architecture)|Preallocated Fields]] ====
===Extension pointer===
<p>
Every "portion" of every Model&nbsp;204 record (base or one of perhaps many extensions) begins with an extension record pointer.</p>


<p>For files without a Table X, this pointer is three bytes in length.</p>


<p>For files with Table X enabled, this pointer is four bytes in length (to handle the larger number of slots in Table X).</p>


==== [[Field Value Pairs (File Architecture)|Field Value Pairs]] ====
<p>Regardless of the length, this pointer links this portion of the record to its next portion. Where that location is, of course, depends on whether Table X is enabled.</p>


Field Value Pairs, or, perhaps more precisely, ''Field Name 'Equals' Value Pairs'' are the most common way data is held in a Model 204 Record. (Even the Repeating Field Group structure is a set of Field Value Pairs which occur in tandem).
===Data===
<p>
If there are any preallocated fields in a file (see the [[Field design#Preallocated fields (OCCURS attribute)|OCCURS attribute]]) every record in that file will begin with the set of such fields, immediately after the extension pointer. The structure of the fields is held in [[#Table D (File architecture) Record Map|the record map in Table D]].</p>


<p class="note">'''Note:''' For the sort or hash key (for such files), if it is preallocated (and it should be unless it is wildly variable in size), it will be the first field in the record map. If it is not preallocated, it will be the first "field value pair" in the record.</p>           


<p>After the preallocated fields comes the rest of the data: a series of field value pairs, field groups, and/or pointers to large objects. These will be physically positioned according to either of the following:
<ul>
<li>The order they were [[Data maintenance#ADD statement|ADD]]ed (which always adds the entity at the end of the record)


<li>An [[Processing multiply occurring fields and field groups#INSERT statement|INSERT statement]].
</ul>
Hence, once past the preallocated fields, the application code controls the relative order of the data in the record.</p>


==== [[Repeating Field Groups (File Architecture|Repeating Field Groups]] ====
<p>The following subsections further describe each of the four record data components, but refer to the specific topic for each for further detail.</p> 


==== Large Objects ====
====Preallocated fields====
<p>
All records stored in a Model 204 file will contain all of the [[Field design#Preallocated fields (OCCURS attribute)|preallocated fields]] (those with the <var>OCCURS</var> attribute) (if any) for that file. These are stored based on the [[Table D (File architecture)#Record map of preallocated fields|Record map of preallocated fields]] stored in Table D.</p>


===== [[BLOBs and CLOBs (File Architecture|Native Large Objects]] =====
====Field value pairs====


===== Large Objects Without Table E =====
[[Field value pairs (File architecture)|Field value pairs]], or, more precisely, ''field name "equals" value pairs'', are the most common way that data is held in a Model 204 record.


<p>If [[Table E (File Architecture)|Table E]] is not enabled, and the File Manager still wishes to store data which is more than 255 bytes long, it must be done by the application code. </p>
Even the [[Field group (File architecture)|field group]] structure is a set of field value pairs which occur in tandem.


<p>The technique is to store the data as a series of repeating fields, each containing (up to) 255 bytes. The 
====Field groups====
<p>
A field group allows sets of data to be managed together. Inside of the field group is a series of field value pairs (as above), but these fields are not independent of each other.</p>


====Large objects====
<p>
Large objects (data longer than 255 bytes) can be handled in a Model&nbsp;204 record in two ways: </p>
<ul>
<li>Native large objects in Table E
<p>
Large objects stored as part of a record contain a pointer locally (within the record) to the starting point of the large object in Table E.</p>
<p>
Starting in Model 204 V7.5, large objects can be handled by using the <var>[[FILEORG parameter|FILEORG]]</var> X'100' bit. See [[Table E (File architecture)|Table E]] for the full discussion.</p></li>


<li>Large objects without Table E
<p>
If Table E is not enabled, and the File Manager still wants to store data that is more than 255 bytes long, storage must be done by the application code. </p>
<p>
The technique is to store the data as a series of repeating fields, each containing (up to) 255 bytes. The application must: </p>
<ol>
<li>Parse the input data into 255-byte chunks to store it. </li>
<li>String the chunks together again to read it. </li>
</ol></li>
</ul>


<p>For efficient storage of large objects, Rocket Software recommends enabling Table E.</p> 


[[Category:File architecture and management]]
[[Category:File architecture]]
[[Category:File architecture]]

Latest revision as of 21:47, 26 April 2018

Overview

A Model 204 record is a collection of fields (either individual, in field groups, or any mixture of these) containing a set of data about a thing. Admittedly, this is a very vague description, because the flexibility of Model 204 permits the File Manager to create files whose records are best suited to the required purpose, whatever it may be.

Each record is variable in length and need contain only the fields (some or all of those contained in the internal file dictionary in Table A) that pertain to it. The limit of the number of field value pairs in a record is in the tens of millions.

There is only a limited fixed format for data within a record (preallocated fields). Almost any number of fields can appear almost any number of times in almost any order. Each record is automatically assigned a unique internal record number (IRN) that is used by the system to build index entries for the record. The IRN is temporary and will likely change when the file is reorganized.

For example, the following code does, in fact, physically store a record:

IN filename STORE RECORD END STORE

The record is assigned an Internal Record Number and contains the preallocated field structure (if any) but no data.

Records, base records, and extension records

The term "record" is actually used three ways:

  • The record as a full collection of all of the data about a "thing"
  • Base record: the initial portion of the record data
  • Extension record(s): a (possible) string of additional portions of the data

These "record" terms are discussed in the subsections that follow.

Record as container

A record as a container is just that: a way to describe all of the data associated with the record key (if a record key exists).

Base records

The base record is the first (and often, only) part of the record. It is assigned a slot in Table B, and its internal record number is how Model 204 identifies the record for indexing purposes as well as in record sets and lists.

Extension records

When a full record is not able to fit on a particular page (either, it itself may be more than 6140 bytes, or other records on the page do not leave sufficient space for the record), one or more extension records are stored.

If Table X is enabled, all such extensions are stored there, otherwise the extension(s) are stored in Table B.

Record growth

The ADD statement (or the file load process) always places new data at the end of a record, so the record growth is straightforward: add a field at the end of the record, if it doesn't fit, start a new extension.

insertions are more complicated. Whether updating fields or field groups, the correct location must be found within a record, and the data inserted at that point.

It is quite possible, that there will not be enough space on the page to insert a record, in which case space is made by moving data from this record (after the insert point) on to an extension record. So, with large extension records, it is conceivable that a single field added to the base record might cause all of the extensions for a record.

Internal record number

The internal record number (IRN) is the base record number (or slot) assigned when the record is first stored. While, once assigned, it can be passed around inside a program as an easy way to refer back to the set of data (see the For Record Number statement), it is not a permanent identifier: when the file is reorganized, the position where a record is reloaded may be quite different than where it was before.

The IRN (represented by a single bit in a bitmap) is used as a target for Model 204 indexing (both hashed and B-tree indexing). See Bitmaps for more information.

The structure of a Model 204 record

As shown in Table B, Model 204 records grow from the end toward the beginning of the page it is stored upon. So, a proper representation of the record shows it as:

In any discussion of the record, the "beginning" of the record is to the left, and the growth (represented by the blue arrow) is toward the right.

Extension pointer

Every "portion" of every Model 204 record (base or one of perhaps many extensions) begins with an extension record pointer.

For files without a Table X, this pointer is three bytes in length.

For files with Table X enabled, this pointer is four bytes in length (to handle the larger number of slots in Table X).

Regardless of the length, this pointer links this portion of the record to its next portion. Where that location is, of course, depends on whether Table X is enabled.

Data

If there are any preallocated fields in a file (see the OCCURS attribute) every record in that file will begin with the set of such fields, immediately after the extension pointer. The structure of the fields is held in the record map in Table D.

Note: For the sort or hash key (for such files), if it is preallocated (and it should be unless it is wildly variable in size), it will be the first field in the record map. If it is not preallocated, it will be the first "field value pair" in the record.

After the preallocated fields comes the rest of the data: a series of field value pairs, field groups, and/or pointers to large objects. These will be physically positioned according to either of the following:

Hence, once past the preallocated fields, the application code controls the relative order of the data in the record.

The following subsections further describe each of the four record data components, but refer to the specific topic for each for further detail.

Preallocated fields

All records stored in a Model 204 file will contain all of the preallocated fields (those with the OCCURS attribute) (if any) for that file. These are stored based on the Record map of preallocated fields stored in Table D.

Field value pairs

Field value pairs, or, more precisely, field name "equals" value pairs, are the most common way that data is held in a Model 204 record.

Even the field group structure is a set of field value pairs which occur in tandem.

Field groups

A field group allows sets of data to be managed together. Inside of the field group is a series of field value pairs (as above), but these fields are not independent of each other.

Large objects

Large objects (data longer than 255 bytes) can be handled in a Model 204 record in two ways:

  • Native large objects in Table E

    Large objects stored as part of a record contain a pointer locally (within the record) to the starting point of the large object in Table E.

    Starting in Model 204 V7.5, large objects can be handled by using the FILEORG X'100' bit. See Table E for the full discussion.

  • Large objects without Table E

    If Table E is not enabled, and the File Manager still wants to store data that is more than 255 bytes long, storage must be done by the application code.

    The technique is to store the data as a series of repeating fields, each containing (up to) 255 bytes. The application must:

    1. Parse the input data into 255-byte chunks to store it.
    2. String the chunks together again to read it.

For efficient storage of large objects, Rocket Software recommends enabling Table E.