Floating point conversion, rounding, and precision rules

Conversion

Values can be stored in floating point fields using any of the file update statements (for example, STORE RECORD) or can be stored in a %variable. When the value to be stored is supplied as a string, Model 204 attempts to convert the string to floating point.

Significant digits past the fifteenth digit are ignored and are treated as zeros regardless of the precision of the value's receptacle. This applies to numbers assigned to fields or %variables or used in arithmetic expressions.

Mapping and precision adjustment

For 8-byte and 16-byte floating point numbers, Model 204 maintains 15 significant decimal digits of precision. For 4-byte floating point numbers, Model 204 maintains six significant digits of precision.

Floating point format is an approximate numeric representation that cannot always directly represent decimal values. To provide exact equality in comparisons, Model 204 maps floating point numbers to the floating point number closest in value to a 6- or 15-digit decimal number. This mapping process is, in effect, a decimal rounding process. This mapping occurs in the following cases:

1. Decimal rounding occurs after addition and subtraction arithmetic operations involving floating point numbers.
2. If the field is defined as KEY, the value to be indexed is rounded.
3. If the field is defined as ORDERED NUMERIC, the value to be indexed is rounded.
4. If the field is used in a direct Table B search in a FIND statement, both the value to be searched for and each field accessed in Table B are rounded.
5. If the field is to be used as a key in a SORT statement, the value is rounded before being concatenated to the key.

Assigning floating point numbers to floating point numbers of different lengths

Assigning floating point numbers to floating point numbers of different lengths indicates how precision is adjusted with floating point numbers of different lengths.

Floating point precision adjustment

Assignment, comparison, or conversion	Precision adjustment
FLOAT 8 to FLOAT 4	Hexadecimal rounding (that is, if second half of an 8-byte floating point number has its high order bit on, a bit is added to the first half of the 8-byte number).
FLOAT 16 to FLOAT 8	Truncated.
FLOAT 4 to FLOAT 8	Decimal rounding (that is, value is converted to the 8-byte floating point number closest in value to the 6-digit decimal representation of the FLOAT 4 value).
FLOAT 8 to FLOAT 16	Padded with zeros.