Release notes for Sirius Mods V8.0: Difference between revisions

From m204wiki
Jump to navigation Jump to search
m (Created page with "{{Template:Content index: V8.0 release notes}} These release notes list the enhancements and other changes contained in ''Sirius Mods'' version 8.0, which is still in developmen...")
 
mNo edit summary
 
(47 intermediate revisions by 6 users not shown)
Line 1: Line 1:
{{Template:Content index: V8.0 release notes}}
These release notes list the enhancements and other changes contained in <var class="product">Sirius Mods</var> version 8.0, which was released on April 16, 2012. The immediately preceding and following versions of the <var class="product">Sirius Mods</var> are:
<ul>
<li>Version 7.9, available in July, 2011 &mdash; [[Release notes for Sirius Mods V7.9|Release notes]]</li>
<li>Version 8.1, available in November, 2012 &mdash; [[Release notes for Sirius Mods V8.1|Release notes]]</li>
</ul>
==Changes to classes and methods==
===Collection classes===
====New exception class: ItemNotPresent====
The <var>[[ItemNotPresent class|ItemNotPresent]]</var> exception class indicates that the name used for a [[NamedArraylist class|NamedArraylist]], [[FloatNamedArraylist class|FloatNamedArraylist]], or [[UnicodeNamedArraylist class|UnicodeNamedArraylist]] <var>Item</var> method is not present in the collection.
===Email class===
====New method: RemoveRecipient====
The <var>[[RemoveRecipient (Email function)|RemoveRecipient]]</var> method removes a specified name from the "To:", "CC:", and "BCC:" lists of recipients of an <var>Email</var> object.
===Intrinsic classes===
The following subsections describe new and changed [[Intrinsic]] methods available in <var class="product">Sirius Mods</var> 8.0.
====New String method: CharacterToUnicode====
<var>[[CharacterToUnicode (String function)|CharacterToUnicode]]</var> uses a (new in <var class="product">Mods</var> 8.0) <var>[[#New classes: CharacterToUnicodeMap, UdpSocket|CharacterToUnicodeMap]]</var> object to translate a given character to Unicode.
===XmlDoc API===
The following sections describe new and changed [[XmlDoc API]] methods available in <var class="product">Sirius Mods</var> 8.0.
====XmlDoc/XmlNode Serial function support for untranslatable Unicode====
<var>[[Serial (XmlDoc/XmlNode function)|Serial]]</var> with the <var>EBCDIC</var> option now produces the character encoding for an untranslatable Unicode character. Prior to version 8.0, an untranslatable Unicode character resulted in a request cancellation.
<p>Also, the <var>CharacterEncodeAll</var> option can now be used with <var>Serial</var>.
</p>
===New classes: CharacterToUnicodeMap, UdpSocket===
<ul>
<li>The <var>[[CharacterToUnicodeMap class|CharacterToUnicodeMap]]</var> class is similar to the <var>[[CharacterMap class|CharacterMap]]</var> class, except it maps single-byte code points to Unicode. This class lets a <var class="product">User Language</var> programmer dynamically build a codepage mapping for any codepage they might encounter.</li>
<li>The <var>[[UdpSocket class|UdpSocket]]</var> class lets <var class="product">User Language</var> programs send and receive UDP ([http://en.wikipedia.org/wiki/User_Datagram_Protocol User Datagram Protocol]) messages.</li>
</ul>
===Generic Print, Audit, and Trace methods for user objects===
Available only for non-<var>[[System classes and methods|System]]</var> <var class="product">User Language</var> objects, the [[Print, Audit, and Trace methods for user objects|Print, Audit, and Trace]] methods
output a display of the current values of the <var>Variable</var> members of the method object.
These methods are intended as programmer debugging and auditing tools and are <strong>not</strong> intended to be a core part of applications.
==Other feature changes==
===SSL security===
====SSL support for Janus FTP Server ports====
SSL (Secure Sockets Layer) data transmission is now [[Janus_FTP_Server#Security_and_Janus_FTP_Server|available for Janus FTP Server]] ports. [http://en.wikipedia.org/wiki/FTPS#Methods_of_invoking_security Explicit FTPS] connections only are supported. To enable this SSL/TLS support for an <var>[[JANUS DEFINE#type|FTPSERVER]]</var> port, you include the <var>SSL</var> parameter in the port's <var>[[Janus_FTP_Server_command_reference#JANUS_DEFINE_for_FTP|JANUS DEFINE]]</var> specification just as for other Janus server ports. If you also include the <var>SSLOPT</var> parameter on the port definition, clients can choose whether to use SSL/TLS or not.
====New and updated methods for SSL authentication support====


These release notes list the enhancements and other changes contained in ''Sirius Mods'' version 8.0, which is still in development. The previous version of the ''Sirius Mods'', 7.9, became available in July, 2011. [[Release Notes for Sirius Mods V7.9]] are also available.
<ul>
<li><b>Longstring version of SSL entities</b>
<p>
Two new methods and three updated methods now obtain a longstring copy of the binary (ASN.1 encoded) contents of an SSL certificate (client or server):
<ul>
<li>The new <var>System</var> class <var>[[ClientCertificate (System function)|ClientCertificate]]</var> method</li>
<li>The new <var>Socket</var> class <var>[[Certificate (Socket function)|Certificate]]</var> method</li>
<li>The updated <var>HttpRequest</var> class <var>[[Get (HttpRequest function)|Get]]</var>, <var>[[Post (HttpRequest function)|Post]]</var>, and <var>[[Send (HttpRequest function)|Send]]</var> methods
This method extracts a PEM-encoded string (certificate, CRL) from a <var>Stringlist</var>:</li>
<li>The new <var>[[Stringlist class|Stringlist]]</var> class <var>[[PemToString_(Stringlist_function)|PemToString]]</var> method</li>
</ul>
</p>
<div id="DER2Xml"></div>
<li><b>XmlDoc version of SSL entities</b>
<ul>
<li><var>[[DerToXmlDoc_(String_function)|DerToXmlDoc]]</var> converts almost any DER-encoded string to an <var>XmlDoc</var>.</li>
</ul>
The following <var>[[String class|String]]</var> functions also convert DER-encoded strings to an <var>XmlDoc</var>, but they do so with XML tags that more specifically suit the source entity (CRL, private key, certificate) than does the more generic <var>DerToXmlDoc</var>:
<ul>
<li><var>[[X509CrlToXmlDoc_(String_function)|X509CrlToXmlDoc]]</var></li>
<li><var>[[RSAPrivateKeyToXmlDoc _(String_function)|RSAPrivateKeyToXmlDoc]]</var></li>
<li><var>[[X509CertificateToXmlDoc_(String_function)|X509CertificateToXmlDoc]]</var></li>
</li></ul>
</ul>


====New Exception classes====
These exception classes are associated with some of the new methods described above:
<ul>
<li><b>NotJanusConnection</b>
<p>
A <var>[[NotJanusConnection class|NotJanusConnection]]</var>
exception indicates that the method call originated on a thread whose associated Janus port is not a
Janus server thread.
</p>
<li><b>InvalidPemData</b>
<p>
An <var>[[InvalidPemData class|InvalidPemData]]</var>
exception indicates that data does not conform to the PEM (Privacy Enhanced Mail) protocol.
</p>
<li><b>InvalidBerData</b>
<p>
An <var>[[InvalidBerData class|InvalidBerData]]</var>
exception indicates that data does not conform to the BER (Basic Encoding Rules) protocol.
</p>
</ul>
===Model 204 commands and parameters===
====New commands: PUSHUTABLE and POPUTABLE====
<var class="product">Sirius Mods</var> Version 8.0 contains support for the new <var class="product">Model 204</var> [[PUSHUTABLE and POPUTABLE commands]].
For example:<p class="code">PROCEDURE P.VERYBIG
PUSHUTABLE LSTBL 20000+ LQTBL 120% LVTBL 2000+ LFSCB 20000-
b
...
end  </p>
'''Note:''' The parameter settings in the <var>PUSHUTABLE</var> command above are relative values. <code>LSTBL 20000+</code> indicates a value that is 20,000 bytes greater than the LSTBL setting currently in effect. <code>LQTBL 120%</code> indicates a value that is 20% greater than the current LQTBL setting. <code>LFSCB 20000-</code> indicates a value that is 20,000 less than the current LFSCB setting. These three indicators &mdash; a following plus sign (<tt>+</tt>), minus sign (<tt>-</tt>), or percent sign (<tt>%</tt>) &mdash; are the only relative setting indicators. They are available for all <var class="product">Model 204</var> numeric parameters.
====Relative values for numeric parameters====
<var class="product">Sirius Mods</var> Version 8.0 adds support for specifying [[Relative values for Model 204 numeric parameters|relative values for Model 204 parameters]]. The feature is demonstrated in the example above.
===User Language enhancements===
As described in [[Janus SOAP User Language Interface#Other User Language Enhancements|"Other User Language Enhancements"]], the <var class="product">Janus SOAP User Language Interface</var> provides a number of enhancements to <var class="product">User Language</var> in addition to the object-oriented enhancements.
Features among this set that are new in version 8.0 of the <var class="product">Sirius Mods</var> are shown in the sections below.
====Relative values for parameter settings====
See [[#Relative values for numeric parameters|"Relative values for numeric parameters"]] for more information about this feature.
===Additional methods displayable for debugging===
A large number of <var>[[StringTokenizer class|StringTokenizer]]</var> functions and properties which previously were not displayable for debugging
have been modified so that their result can now be displayed using the <var class="product">[[Debugger|Janus Debugger]]</var> or the <var class="product">[[Debugger|Sirius Debugger]]</var> or using <var class="product">[[SirFact]]</var>.
===Deflate compression performance enhancement===
Changes in the deflate compression algorithm reduce CPU consumption by about 20 percent when compared with <var class="product">Sirius Mods</var> Version 7.9. The difference in content compression is negligible, generally less than one-half of one percent. CPU savings can be greater than 20 percent when using higher compression values in your <var class="product">Janus Web</var> port definitions (<var>[[COMPRESS]]</var>) or in the <var>[[Deflate (String function)|Deflate]]</var> or <var>[[Gzip (String function)|Gzip]]</var> methods (<var>Maxchain</var> parameter).
==Compatibility and fixes==
===Support for Model 204 V7R4===
<!-- the usual section header is:
===Model 204 support===
-->
<var class="product">Sirius Mods</var> version 8.0 is the earliest version of the <var class="product">Sirius Mods</var> that supports V7R4 of <var class="product">Model 204</var>.
Version 8.0 also supports <var class="product">Model 204</var> V7R1.
===Backwards compatibility with Sirius Mods 7.9 and Sirius Mods 8.0===
<!-- ***  No incompatibilities, so none listed, and no zaps: ***
    Sorry that's not the case; if it were, comment out the following paragraph:
-->
This section lists any differences in processing that result from execution with <var class="product">Sirius Mods</var> version 8.0, as compared with the same inputs to <var class="product">Sirius Mods</var> version 7.9 at current maintenance levels. In some cases zaps have been delivered to change the behavior to be the same as the version 8.0 behavior; these cases are explicitly listed.
In general, backwards incompatibility means that an operation which was
previously performed without any indication of error, now operates, given the same inputs and conditions, in a different manner. We may not list as backwards incompatibilities those cases in which the previous behaviour, although not indicating an error, was "clearly and obviously" incorrect, and which are introduced as normal bug fixes (whether or not they had been fixed with previous maintenance).
<!--
''No incompatibilies have been introduced'' in <var class="product">Sirius Mods</var> version 7.8.
-->
====Allowed WebSend (or AddXml or Xml) of non-empty XmlDoc with no element====
If a non-EMPTY <var>XmlDoc</var> does not contain an element node, some serialization methods (that is, <var>[[WebSend (XmlDoc subroutine)|WebSend]]</var>, <var>[[Xml (XmlDoc function)|Xml]]</var> and <var>[[AddXml (HttpRequest subroutine)|AddXml]]</var>) should cancel the request, since that is not a valid XML document.  However, the serialization is allowed. For example:
<p class="code">%x:addPI('a', 'b')
print %x:xml:AsciiToEbcdic
</p>
This now results in:
<p class="output">CANCELLING REQUEST: ...non-EMPTY XmlDoc with no Element node ...
</p>
With previous versions of the <var class="product">Sirius Mods</var>, the result was:
<p class="output"><?a b?>
</p>
This fix was delivered as maintenance to version 7.9 of the <var class="product">Sirius Mods</var> as ZAP7985.
====StringTokenizer incompatibilities====
As noted below in [[#StringTokenizer bug fixes|"StringTokenizer bug fixes"]], fixes to some <var>StringTokenizer</var> methods have resulted in compatibiliy issues.  Also note that the documentation of the <var>[[StringTokenizer class|StringTokenizer]]</var> class and <var>StringTokenizer</var> methods now assumes usage with <var class="product">Sirius Mods</var> version 8.0 or later.
===Fixes in Sirius Mods 8.0 but not in 7.9===
<!--As of the date of the release of version 8.0 of the <var class="product">Sirius Mods</var>, there are no fixes to functionality existing in version 7.9 which have not fixed in that version.
-->
This section lists fixes to functionality existing in the <var class="product">Sirius Mods</var> version 7.9 but which, due to the absence of customer problems, have not, as of the date of the release, been fixed in that version.
====Error position in XML parsing====
The character position of an XML parse error has been changed if the
input to the parse is a <var>[[Stringlist class|Stringlist]]</var>.  Previously, the line-end character inserted between <var>Stringlist</var> items were not counted.  With this
change, you can now do something like the following:
<p class="code">%p = %doc:[[LoadXml (XmlDoc/XmlNode function)|LoadXml]](%stringlis, 'ERRRET')
If %p Then
  %junk = %stringlis:CreateLines
  PrintText Bigger snippet at {%p} than error messages: {%junk:Substring(%p - 100, 200)}
End If
</p>
====Stringtokenizer bug fixes====
This section describes fixes to some <var>StringTokenizer</var> methods. The fixes
were delivered in version 8.0 of the <var>Sirius Mods</var> as ZAP80G4.
The first of these fixes, "driving <var>OutOfBounds</var> handling", represents
an incompatibility that existed prior to the introduction of this fix (that is, with version 7.9 of the <var>Sirius Mods</var>).
Although it is perhaps an "edge case",
it corrects behavior that is central to the <var>StringTokenizer</var> semantics. Therefore,
the documentation of the <var>[[StringTokenizer class|StringTokenizer]]</var> class and <var>StringTokenizer</var>
methods now assumes you are using <var class="product">Sirius Mods</var> version 8.0 or later.
=====Skip processing should drive OutOfBounds handling=====
<b>Skip</b> processing should drive <var>OutOfBounds</var> handling,
either causing an exception or returning the <var>Default</var> value.
For example:
<p class="code">%t:string = 'foo bar'
%t:separators = ','
printText {~=%t:nextToken(skip=2, default='qqq')}
%t:string = 'foo bar'
printText {~=%t:nextToken(skip=2)}
</p>
Prior to this fix, the result of the above fragment was:
<p class="output">%t:nextToken(skip=2, default='qqq')=
%t:nextToken(skip=2)=
</p>
Now, with the fix, the result is:
<p class="output"><nowiki>%t:nextToken(skip=2, default='qqq')=qqq
%t:nextToken(skip=2)=
***  1  CANCELLING REQUEST: ... function NextToken: OutOfBounds exception: Past end of string ...
</nowiki></p>
The new result is correct: the <var>Default</var> value is returned
by the first <var>NextToken</var>, and the second <var>NextToken</var> throws an
exception.
As another example:
<p class="code">%t:string = ''
%t:skipTokens(1)
print 'Should not be here'
</p>
Prior to this fix, the result of the above fragment was:
<p class="output">Should not be here
</p>
Now, with the fix, the result is:
<p class="output"><nowiki>***  1  CANCELLING REQUEST: ... subroutine SkipTokens: OutOfBounds exception: Past end of string ...
</nowiki></p>
The new result is correct: <var>SkipTokens</var> throws an exception here.
All of the above changes, strictly speaking, represent
incompatibilities with the previous behavior of the <var>StringTokenizer</var>.
=====Self-delimited token followed by final separator is not the final token=====
If a self-delimited token is immediately followed by a trailing separator,
there is still another remaining token (that being the null string).
For example:
<p class="code">%t:string = '"asdf",'
%t:quotes = '"'
printText {~=%t:nextToken} {~=%t:nextToken}
</p>
Prior to this fix, the result of the above fragment was:
<p class="output"><nowiki>%t:nextToken=asdf %t:nextToken=
***  1  CANCELLING REQUEST: ... function NextToken: OutOfBounds exception: Past end of string ...
</nowiki></p>
Now, with the fix, the result is:
<p class="output">%t:nextToken=asdf %t:nextToken=
</p>
The new result is correct: <var>NextToken</var> does not throw an exception here.
<!-- Uncomment if this section has any contents
===Version corequisites===
This section lists any restrictions on usage of various products (including <var class="product">Sirius Mods</var> itself) that will be imposed by use of version 8.0 of <var class="product">Sirius Mods</var>.
-->
[[Category:Sirius Mods release notes|Sirius Mods V8.0]]
[[Category:Sirius Mods release notes|Sirius Mods V8.0]]
[[Category:Release notes|Sirius Mods V8.0]]
[[Category:Release notes|Sirius Mods V8.0]]

Latest revision as of 15:46, 27 August 2014

These release notes list the enhancements and other changes contained in Sirius Mods version 8.0, which was released on April 16, 2012. The immediately preceding and following versions of the Sirius Mods are:

Changes to classes and methods

Collection classes

New exception class: ItemNotPresent

The ItemNotPresent exception class indicates that the name used for a NamedArraylist, FloatNamedArraylist, or UnicodeNamedArraylist Item method is not present in the collection.

Email class

New method: RemoveRecipient

The RemoveRecipient method removes a specified name from the "To:", "CC:", and "BCC:" lists of recipients of an Email object.

Intrinsic classes

The following subsections describe new and changed Intrinsic methods available in Sirius Mods 8.0.

New String method: CharacterToUnicode

CharacterToUnicode uses a (new in Mods 8.0) CharacterToUnicodeMap object to translate a given character to Unicode.

XmlDoc API

The following sections describe new and changed XmlDoc API methods available in Sirius Mods 8.0.

XmlDoc/XmlNode Serial function support for untranslatable Unicode

Serial with the EBCDIC option now produces the character encoding for an untranslatable Unicode character. Prior to version 8.0, an untranslatable Unicode character resulted in a request cancellation.

Also, the CharacterEncodeAll option can now be used with Serial.

New classes: CharacterToUnicodeMap, UdpSocket

  • The CharacterToUnicodeMap class is similar to the CharacterMap class, except it maps single-byte code points to Unicode. This class lets a User Language programmer dynamically build a codepage mapping for any codepage they might encounter.
  • The UdpSocket class lets User Language programs send and receive UDP (User Datagram Protocol) messages.

Generic Print, Audit, and Trace methods for user objects

Available only for non-System User Language objects, the Print, Audit, and Trace methods output a display of the current values of the Variable members of the method object.

These methods are intended as programmer debugging and auditing tools and are not intended to be a core part of applications.

Other feature changes

SSL security

SSL support for Janus FTP Server ports

SSL (Secure Sockets Layer) data transmission is now available for Janus FTP Server ports. Explicit FTPS connections only are supported. To enable this SSL/TLS support for an FTPSERVER port, you include the SSL parameter in the port's JANUS DEFINE specification just as for other Janus server ports. If you also include the SSLOPT parameter on the port definition, clients can choose whether to use SSL/TLS or not.

New and updated methods for SSL authentication support

  • Longstring version of SSL entities

    Two new methods and three updated methods now obtain a longstring copy of the binary (ASN.1 encoded) contents of an SSL certificate (client or server):

  • XmlDoc version of SSL entities
    • DerToXmlDoc converts almost any DER-encoded string to an XmlDoc.

    The following String functions also convert DER-encoded strings to an XmlDoc, but they do so with XML tags that more specifically suit the source entity (CRL, private key, certificate) than does the more generic DerToXmlDoc:

New Exception classes

These exception classes are associated with some of the new methods described above:

  • NotJanusConnection

    A NotJanusConnection exception indicates that the method call originated on a thread whose associated Janus port is not a Janus server thread.

  • InvalidPemData

    An InvalidPemData exception indicates that data does not conform to the PEM (Privacy Enhanced Mail) protocol.

  • InvalidBerData

    An InvalidBerData exception indicates that data does not conform to the BER (Basic Encoding Rules) protocol.

Model 204 commands and parameters

New commands: PUSHUTABLE and POPUTABLE

Sirius Mods Version 8.0 contains support for the new Model 204 PUSHUTABLE and POPUTABLE commands.

For example:

PROCEDURE P.VERYBIG PUSHUTABLE LSTBL 20000+ LQTBL 120% LVTBL 2000+ LFSCB 20000- b ... end

Note: The parameter settings in the PUSHUTABLE command above are relative values. LSTBL 20000+ indicates a value that is 20,000 bytes greater than the LSTBL setting currently in effect. LQTBL 120% indicates a value that is 20% greater than the current LQTBL setting. LFSCB 20000- indicates a value that is 20,000 less than the current LFSCB setting. These three indicators — a following plus sign (+), minus sign (-), or percent sign (%) — are the only relative setting indicators. They are available for all Model 204 numeric parameters.

Relative values for numeric parameters

Sirius Mods Version 8.0 adds support for specifying relative values for Model 204 parameters. The feature is demonstrated in the example above.

User Language enhancements

As described in "Other User Language Enhancements", the Janus SOAP User Language Interface provides a number of enhancements to User Language in addition to the object-oriented enhancements.

Features among this set that are new in version 8.0 of the Sirius Mods are shown in the sections below.

Relative values for parameter settings

See "Relative values for numeric parameters" for more information about this feature.

Additional methods displayable for debugging

A large number of StringTokenizer functions and properties which previously were not displayable for debugging have been modified so that their result can now be displayed using the Janus Debugger or the Sirius Debugger or using SirFact.

Deflate compression performance enhancement

Changes in the deflate compression algorithm reduce CPU consumption by about 20 percent when compared with Sirius Mods Version 7.9. The difference in content compression is negligible, generally less than one-half of one percent. CPU savings can be greater than 20 percent when using higher compression values in your Janus Web port definitions (COMPRESS) or in the Deflate or Gzip methods (Maxchain parameter).

Compatibility and fixes

Support for Model 204 V7R4

Sirius Mods version 8.0 is the earliest version of the Sirius Mods that supports V7R4 of Model 204.

Version 8.0 also supports Model 204 V7R1.

Backwards compatibility with Sirius Mods 7.9 and Sirius Mods 8.0

This section lists any differences in processing that result from execution with Sirius Mods version 8.0, as compared with the same inputs to Sirius Mods version 7.9 at current maintenance levels. In some cases zaps have been delivered to change the behavior to be the same as the version 8.0 behavior; these cases are explicitly listed.

In general, backwards incompatibility means that an operation which was previously performed without any indication of error, now operates, given the same inputs and conditions, in a different manner. We may not list as backwards incompatibilities those cases in which the previous behaviour, although not indicating an error, was "clearly and obviously" incorrect, and which are introduced as normal bug fixes (whether or not they had been fixed with previous maintenance).

Allowed WebSend (or AddXml or Xml) of non-empty XmlDoc with no element

If a non-EMPTY XmlDoc does not contain an element node, some serialization methods (that is, WebSend, Xml and AddXml) should cancel the request, since that is not a valid XML document. However, the serialization is allowed. For example:

%x:addPI('a', 'b') print %x:xml:AsciiToEbcdic

This now results in:

CANCELLING REQUEST: ...non-EMPTY XmlDoc with no Element node ...

With previous versions of the Sirius Mods, the result was:

<?a b?>

This fix was delivered as maintenance to version 7.9 of the Sirius Mods as ZAP7985.

StringTokenizer incompatibilities

As noted below in "StringTokenizer bug fixes", fixes to some StringTokenizer methods have resulted in compatibiliy issues. Also note that the documentation of the StringTokenizer class and StringTokenizer methods now assumes usage with Sirius Mods version 8.0 or later.

Fixes in Sirius Mods 8.0 but not in 7.9

This section lists fixes to functionality existing in the Sirius Mods version 7.9 but which, due to the absence of customer problems, have not, as of the date of the release, been fixed in that version.

Error position in XML parsing

The character position of an XML parse error has been changed if the input to the parse is a Stringlist. Previously, the line-end character inserted between Stringlist items were not counted. With this change, you can now do something like the following:

%p = %doc:LoadXml(%stringlis, 'ERRRET') If %p Then %junk = %stringlis:CreateLines PrintText Bigger snippet at {%p} than error messages: {%junk:Substring(%p - 100, 200)} End If

Stringtokenizer bug fixes

This section describes fixes to some StringTokenizer methods. The fixes were delivered in version 8.0 of the Sirius Mods as ZAP80G4.

The first of these fixes, "driving OutOfBounds handling", represents an incompatibility that existed prior to the introduction of this fix (that is, with version 7.9 of the Sirius Mods). Although it is perhaps an "edge case", it corrects behavior that is central to the StringTokenizer semantics. Therefore, the documentation of the StringTokenizer class and StringTokenizer methods now assumes you are using Sirius Mods version 8.0 or later.

Skip processing should drive OutOfBounds handling

Skip processing should drive OutOfBounds handling, either causing an exception or returning the Default value. For example:

%t:string = 'foo bar' %t:separators = ',' printText {~=%t:nextToken(skip=2, default='qqq')} %t:string = 'foo bar' printText {~=%t:nextToken(skip=2)}

Prior to this fix, the result of the above fragment was:

%t:nextToken(skip=2, default='qqq')= %t:nextToken(skip=2)=

Now, with the fix, the result is:

%t:nextToken(skip=2, default='qqq')=qqq %t:nextToken(skip=2)= *** 1 CANCELLING REQUEST: ... function NextToken: OutOfBounds exception: Past end of string ...

The new result is correct: the Default value is returned by the first NextToken, and the second NextToken throws an exception.

As another example:

%t:string = %t:skipTokens(1) print 'Should not be here'

Prior to this fix, the result of the above fragment was:

Should not be here

Now, with the fix, the result is:

*** 1 CANCELLING REQUEST: ... subroutine SkipTokens: OutOfBounds exception: Past end of string ...

The new result is correct: SkipTokens throws an exception here.

All of the above changes, strictly speaking, represent incompatibilities with the previous behavior of the StringTokenizer.

Self-delimited token followed by final separator is not the final token

If a self-delimited token is immediately followed by a trailing separator, there is still another remaining token (that being the null string). For example:

%t:string = '"asdf",' %t:quotes = '"' printText {~=%t:nextToken} {~=%t:nextToken}

Prior to this fix, the result of the above fragment was:

%t:nextToken=asdf %t:nextToken= *** 1 CANCELLING REQUEST: ... function NextToken: OutOfBounds exception: Past end of string ...

Now, with the fix, the result is:

%t:nextToken=asdf %t:nextToken=

The new result is correct: NextToken does not throw an exception here.