Separators (StringTokenizer property)

Characters to be interpreted as token separators (StringTokenizer class)

[Introduced in Sirius Mods 7.8]

Separators provide a way to handle consecutive occurrences of the same token delimiter character, for example, in a comma-separated value (csv) file, where they indicate a missing value.

Separators characters delimit, or separate, tokens as do Spaces and TokenChars characters, but Separators characters differ in that they:

• do not compress to a single separator (like Spaces characters)
• are not themselves tokens (like TokenChars characters), so are not returned by repeated NextToken calls that encounter consecutive Separators characters

Syntax

%currentString = stringTokenizer:Separators stringTokenizer:Separators = newString

Syntax terms

%currentString	A string variable to contain the returned value of the current separator characters. Each character in the string is a separator.
stringTokenizer	A StringTokenizer object.
newString	A string variable to contain the new Separators characters. Each character in the string is a separator, and no character may repeat (except for apostrophe, which may be doubled).

Usage notes

• Separators do not override explicitly defined TokenChars characters. If both separators and token characters are defined, all such characters act as token delimiters.
• Separators override default and explicitly defined Spaces characters. For example, if the only change to the example above is that the tokenizer string is "please, don't go", the result is:

  %toke:nextToken = 'please' %toke:nextToken = 'don't go'

  The blank after don't does not act as a token delimiter.

Examples

The adjacent separators in the token string below are detected and returned as nulls by the NextToken method:

b %toke is object StringTokenizer %toke = new(separators=',;') %toke:string = '0,1,2,,4,;6' repeat while %toke:notAtEnd printtext {~} = '{%toke:nextToken}' end repeat end

The result is:

%toke:nextToken = '0' %toke:nextToken = '1' %toke:nextToken = '2' %toke:nextToken = '' %toke:nextToken = '4' %toke:nextToken = '' %toke:nextToken = '6'

See also

StringTokenizer class	List of StringTokenizer methods	StringTokenizer methods syntax
Notation conventions for methods