Separators (StringTokenizer property)
Characters to be interpreted as token separators (StringTokenizer class)
[Introduced in Sirius Mods 7.8]
Separators provide a way to handle consecutive occurrences of the same token delimiter character, for example, in a comma-separated value (csv) file, where they indicate a missing value.
Separators characters delimit, or separate, tokens as do Spaces and TokenChars characters, but Separators characters differ in that they:
- do not compress to a single separator (like Spaces characters)
- are not themselves tokens (like TokenChars characters), so are not returned by repeated NextToken calls that encounter consecutive Separators characters
Syntax
%currentString = stringTokenizer:Separators stringTokenizer:Separators = newString
Syntax terms
| %currentString | A string variable to contain the returned value of the current separator characters. Each character in the string is a separator. |
|---|---|
| stringTokenizer | A StringTokenizer object. |
| newString | A string variable to contain the new Separators characters. Each character in the string is a separator, and no character may repeat (except for apostrophe, which may be doubled). |
Usage notes
- Separators do not override explicitly defined TokenChars characters. If both separators and token characters are defined, all such characters act as token delimiters.
- Separators override default and explicitly defined Spaces characters.
For example, if the only change to the example above is that the tokenizer string is
"please, don't go", the result is:%toke:nextToken = 'please' %toke:nextToken = 'don't go'
The blank after
don'tdoes not act as a token delimiter.
Examples
The adjacent separators in the token string below are detected and returned as nulls by the NextToken method:
b %toke is object StringTokenizer %toke = new(separators=',;') %toke:string = '0,1,2,,4,;6' repeat while %toke:notAtEnd printtext {~} = '{%toke:nextToken}' end repeat end
The result is:
%toke:nextToken = '0' %toke:nextToken = '1' %toke:nextToken = '2' %toke:nextToken = '' %toke:nextToken = '4' %toke:nextToken = '' %toke:nextToken = '6'