Separators (StringTokenizer property): Difference between revisions

From m204wiki
Jump to navigation Jump to search
m (Automatically generated page update)
 
mNo edit summary
 
(One intermediate revision by the same user not shown)
Line 1: Line 1:
{{Template:StringTokenizer:Separators subtitle}}
{{Template:StringTokenizer:Separators subtitle}}


This page is [[under construction]].
Separators provide a way to handle consecutive occurrences of the same token delimiter character, for example, in a comma-separated value (csv) file, where they indicate a missing value.
 
<var>Separators</var> characters delimit, or separate, tokens as do <var>[[Spaces (StringTokenizer property)|Spaces]]</var> and <var>[[TokenChars (StringTokenizer property)|TokenChars]]</var> characters, but <var>Separators</var> characters differ in that they:
<ul>
<li>do not compress to a single separator (like <var>Spaces</var> characters)
<li>are not themselves tokens (like <var>TokenChars</var> characters), so are not returned by repeated <var>[[NextToken (StringTokenizer function)|NextToken]]</var> calls that encounter consecutive <var>Separators</var> characters
</ul>
 
==Syntax==
==Syntax==
{{Template:StringTokenizer:Separators syntax}}
{{Template:StringTokenizer:Separators syntax}}
===Syntax terms===
===Syntax terms===
<table class="syntaxTable">
<table class="syntaxTable">
<tr><th>%string</th><td>string</td></tr>
<tr><th>%currentString</th>
<td>A string variable to contain the returned value of the current separator characters. 
Each character in the string is a separator.</td></tr>
 
<tr><th>stringTokenizer</th>
<tr><th>stringTokenizer</th>
<td>StringTokenizer object</td></tr>
<td>A <var>StringTokenizer</var> object.</td></tr>
 
<tr><th>newString</th>
<td>A string variable to contain the new <var>Separators</var> characters. Each character in the string is a separator, and no character may repeat (except for apostrophe, which may be doubled).
</table>
</table>
==Usage notes==
==Usage notes==
<ul>
<li>Separators do '''not''' override explicitly defined <var>TokenChars</var> characters. If both separators and token characters are defined, all such characters act as token delimiters.
<li>Separators override default and explicitly defined <var>Spaces</var> characters.
For example, if the only change to the example above is that the tokenizer string is <code>"please, don't go"</code>,
the result is:
<p class="output"> %toke:nextToken = 'please'
%toke:nextToken = 'don't go'
</p>
The blank after <code>don't</code> does not act as a token delimiter.
</ul>
==Examples==
==Examples==
The adjacent separators in the token string below are detected and returned as nulls by the <var>NextToken</var> method:
<p class="code">b
%toke is object StringTokenizer
%toke = new(separators=',;')
%toke:string = '0,1,2,,4,;6'
repeat while %toke:notAtEnd
  printtext {~} = '{%toke:nextToken}'
end repeat
end
</p>
The result is:
<p class="output">%toke:nextToken = '0'
%toke:nextToken = '1'
%toke:nextToken = '2'
%toke:nextToken = <nowiki>''</nowiki>
%toke:nextToken = '4'
%toke:nextToken = <nowiki>''</nowiki>
%toke:nextToken = '6'
</p>
==See also==
==See also==
{{Template:StringTokenizer:Separators footer}}
{{Template:StringTokenizer:Separators footer}}

Latest revision as of 20:33, 5 July 2012

Characters to be interpreted as token separators (StringTokenizer class)

[Introduced in Sirius Mods 7.8]


Separators provide a way to handle consecutive occurrences of the same token delimiter character, for example, in a comma-separated value (csv) file, where they indicate a missing value.

Separators characters delimit, or separate, tokens as do Spaces and TokenChars characters, but Separators characters differ in that they:

  • do not compress to a single separator (like Spaces characters)
  • are not themselves tokens (like TokenChars characters), so are not returned by repeated NextToken calls that encounter consecutive Separators characters

Syntax

%currentString = stringTokenizer:Separators stringTokenizer:Separators = newString

Syntax terms

%currentString A string variable to contain the returned value of the current separator characters. Each character in the string is a separator.
stringTokenizer A StringTokenizer object.
newString A string variable to contain the new Separators characters. Each character in the string is a separator, and no character may repeat (except for apostrophe, which may be doubled).

Usage notes

  • Separators do not override explicitly defined TokenChars characters. If both separators and token characters are defined, all such characters act as token delimiters.
  • Separators override default and explicitly defined Spaces characters. For example, if the only change to the example above is that the tokenizer string is "please, don't go", the result is:

    %toke:nextToken = 'please' %toke:nextToken = 'don't go'

    The blank after don't does not act as a token delimiter.

Examples

The adjacent separators in the token string below are detected and returned as nulls by the NextToken method:

b %toke is object StringTokenizer %toke = new(separators=',;') %toke:string = '0,1,2,,4,;6' repeat while %toke:notAtEnd printtext {~} = '{%toke:nextToken}' end repeat end

The result is:

%toke:nextToken = '0' %toke:nextToken = '1' %toke:nextToken = '2' %toke:nextToken = '' %toke:nextToken = '4' %toke:nextToken = '' %toke:nextToken = '6'

See also