TokenChars (StringTokenizer property): Difference between revisions
mNo edit summary |
mNo edit summary |
||
| Line 1: | Line 1: | ||
{{Template:StringTokenizer:TokenChars subtitle}} | {{Template:StringTokenizer:TokenChars subtitle}} | ||
This readWrite property returns or sets the characters that are recognized as | This [[Classes and Objects#readWrite|readWrite]] property returns or sets the characters that are recognized as | ||
token-delimiter characters, that is, as single-character delimiters that are | token-delimiter characters, that is, as single-character delimiters that are also tokens themselves. | ||
also tokens themselves. | |||
As a delimiter, a token-delimiter character is a non-included boundary at the | As a delimiter, a token-delimiter character is a non-included boundary at the | ||
end or beginning of a token, unless the character is part of a quoted region | end or beginning of a token, unless the character is part of a quoted region | ||
| Line 27: | Line 26: | ||
<li>The <var>TokenChars</var> tokens are initially settable in the | <li>The <var>TokenChars</var> tokens are initially settable in the | ||
<var>[[New (StringTokenizer constructor)|New]]</var> call that creates the tokenizer instance. The default value for a new tokenizer instance is a null. | <var>[[New (StringTokenizer constructor)|New]]</var> call that creates the tokenizer instance. The default value for a new tokenizer instance is a null. | ||
<li>No character can be one of the <var>TokenChars</var> characters and also | <li>No character can be one of the <var>TokenChars</var> characters and also | ||
one of the <var>[[Spaces (StringTokenizer property)|Spaces]]</var>, <var>[[Quotes (StringTokenizer property)|Quotes]]</var>, or <var>[[Separators (StringTokenizer property)|Separators]]</var> characters. | one of the <var>[[Spaces (StringTokenizer property)|Spaces]]</var>, <var>[[Quotes (StringTokenizer property)|Quotes]]</var>, or <var>[[Separators (StringTokenizer property)|Separators]]</var> characters. | ||
| Line 36: | Line 36: | ||
%tok:string = 'y = (x + 1)(x - 1)' | %tok:string = 'y = (x + 1)(x - 1)' | ||
repeat while not %tok:atEnd | repeat while not %tok:atEnd | ||
printText {~} is {%tok:nextToken} | [[Targeted Text statements#AuditText, PrintText, and TraceText|printText]] {~} is {%tok:nextToken} | ||
end repeat | end repeat | ||
</p> | </p> | ||
Latest revision as of 20:58, 15 November 2012
Characters to be interpreted as tokens (StringTokenizer class)
This readWrite property returns or sets the characters that are recognized as
token-delimiter characters, that is, as single-character delimiters that are also tokens themselves.
As a delimiter, a token-delimiter character is a non-included boundary at the
end or beginning of a token, unless the character is part of a quoted region
(in which case it is treated as a non-delimiter character).
Syntax
%currentString = stringTokenizer:TokenChars stringTokenizer:TokenChars = newString
Syntax terms
| %currentString | A string variable to contain the returned value of the current token-delimiter character(s). Each character in the string is a token character. |
|---|---|
| stringTokenizer | A StringTokenizer object. |
| newString | The String value to assign to stringTokenizer's TokenChars property. Each character in the string is a token character — that is, you may not separate characters — and no character may repeat (except for apostrophe, which may be doubled). |
Usage notes
- The TokenChars tokens are initially settable in the New call that creates the tokenizer instance. The default value for a new tokenizer instance is a null.
- No character can be one of the TokenChars characters and also one of the Spaces, Quotes, or Separators characters.
Examples
The statements in the sequence below tokenize the given equation:
%tok = new(tokenchars='()=+') %tok:string = 'y = (x + 1)(x - 1)' repeat while not %tok:atEnd printText {~} is {%tok:nextToken} end repeat
The result is:
%tok:nextToken is y %tok:nextToken is = %tok:nextToken is ( %tok:nextToken is x %tok:nextToken is + %tok:nextToken is 1 %tok:nextToken is ) %tok:nextToken is ( %tok:nextToken is x %tok:nextToken is - %tok:nextToken is 1 %tok:nextToken is )