TokenChars (StringTokenizer property)
Characters to be interpreted as tokens (StringTokenizer class)
This readWrite property returns or sets the characters that are recognized as
token-delimiter characters, that is, as single-character delimiters that are
also tokens themselves.
As a delimiter, a token-delimiter character is a non-included boundary at the
end or beginning of a token, unless the character is part of a quoted region
(in which case it is treated as a non-delimiter character).
Syntax
%currentString = stringTokenizer:TokenChars stringTokenizer:TokenChars = newString
Syntax terms
| %string | A string variable to contain the returned value of the current token-delimiter character(s) or to be set as the new value(s). The default value for a new tokenizer instance is a null.
If you are setting TokenChars, each character in the string is a token character — that is, you may not separate characters — and no character may repeat (except for apostrophe, which may be doubled). |
|---|---|
| %tok | A StringTokenizer object variable. |
Usage notes
- The TokenChars tokens are initially settable in the New call that creates the tokenizer instance.
- No character can be one of the TokenChars characters and also one of either the Quotes characters or the Spaces characters.
Examples
The statements in the sequence below tokenize the given equation:
%tok = new(tokenchars='()=+') %tok:string = 'y = (x + 1)(x - 1)' repeat while not %tok:atEnd printText {~} is {%tok:nextToken} end repeat
The result is:
%tok:nextToken is y %tok:nextToken is = %tok:nextToken is ( %tok:nextToken is x %tok:nextToken is + %tok:nextToken is 1 %tok:nextToken is ) %tok:nextToken is ( %tok:nextToken is x %tok:nextToken is - %tok:nextToken is 1 %tok:nextToken is )