TokenChars (StringTokenizer property)
Characters to be interpreted as tokens (StringTokenizer class)
This readWrite property returns or sets the characters that are recognized as
token-delimiter characters, that is, as single-character delimiters that are also tokens themselves.
As a delimiter, a token-delimiter character is a non-included boundary at the
end or beginning of a token, unless the character is part of a quoted region
(in which case it is treated as a non-delimiter character).
Syntax
%currentString = stringTokenizer:TokenChars stringTokenizer:TokenChars = newString
Syntax terms
| %currentString | A string variable to contain the returned value of the current token-delimiter character(s). Each character in the string is a token character. |
|---|---|
| stringTokenizer | A StringTokenizer object. |
| newString | The String value to assign to stringTokenizer's TokenChars property. Each character in the string is a token character — that is, you may not separate characters — and no character may repeat (except for apostrophe, which may be doubled). |
Usage notes
- The TokenChars tokens are initially settable in the New call that creates the tokenizer instance. The default value for a new tokenizer instance is a null.
- No character can be one of the TokenChars characters and also one of the Spaces, Quotes, or Separators characters.
Examples
The statements in the sequence below tokenize the given equation:
%tok = new(tokenchars='()=+') %tok:string = 'y = (x + 1)(x - 1)' repeat while not %tok:atEnd printText {~} is {%tok:nextToken} end repeat
The result is:
%tok:nextToken is y %tok:nextToken is = %tok:nextToken is ( %tok:nextToken is x %tok:nextToken is + %tok:nextToken is 1 %tok:nextToken is ) %tok:nextToken is ( %tok:nextToken is x %tok:nextToken is - %tok:nextToken is 1 %tok:nextToken is )