TokenChars (StringTokenizer property)

From m204wiki
Jump to navigation Jump to search

Characters to be interpreted as tokens (StringTokenizer class)

This readWrite property returns or sets the characters that are recognized as token-delimiter characters, that is, as single-character delimiters that are also tokens themselves. As a delimiter, a token-delimiter character is a non-included boundary at the end or beginning of a token, unless the character is part of a quoted region (in which case it is treated as a non-delimiter character).


%currentString = stringTokenizer:TokenChars stringTokenizer:TokenChars = newString

Syntax terms

%currentString A string variable to contain the returned value of the current token-delimiter character(s). Each character in the string is a token character.
stringTokenizer A StringTokenizer object.
newString The String value to assign to stringTokenizer's TokenChars property. Each character in the string is a token character — that is, you may not separate characters — and no character may repeat (except for apostrophe, which may be doubled).

Usage notes

  • The TokenChars tokens are initially settable in the New call that creates the tokenizer instance. The default value for a new tokenizer instance is a null.
  • No character can be one of the TokenChars characters and also one of the Spaces, Quotes, or Separators characters.


The statements in the sequence below tokenize the given equation:

%tok = new(tokenchars='()=+') %tok:string = 'y = (x + 1)(x - 1)' repeat while not %tok:atEnd printText {~} is {%tok:nextToken} end repeat

The result is:

%tok:nextToken is y %tok:nextToken is = %tok:nextToken is ( %tok:nextToken is x %tok:nextToken is + %tok:nextToken is 1 %tok:nextToken is ) %tok:nextToken is ( %tok:nextToken is x %tok:nextToken is - %tok:nextToken is 1 %tok:nextToken is )

See also