TokenChars (StringTokenizer property)

From m204wiki
Revision as of 01:30, 6 July 2012 by JAL2 (talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Characters to be interpreted as tokens (StringTokenizer class)


This readWrite property returns or sets the characters that are recognized as token-delimiter characters, that is, as single-character delimiters that are also tokens themselves. As a delimiter, a token-delimiter character is a non-included boundary at the end or beginning of a token, unless the character is part of a quoted region (in which case it is treated as a non-delimiter character).

Syntax

%currentString = stringTokenizer:TokenChars stringTokenizer:TokenChars = newString

Syntax terms

%currentString A string variable to contain the returned value of the current token-delimiter character(s). Each character in the string is a token character.
stringTokenizer A StringTokenizer object.
newString The String value to assign to stringTokenizer's TokenChars property. Each character in the string is a token character — that is, you may not separate characters — and no character may repeat (except for apostrophe, which may be doubled).

Usage notes

  • The TokenChars tokens are initially settable in the New call that creates the tokenizer instance. The default value for a new tokenizer instance is a null.
  • No character can be one of the TokenChars characters and also one of the Spaces, Quotes, or Separators characters.

Examples

The statements in the sequence below tokenize the given equation:

%tok = new(tokenchars='()=+') %tok:string = 'y = (x + 1)(x - 1)' repeat while not %tok:atEnd printText {~} is {%tok:nextToken} end repeat

The result is:

%tok:nextToken is y %tok:nextToken is = %tok:nextToken is ( %tok:nextToken is x %tok:nextToken is + %tok:nextToken is 1 %tok:nextToken is ) %tok:nextToken is ( %tok:nextToken is x %tok:nextToken is - %tok:nextToken is 1 %tok:nextToken is )

See also