TokenChars (StringTokenizer property)

What are the token-delimiter characters?

TokenChars is a member of the StringTokenizer class.

This readWrite property returns or sets the characters that are recognized as token-delimiter characters, that is, as single-character delimiters that are also tokens themselves. As a delimiter, a token-delimiter character is a non-included boundary at the end or beginning of a token, unless the character is part of a quoted region (in which case it is treated as a non-delimiter character).

Syntax

  %string = %tok:TokenChars

  %tok:TokenChars = %string

Syntax terms

%string

A string variable to contain the returned value of the current token-delimiter character(s) or to be set as the new value(s). The default value for a new tokenizer instance is a null. If you are setting TokenChars, each character in the string is a token character — that is, you may not separate characters — and no character may repeat (except for apostrophe, which may be doubled).

%tok

A StringTokenizer object variable.

Usage Notes

• The TokenChars tokens are initially settable in the New call that creates the tokenizer instance.
• No character can be one of the TokenChars characters and also one of either the Quotes characters or the Spaces characters.

Examples

The statements in the sequence below tokenize the given equation:

    %tok = new(tokenchars='()=+')
    %tok:string = 'y = (x + 1)(x - 1)'
    repeat while not %tok:atEnd
       printText {~} is {%tok:nextToken}
    end repeat

The result is:

    %tok:nextToken is y
    %tok:nextToken is =
    %tok:nextToken is (
    %tok:nextToken is x
    %tok:nextToken is +
    %tok:nextToken is 1
    %tok:nextToken is )
    %tok:nextToken is (
    %tok:nextToken is x
    %tok:nextToken is -
    %tok:nextToken is 1
    %tok:nextToken is )