Spaces (StringTokenizer property)

Characters which are the token-delimiting "whitespace" charact

Spaces is a member of the StringTokenizer class.

This readWrite property returns or sets the characters that are recognized as “whitespace”characters, that is, as characters that separate tokens. Such a character is also called a “non-token delimiter,” because it is a delimiter that is not itself a token.

You can specify “token delimiters,” which are delimiters that are also tokens, using the TokenChars property or the New constructor TokenChars parameter.

One or multiple consecutive whitespace characters mark the end or beginning of a token, except when these spaces are part of a quoted region. In that case, the spaces become non-delimiters.

Syntax

  %string = %tok:Spaces

  %tok:Spaces = %string

Syntax terms

%string

A string variable to contain the returned value of the current whitespace characters or to be set as the new value(s). The default value for a new tokenizer instance is a blank character. If you are setting Spaces, each character in the string is a space character — that is, you may not separate characters — and no character may repeat (except for apostrophe, which may be doubled).

%tok

A StringTokenizer object variable.

Usage Notes

• The Spaces characters are initially settable in the New call that creates the tokenizer instance.
• No character can be one of the Spaces characters and also one of either the Quotes characters or the TokenChars characters.

Examples

In the following example, a Stringtokenizer object is instantiated with the default parameter settings for the New constructor. Then the input string's tokens are returned in sequence:

    %tok is object stringtokenizer
    %tok = new
    %tok:string = 'Bad,F  ?3mO ,7,{x Z}'
    repeat while not %tok:atEnd
       printText {~} is {%tok:nextToken}
    end repeat

The resulting tokens, which were delimited by the default space character (blank), are:

    %tok:nextToken is Bad,F
    %tok:nextToken is ?3mO
    %tok:nextToken is ,7,{x
    %tok:nextToken is Z}

If instead the tokenizer object is instantiated with a comma as the space character
(%tok = new(Spaces=',')), the resulting tokens are:

    %tok:nextToken is Bad
    %tok:nextToken is F  ?3mO
    %tok:nextToken is 7
    %tok:nextToken is {x Z}