Spaces (StringTokenizer property)

From m204wiki
Jump to navigation Jump to search

Characters to be interpreted as token-delimiting whitespace (StringTokenizer class)

This readWrite property returns or sets the characters that are recognized as "whitespace" characters, that is, as characters that separate tokens. Such a character is also called a "non-token delimiter," because it is a delimiter that is not itself a token.

You can specify "token delimiters," which are delimiters that are also tokens, using the TokenChars property or the New constructor TokenChars parameter.

One or multiple consecutive whitespace characters mark the end or beginning of a token, except when these spaces are part of a quoted region. In that case, the spaces become non-delimiters.


%currentString = stringTokenizer:Spaces stringTokenizer:Spaces = newString

Syntax terms

%currentString A string variable to contain the returned value of the current whitespace characters. Each character in the string is a space character.
stringTokenizer A StringTokenizer object expression.
newString The String value to assign stringTokenizer's Spaces property. Each character in the string is a space character — that is, you may not separate characters — and no character may repeat (except for apostrophe, which may be doubled).

Usage notes

  • The Spaces characters are initially settable in the New call that creates the tokenizer instance. The default value for a new tokenizer instance is a blank character.
  • No character can be one of the Spaces characters and also one of the Quotes, TokenChars, or Separators characters.


In the following example, a StringTokenizer object is instantiated with the default parameter settings for the New constructor. Then the input string's tokens are returned in sequence:

%tok is object stringtokenizer %tok = new %tok:string = 'Bad,F ?3mO ,7,{x Z}' repeat while not %tok:atEnd printText {~} is {%tok:nextToken} end repeat

The resulting tokens, which were delimited by the default space character (blank), are:

%tok:nextToken is Bad,F %tok:nextToken is ?3mO %tok:nextToken is ,7,{x %tok:nextToken is Z}

If instead the tokenizer object is instantiated with a comma as the space character
(%tok = new(Spaces=',')), the resulting tokens are:

%tok:nextToken is Bad %tok:nextToken is F ?3mO %tok:nextToken is 7 %tok:nextToken is {x Z}

See also