New (StringTokenizer constructor)

From m204wiki
Revision as of 21:48, 6 February 2011 by Wiccan (talk | contribs) (1 revision)
Jump to navigation Jump to search

Create new StringTokenizer object (StringTokenizer class)


This method returns a new instance of a StringTokenizer object. It has three optional arguments that let you specify the delimiter characters that determine the tokens in the string that is being tokenized.

Syntax

%stringTokenizer = [%(StringTokenizer):]New[( [TokenChars= string], - [Spaces= string], - [Quotes= string], - [Separators= string])]

Syntax terms

%tok A StringTokenizer object variable to contain the new object instance.
TokenChars= chars This name required string argument (TokenChars) is a set of single-character token-delimiters (delimiters that are also tokens) that may be separated by whitespace characters.

TokenChars is an optional argument that defaults to a null string.

Spaces= chars This name required string argument (Spaces) is a set of "whitespace" characters, that is, characters that separate tokens. Each of these characters is a "non-token delimiter," a delimiter that is not itself a token.

Spaces is an optional argument that defaults to a blank character.

Quotes= chars This name required string argument (Quotes) is a set of quotation characters. The text between each disjoint pair of identical quotation characters (a "quoted region") is treated as a single token, and any delimiter characters (Quote, Space, or TokenChar) within a quoted region are treated as non-delimiters.

Quotes is an optional argument that defaults to a null string.

Usage notes

  • A character may belong to at most one of the Spaces, Quotes, or TokenChars sets of characters.
  • If you are specifying Spaces, Quotes, or TokenChars, each character in the string is a quotation character — that is, you may not separate characters — and no character may repeat (except for apostrophe, which may be doubled).
  • A quoted region is not affected by the TokensToLower and TokensToUpper properties.

Examples

In the following request, a variety of delimiters is on display:

begin %tok is object stringtokenizer %tok = new(spaces=' ;', tokenchars='=_', quotes='"') %tok:string = '--=_alternative 0016B5A2CA2574DD_=; - Content-Type: text/plain; charset="US-ASCII"' repeat while not %tok:atEnd printText {~} is {%tok:nextToken} end repeat end

The result is:

%tok:nextToken is -- %tok:nextToken is = %tok:nextToken is _ %tok:nextToken is alternative %tok:nextToken is 0016B5A2CA2574DD %tok:nextToken is _ %tok:nextToken is = %tok:nextToken is Content-Type: %tok:nextToken is text/plain %tok:nextToken is charset %tok:nextToken is = %tok:nextToken is US-ASCII

See also