New (StringTokenizer constructor)
Create new StringTokenizer object (StringTokenizer class)
This method returns a new instance of a StringTokenizer object.
It has three optional arguments that let you specify the delimiter characters
that determine the tokens in the string that is being tokenized.
Syntax
%stringTokenizer = [%(StringTokenizer):]New[( [TokenChars= string], - [Spaces= string], - [Quotes= string], - [Separators= string])]
Syntax terms
| %tok | A StringTokenizer object variable to contain the new object instance. |
|---|---|
| TokenChars= chars | This name required string argument (TokenChars) is a set of single-character token-delimiters (delimiters that are also tokens) that may be separated by whitespace characters.
TokenChars is an optional argument that defaults to a null string. |
| Spaces= chars | This name required string argument (Spaces) is a set of "whitespace" characters, that is, characters that separate tokens. Each of these characters is a "non-token delimiter," a delimiter that is not itself a token.
Spaces is an optional argument that defaults to a blank character. |
| Quotes= chars | This name required string argument (Quotes) is a set of quotation characters. The text between each disjoint pair of identical quotation characters (a "quoted region") is treated as a single token, and any delimiter characters (Quote, Space, or TokenChar) within a quoted region are treated as non-delimiters.
Quotes is an optional argument that defaults to a null string. |
Usage notes
- A character may belong to at most one of the Spaces, Quotes, or TokenChars sets of characters.
- If you are specifying Spaces, Quotes, or TokenChars, each character in the string is a quotation character — that is, you may not separate characters — and no character may repeat (except for apostrophe, which may be doubled).
- A quoted region is not affected by the TokensToLower and TokensToUpper properties.
Examples
In the following request, a variety of delimiters is on display:
begin %tok is object stringtokenizer %tok = new(spaces=' ;', tokenchars='=_', quotes='"') %tok:string = '--=_alternative 0016B5A2CA2574DD_=; - Content-Type: text/plain; charset="US-ASCII"' repeat while not %tok:atEnd printText {~} is {%tok:nextToken} end repeat end
The result is:
%tok:nextToken is -- %tok:nextToken is = %tok:nextToken is _ %tok:nextToken is alternative %tok:nextToken is 0016B5A2CA2574DD %tok:nextToken is _ %tok:nextToken is = %tok:nextToken is Content-Type: %tok:nextToken is text/plain %tok:nextToken is charset %tok:nextToken is = %tok:nextToken is US-ASCII