New (StringTokenizer constructor)
Create new StringTokenizer object (StringTokenizer class)
This method returns a new instance of a StringTokenizer object.
It has three optional arguments that let you specify the delimiter characters
that determine the tokens in the string that is being tokenized.
Syntax
%stringTokenizer = [%(StringTokenizer):]New[( [TokenChars= string], - [Spaces= string], - [Quotes= string], - [Separators= string])]
Syntax terms
| %tok | A StringTokenizer object variable to contain the new object instance. |
|---|---|
| TokenChars= chars | This name required string argument (TokenChars) is a set of single-character token-delimiters (delimiters that are also tokens) that may be separated by whitespace characters. TokenChars is an optional argument that defaults to a null string. |
| Spaces= chars | This name required string argument (Spaces) is a set of "whitespace" characters, that is, characters that separate tokens. Each of these characters is a "non-token delimiter," a delimiter that is not itself a token. Spaces is an optional argument that defaults to a blank character. |
| Quotes= chars | This name required string argument (Quotes) is a set of quotation characters. The text between each disjoint pair of identical quotation characters (a "quoted region") is treated as a single token, and any delimiter characters (Quote, Space, or TokenChar) within a quoted region are treated as non-delimiters. Quotes is an optional argument that defaults to a null string. |
Usage notes
- A character may belong to at most one of the Spaces, Quotes, or TokenChars sets of characters.
- If you are specifying Spaces, Quotes, or TokenChars, each character in the string is a quotation character — that is, you may not separate characters — and no character may repeat (except for apostrophe, which may be doubled).
- A quoted region is not affected by the TokensToLower and TokensToUpper properties.
Examples
In the following request, a variety of delimiters is on display:
begin
%tok is object stringtokenizer
%tok = new(spaces=' ;', tokenchars='=_', quotes='"')
%tok:string = '--=_alternative 0016B5A2CA2574DD_=; -
Content-Type: text/plain; charset="US-ASCII"'
repeat while not %tok:atEnd
printText {~} is {%tok:nextToken}
end repeat
end
The result is:
%tok:nextToken is --
%tok:nextToken is =
%tok:nextToken is _
%tok:nextToken is alternative
%tok:nextToken is 0016B5A2CA2574DD
%tok:nextToken is _
%tok:nextToken is =
%tok:nextToken is Content-Type:
%tok:nextToken is text/plain
%tok:nextToken is charset
%tok:nextToken is =
%tok:nextToken is US-ASCII