StringTokenizer (String function)

Create a tokenizer using the method object string (String class)

[Introduced in Sirius Mods 7.8]

This method returns a new instance of a StringTokenizer object using the method string as the tokenizer string. It has three optional arguments that let you specify the delimiter characters that determine the tokens in the string that is being tokenized.

Syntax

%stringTokenizer = string:StringTokenizer[( [TokenChars= string], - [Spaces= string], - [Quotes= string], - [Separators= string])]

Syntax terms

%stringTokenizer	A StringTokenizer object expression to contain the new object instance.
string	The string to be tokenized.
TokenChars	This name required string argument TokenChars is a set of single-character token-delimiters (delimiters that are also tokens) that may be separated by whitespace characters. TokenChars is an optional argument that defaults to a null string.
Spaces	This name required string argument Spaces is a set of "whitespace" characters, that is, characters that separate tokens. Each of these characters is a "non-token delimiter," a delimiter that is not itself a token. Spaces is an optional argument that defaults to a blank character.
Quotes	This name required string argument Quotes is a set of quotation characters. The text between each disjoint pair of identical quotation characters (a "quoted region") is treated as a single token, and any delimiter characters (Quote, Space, or TokenChar) within a quoted region are treated as non-delimiters. Quotes is an optional argument that defaults to a null string.

Usage notes

• A character may belong to at most one of the Spaces, Quotes, or TokenChars sets of characters.
• If you are specifying Spaces, Quotes, or TokenChars, each character in the string is a quotation character — that is, you may not separate characters — and no character may repeat (except for apostrophe, which may be doubled).
• A quoted region is not affected by the TokensToLower and TokensToUpper properties.

Examples

begin %tok is object stringtokenizer %tok = 'foo bar':stringTokenizer printText {~} is '{%tok:string}' repeat while not %tok:atEnd printText {~} is '{%tok:nextToken}' end repeat end

The result is:

%tok:string is 'foo bar' %tok:nextToken is 'foo' %tok:nextToken is 'bar'

See also

New_(StringTokenizer_constructor)

Float class String class Unicode class	List of Float methods List of String methods List of Unicode methods List of Intrinsic methods	Float methods syntax String methods syntax Unicode methods syntax
Notation conventions for methods