StringTokenizer (String function): Difference between revisions

From m204wiki
Jump to navigation Jump to search
m (1 revision)
No edit summary
 
(6 intermediate revisions by 3 users not shown)
Line 1: Line 1:
{{Template:String:StringTokenizer subtitle}}
{{Template:String:StringTokenizer subtitle}}
 
This method returns a new instance of a <var>[[StringTokenizer class|StringTokenizer]] </var> object using the method string as the tokenizer string.
This method returns a new instance of a <var>StringTokenizer</var> object using the method string as the tokenizer string.
It has three optional arguments that let you specify the delimiter characters
It has three optional arguments that let you specify the delimiter characters
that determine the tokens in the string that is being tokenized.
that determine the tokens in the string that is being tokenized.
==Syntax==
==Syntax==
{{Template:String:StringTokenizer syntax}}
{{Template:String:StringTokenizer syntax}}
===Syntax terms===
===Syntax terms===
<table class="syntaxTable">
<table class="syntaxTable">
<tr><th>%stringTokenizer</th>
<tr><th nowrap>%stringTokenizer</th>
<td>A <var>StringTokenizer</var> object expression to contain the new object instance. </td></tr>
<td>A <var>StringTokenizer</var> object expression to contain the new object instance. </td></tr>


Line 15: Line 16:


<tr><th><var>TokenChars</var></th>
<tr><th><var>TokenChars</var></th>
<td>This name required string argument <var>TokenChars</var> is a set of single-character token-delimiters (delimiters that are also tokens) that may be separated by whitespace characters.
<td>This [[Notation conventions for methods#Named parameters|name required]] string argument is a set of single-character token-delimiters (delimiters that are also tokens) that may be separated by whitespace characters.
<var>TokenChars</var> is an optional argument that defaults to a null string. </td></tr>
<p>
<var>TokenChars</var> is an optional argument that defaults to a null string. </p></td></tr>
 
<tr><th><var>Spaces</var></th>
<tr><th><var>Spaces</var></th>
<td>This name required string argument <var>Spaces</var> is a set of "whitespace" characters, that is, characters that separate tokens. Each of these characters is a "non-token delimiter," a delimiter that is not itself a token.
<td>This name required string argument is a set of "whitespace" characters, that is, characters that separate tokens. Each of these characters is a "non-token delimiter," a delimiter that is not itself a token.
<p>
<var>Spaces</var> is an optional argument that defaults to a blank character. </p></td></tr>


<var>Spaces</var> is an optional argument that defaults to a blank character. </td></tr>
<tr><th><var>Quotes</var></th>
<tr><th><var>Quotes</var></th>
<td>This name required string argument <var>Quotes</var> is a set of quotation characters. The text between each disjoint pair of identical quotation characters (a "quoted region") is treated as a single token, and any delimiter characters (Quote, Space, or TokenChar) within a quoted region are treated as non-delimiters.
<td>This name required string argument is a set of quotation characters. The text between each disjoint pair of identical quotation characters (a "quoted region") is treated as a single token, and any delimiter characters (<var>Quotes</var>, <var>Spaces</var>, <var>Separators</var>, or <var>TokenChars</var>) within a quoted region are treated as non-delimiters.
 
<p>
<var>Quotes</var> is an optional argument that defaults to a null string.</td></tr>
<var>Quotes</var> is an optional argument that defaults to a null string.</p></td></tr>


<tr><th><var>Separators</var></th>
<td>This name required string argument is a set of single characters that separate tokens. Each of these characters is a "non-token delimiter," a delimiter that is not itself a token.
<p>
Multiple <var>Separators</var> characters do not compress to a single separator (like <var>Spaces</var> characters). </p>
<p>
<var>Separators</var> is an optional argument that defaults to a null string. It is available as of <var class="product">Sirius Mods</var> version 7.8.  </p></td></tr>
</table>
</table>


==Usage notes==
==Usage notes==
<ul>
<ul>
<li>A character may belong to at most one of the Spaces,
<li>Do not confuse the <var>StringTokenizer</var> <b>function</b>, which creates a new object, with the <var>[[String (StringTokenizer property)|String]]</var> property of the <var>StringTokenizer</var> <b>class</b>, which only replaces the string to be tokenized (and resets the object to start tokenizing at the beginning), preserving various tokenizing properties such as <var>[[Separators (StringTokenizer property)|Separators]]</var> and <var>[[Quotes (StringTokenizer property)|Quotes]]</var>.
Quotes, or TokenChars sets of characters.
 
<li>If you are specifying Spaces, Quotes, or TokenChars,
<li>A character may belong to at most one of the <var>Spaces</var>, <var>Quotes</var>, or <var>TokenChars</var> sets of characters.
each character in the string is a quotation
 
character &amp;mdash; that is, you may not separate characters &amp;mdash; and no character
<li>If you are specifying <var>Quotes</var>, <var>Spaces</var>, <var>Separators</var>, or <var>TokenChars</var> characters,
may repeat (except for apostrophe, which may be doubled).
each character in the string is a member of the defined set &mdash; that is, you may not separate characters &mdash; and no character may repeat (except for apostrophe, which may be doubled).
<li>A quoted region is not affected by the [[TokensToLower (StringTokenizer property)|TokensToLower]]
 
and [[TokensToUpper (StringTokenizer property)|TokensToUpper]] properties.
<li>A quoted region is not affected by the <var>[[TokensToLower (StringTokenizer property)|TokensToLower]]</var>
and <var>[[TokensToUpper (StringTokenizer property)|TokensToUpper]]</var> properties.
</ul>
</ul>
==Examples==
==Examples==


Line 58: Line 70:
%tok:nextToken is 'foo'
%tok:nextToken is 'foo'
%tok:nextToken is 'bar'
%tok:nextToken is 'bar'
</p>
</p>
==See also==
==See also==
[[New_(StringTokenizer_constructor)]]
<ul>
<li><var>StringTokenizer</var> <var>[[New (StringTokenizer_constructor)|New]]</var> constructor
</ul>
 
{{Template:String:StringTokenizer footer}}
{{Template:String:StringTokenizer footer}}

Latest revision as of 21:56, 3 September 2015

Create a tokenizer using the method object string (String class)

[Introduced in Sirius Mods 7.8]

This method returns a new instance of a StringTokenizer object using the method string as the tokenizer string. It has three optional arguments that let you specify the delimiter characters that determine the tokens in the string that is being tokenized.

Syntax

%stringTokenizer = string:StringTokenizer[( [TokenChars= string], - [Spaces= string], - [Quotes= string], - [Separators= string])]

Syntax terms

%stringTokenizer A StringTokenizer object expression to contain the new object instance.
string The string to be tokenized.
TokenChars This name required string argument is a set of single-character token-delimiters (delimiters that are also tokens) that may be separated by whitespace characters.

TokenChars is an optional argument that defaults to a null string.

Spaces This name required string argument is a set of "whitespace" characters, that is, characters that separate tokens. Each of these characters is a "non-token delimiter," a delimiter that is not itself a token.

Spaces is an optional argument that defaults to a blank character.

Quotes This name required string argument is a set of quotation characters. The text between each disjoint pair of identical quotation characters (a "quoted region") is treated as a single token, and any delimiter characters (Quotes, Spaces, Separators, or TokenChars) within a quoted region are treated as non-delimiters.

Quotes is an optional argument that defaults to a null string.

Separators This name required string argument is a set of single characters that separate tokens. Each of these characters is a "non-token delimiter," a delimiter that is not itself a token.

Multiple Separators characters do not compress to a single separator (like Spaces characters).

Separators is an optional argument that defaults to a null string. It is available as of Sirius Mods version 7.8.

Usage notes

  • Do not confuse the StringTokenizer function, which creates a new object, with the String property of the StringTokenizer class, which only replaces the string to be tokenized (and resets the object to start tokenizing at the beginning), preserving various tokenizing properties such as Separators and Quotes.
  • A character may belong to at most one of the Spaces, Quotes, or TokenChars sets of characters.
  • If you are specifying Quotes, Spaces, Separators, or TokenChars characters, each character in the string is a member of the defined set — that is, you may not separate characters — and no character may repeat (except for apostrophe, which may be doubled).
  • A quoted region is not affected by the TokensToLower and TokensToUpper properties.

Examples

begin %tok is object stringtokenizer %tok = 'foo bar':stringTokenizer printText {~} is '{%tok:string}' repeat while not %tok:atEnd printText {~} is '{%tok:nextToken}' end repeat end

The result is:

%tok:string is 'foo bar' %tok:nextToken is 'foo' %tok:nextToken is 'bar'

See also

  • StringTokenizer New constructor