TokenChars (StringTokenizer property) - Revision history

JAL at 20:58, 15 November 2012

2012-11-15T20:58:07Z

← Older revision		Revision as of 20:58, 15 November 2012
Line 1:		Line 1:
	{{Template:StringTokenizer:TokenChars subtitle}}		{{Template:StringTokenizer:TokenChars subtitle}}

	This readWrite property returns or sets the characters that are recognized as		This [[Classes and Objects#readWrite\|readWrite]] property returns or sets the characters that are recognized as
	token-delimiter characters, that is, as single-character delimiters that are		token-delimiter characters, that is, as single-character delimiters that are also tokens themselves.
	also tokens themselves.
	As a delimiter, a token-delimiter character is a non-included boundary at the		As a delimiter, a token-delimiter character is a non-included boundary at the
	end or beginning of a token, unless the character is part of a quoted region		end or beginning of a token, unless the character is part of a quoted region
Line 27:		Line 26:
	<li>The <var>TokenChars</var> tokens are initially settable in the		<li>The <var>TokenChars</var> tokens are initially settable in the
	<var>[[New (StringTokenizer constructor)\|New]]</var> call that creates the tokenizer instance. The default value for a new tokenizer instance is a null.		<var>[[New (StringTokenizer constructor)\|New]]</var> call that creates the tokenizer instance. The default value for a new tokenizer instance is a null.

	<li>No character can be one of the <var>TokenChars</var> characters and also		<li>No character can be one of the <var>TokenChars</var> characters and also
	one of the <var>[[Spaces (StringTokenizer property)\|Spaces]]</var>, <var>[[Quotes (StringTokenizer property)\|Quotes]]</var>, or <var>[[Separators (StringTokenizer property)\|Separators]]</var> characters.		one of the <var>[[Spaces (StringTokenizer property)\|Spaces]]</var>, <var>[[Quotes (StringTokenizer property)\|Quotes]]</var>, or <var>[[Separators (StringTokenizer property)\|Separators]]</var> characters.
Line 36:		Line 36:
	%tok:string = 'y = (x + 1)(x - 1)'		%tok:string = 'y = (x + 1)(x - 1)'
	repeat while not %tok:atEnd		repeat while not %tok:atEnd
	printText {~} is {%tok:nextToken}		[[Targeted Text statements#AuditText, PrintText, and TraceText\|printText]] {~} is {%tok:nextToken}
	end repeat		end repeat
	</p>		</p>

JAL2 at 01:30, 6 July 2012

2012-07-06T01:30:26Z

New page

{{Template:StringTokenizer:TokenChars subtitle}}

This readWrite property returns or sets the characters that are recognized as
token-delimiter characters, that is, as single-character delimiters that are
also tokens themselves.
As a delimiter, a token-delimiter character is a non-included boundary at the
end or beginning of a token, unless the character is part of a quoted region
(in which case it is treated as a non-delimiter character).

==Syntax==
{{Template:StringTokenizer:TokenChars syntax}}

===Syntax terms===
<table class="syntaxTable">
<tr><th>%currentString</th>
<td>A string variable to contain the returned value of the current token-delimiter character(s). Each character in the string is a token character. </td></tr>

<tr><th>stringTokenizer</th>
<td>A <var>StringTokenizer</var> object.</td></tr>

<tr><th>newString</th>
<td>The <var>String</var> value to assign to <var class="term">stringTokenizer</var>'s <var>TokenChars</var> property. Each character in the string is a token character — that is, you may not separate characters — and no character may repeat (except for apostrophe, which may be doubled).</td></tr>
</table>

==Usage notes==
<ul>
<li>The <var>TokenChars</var> tokens are initially settable in the
<var>[[New (StringTokenizer constructor)|New]]</var> call that creates the tokenizer instance. The default value for a new tokenizer instance is a null.
<li>No character can be one of the <var>TokenChars</var> characters and also
one of the <var>[[Spaces (StringTokenizer property)|Spaces]]</var>, <var>[[Quotes (StringTokenizer property)|Quotes]]</var>, or <var>[[Separators (StringTokenizer property)|Separators]]</var> characters.
</ul>

==Examples==
The statements in the sequence below tokenize the given equation:
<p class="code">%tok = new(tokenchars='()=+')
%tok:string = 'y = (x + 1)(x - 1)'
repeat while not %tok:atEnd
printText {~} is {%tok:nextToken}
end repeat
</p>

The result is:
<p class="output">%tok:nextToken is y
%tok:nextToken is =
%tok:nextToken is (
%tok:nextToken is x
%tok:nextToken is +
%tok:nextToken is 1
%tok:nextToken is )
%tok:nextToken is (
%tok:nextToken is x
%tok:nextToken is -
%tok:nextToken is 1
%tok:nextToken is )
</p>
==See also==
{{Template:StringTokenizer:TokenChars footer}}

Wiccan: 1 revision

2011-02-08T18:16:13Z

1 revision

← Older revision	Revision as of 18:16, 8 February 2011
(No difference)

Wiccan: 1 revision

2011-02-08T18:16:11Z

1 revision

← Older revision		Revision as of 18:16, 8 February 2011
Line 14:		Line 14:
	<tr><th>%string</th>		<tr><th>%string</th>
	<td>A string variable to contain the returned value of the current token-delimiter character(s) or to be set as the new value(s). The default value for a new tokenizer instance is a null.		<td>A string variable to contain the returned value of the current token-delimiter character(s) or to be set as the new value(s). The default value for a new tokenizer instance is a null.
	~~<p class="code">~~If you are setting <var>TokenChars</var>, each character in the string is a token character — that is, you may not separate characters — and no character may repeat (except for apostrophe, which may be doubled). </td></tr>
	~~</p~~>		If you are setting <var>TokenChars</var>, each character in the string is a token character — that is, you may not separate characters — and no character may repeat (except for apostrophe, which may be doubled). </td></tr>
	<tr><th>%tok</th>		<tr><th>%tok</th>
	<td>A <var>StringTokenizer</var> object variable.</td></tr>		<td>A <var>StringTokenizer</var> object variable.</td></tr>

Wiccan: 1 revision

2011-02-06T21:48:06Z

1 revision

← Older revision	Revision as of 21:48, 6 February 2011
(No difference)

Wiccan: 1 revision

2011-02-06T21:48:03Z

1 revision

← Older revision		Revision as of 21:48, 6 February 2011
Line 14:		Line 14:
	<tr><th>%string</th>		<tr><th>%string</th>
	<td>A string variable to contain the returned value of the current token-delimiter character(s) or to be set as the new value(s). The default value for a new tokenizer instance is a null.		<td>A string variable to contain the returned value of the current token-delimiter character(s) or to be set as the new value(s). The default value for a new tokenizer instance is a null.
	If you are setting <var>TokenChars</var>, each character in the string is a token character — that is, you may not separate characters — and no character may repeat (except for apostrophe, which may be doubled). </td></tr>		<p class="code">If you are setting <var>TokenChars</var>, each character in the string is a token character — that is, you may not separate characters — and no character may repeat (except for apostrophe, which may be doubled). </td></tr>
			</p>
	<tr><th>%tok</th>		<tr><th>%tok</th>
	<td>A <var>StringTokenizer</var> object variable.</td></tr>		<td>A <var>StringTokenizer</var> object variable.</td></tr>
Line 28:		Line 29:

	The statements in the sequence below tokenize the given equation:		The statements in the sequence below tokenize the given equation:
	<~~pre~~>		<p class="code">%tok = new(tokenchars='()=+')
	%tok = new(tokenchars='()=+')		%tok:string = 'y = (x + 1)(x - 1)'
	%tok:string = 'y = (x + 1)(x - 1)'		repeat while not %tok:atEnd
	repeat while not %tok:atEnd		printText {~} is {%tok:nextToken}
	printText {~} is {%tok:nextToken}		end repeat
	end repeat		</p>
	</~~pre~~>

	The result is:		The result is:
	<~~pre~~>		<p class="output">%tok:nextToken is y
	%tok:nextToken is y		%tok:nextToken is =
	%tok:nextToken is =		%tok:nextToken is (
	%tok:nextToken is (		%tok:nextToken is x
	%tok:nextToken is x		%tok:nextToken is +
	%tok:nextToken is +		%tok:nextToken is 1
	%tok:nextToken is 1		%tok:nextToken is )
	%tok:nextToken is )		%tok:nextToken is (
	%tok:nextToken is (		%tok:nextToken is x
	%tok:nextToken is x		%tok:nextToken is -
	%tok:nextToken is -		%tok:nextToken is 1
	%tok:nextToken is 1		%tok:nextToken is )
	%tok:nextToken is )		</p>
	</~~pre~~>
	==See also==		==See also==
	{{Template:StringTokenizer:TokenChars footer}}		{{Template:StringTokenizer:TokenChars footer}}

Wiccan: 1 revision

2011-02-06T21:47:23Z

1 revision

← Older revision	Revision as of 21:47, 6 February 2011
(No difference)

Wiccan: 1 revision

2011-02-06T21:47:18Z

1 revision

← Older revision		Revision as of 21:47, 6 February 2011
Line 14:		Line 14:
	<tr><th>%string</th>		<tr><th>%string</th>
	<td>A string variable to contain the returned value of the current token-delimiter character(s) or to be set as the new value(s). The default value for a new tokenizer instance is a null.		<td>A string variable to contain the returned value of the current token-delimiter character(s) or to be set as the new value(s). The default value for a new tokenizer instance is a null.
	If you are setting TokenChars, each character in the string is a token character — that is, you may not separate characters — and no character may repeat (except for apostrophe, which may be doubled). </td></tr>		If you are setting <var>TokenChars</var>, each character in the string is a token character — that is, you may not separate characters — and no character may repeat (except for apostrophe, which may be doubled). </td></tr>
	<tr><th>%tok</th>		<tr><th>%tok</th>
	<td>A StringTokenizer object variable.</td></tr>		<td>A <var>StringTokenizer</var> object variable.</td></tr>
	</table>		</table>
	==Usage notes==		==Usage notes==
	<ul>		<ul>
	<li>The TokenChars tokens are initially settable in the		<li>The <var>TokenChars</var> tokens are initially settable in the
	[[New (StringTokenizer constructor)\|New]] call that creates the tokenizer instance.		[[New (StringTokenizer constructor)\|New]] call that creates the tokenizer instance.
	<li>No character can be one of the TokenChars characters and also		<li>No character can be one of the <var>TokenChars</var> characters and also
	one of either the Quotes characters or the Spaces characters.		one of either the Quotes characters or the Spaces characters.
	</ul>		</ul>

Wiccan: 1 revision

2011-02-06T19:55:21Z

1 revision

← Older revision	Revision as of 19:55, 6 February 2011
(No difference)

Wiccan: 1 revision

2011-02-06T19:55:19Z

1 revision

← Older revision		Revision as of 19:55, 6 February 2011
Line 12:		Line 12:
	===Syntax terms===		===Syntax terms===
	<table class="syntaxTable">		<table class="syntaxTable">
	<tr><th~~><i~~>%string~~</i>~~</th>		<tr><th>%string</th>
	<td>A string variable to contain the returned value of the current token-delimiter character(s) or to be set as the new value(s). The default value for a new tokenizer instance is a null.		<td>A string variable to contain the returned value of the current token-delimiter character(s) or to be set as the new value(s). The default value for a new tokenizer instance is a null.
	If you are setting TokenChars, each character in the string is a token character — that is, you may not separate characters — and no character may repeat (except for apostrophe, which may be doubled). </td></tr>		If you are setting TokenChars, each character in the string is a token character — that is, you may not separate characters — and no character may repeat (except for apostrophe, which may be doubled). </td></tr>
	<tr><th~~><i~~>%tok~~</i>~~</th>		<tr><th>%tok</th>
	<td>A StringTokenizer object variable.</td></tr>		<td>A StringTokenizer object variable.</td></tr>
	</table>		</table>