NextToken (StringTokenizer function): Difference between revisions
Jump to navigation
Jump to search
mNo edit summary |
mNo edit summary |
||
| Line 4: | Line 4: | ||
the next token forward in the current tokenizer string. | the next token forward in the current tokenizer string. | ||
The next token parsing begins from the tokenizing position, | The next token parsing begins from the tokenizing position, | ||
which is the value returned by [[NextPosition (StringTokenizer property)|NextPosition]]. | which is the value returned by <var>[[NextPosition (StringTokenizer property)|NextPosition]]</var>. | ||
When the next token is determined, | When the next token is determined, | ||
| Line 11: | Line 11: | ||
The delimiters that are involved in determining the tokens | The delimiters that are involved in determining the tokens | ||
are set initially by the [[New (StringTokenizer constructor)|New]] method | are set initially by the <var>[[New (StringTokenizer constructor)|New]]</var> method | ||
that instantiates the <var>StringTokenizer</var> object. | that instantiates the <var>StringTokenizer</var> object. | ||
The delimiters can be modified by the [[Spaces (StringTokenizer property)|Spaces]], | The delimiters can be modified by the <var>[[Spaces (StringTokenizer property)|Spaces]]</var>, | ||
[[Quotes (StringTokenizer property)|Quotes]], and [[TokenChars (StringTokenizer property)|TokenChars]] properties. | <var>[[Quotes (StringTokenizer property)|Quotes]]</var>, and <var>[[TokenChars (StringTokenizer property)|TokenChars]]</var> properties. | ||
==Syntax== | ==Syntax== | ||
{{Template:StringTokenizer:NextToken syntax}} | {{Template:StringTokenizer:NextToken syntax}} | ||
| Line 26: | Line 26: | ||
<td>This name required numeric argument specifies a number of tokens to skip before returning the next token. The value of <var class="term">number</var> must be greater than 0, and it must be be less than the number of tokens remaining in the tokenizer string. See the [[#skipEx|example]] below. | <td>This name required numeric argument specifies a number of tokens to skip before returning the next token. The value of <var class="term">number</var> must be greater than 0, and it must be be less than the number of tokens remaining in the tokenizer string. See the [[#skipEx|example]] below. | ||
The <var>Skip</var> | The <var>Skip</var> argument is available as of version 7.7 of the <var class="product">Sirius Mods</var>.</td></tr> | ||
</table> | </table> | ||
| Line 32: | Line 32: | ||
==Usage notes== | ==Usage notes== | ||
<ul> | <ul> | ||
<li>If the value of [[AtEnd (StringTokenizer function)|AtEnd]] is < | <li>If the value of <var>[[AtEnd (StringTokenizer function)|AtEnd]]</var> is <code>True</code>, | ||
issuing <var>NextToken</var> is invalid and cancels the request. | issuing <var>NextToken</var> is invalid and cancels the request. | ||
<li>For a new <var>StringTokenizer</var> instance, issuing <var>NextToken</var>, | <li>For a new <var>StringTokenizer</var> instance, issuing <var>NextToken</var>, | ||
[[FindToken (StringTokenizer function)|FindToken]], or [[SkipTokens (StringTokenizer subroutine)|SkipTokens]] is | <var>[[FindToken (StringTokenizer function)|FindToken]]</var>, or <var>[[SkipTokens (StringTokenizer subroutine)|SkipTokens]]</var> is | ||
required before [[CurrentToken (StringTokenizer function)|CurrentToken]] may be issued without error. | required before <var>[[CurrentToken (StringTokenizer function)|CurrentToken]]</var> may be issued without error. | ||
<li> | <li><var>[[NextPosition (StringTokenizer property)|NextPosition]]</var> returns the position of | ||
the character after the <var>NextToken</var> value. | the character after the <var>NextToken</var> value. | ||
Conversely, the determination of the <var>NextToken</var> value always starts at the | Conversely, the determination of the <var>NextToken</var> value always starts at the | ||
position given by NextPosition. | position given by <var>NextPosition</var>. | ||
If you reset NextPosition, the value returned by your next <var>NextToken</var> call | If you reset <var>NextPosition</var>, the value returned by your next <var>NextToken</var> call | ||
will be based on the new NextPosition value. | will be based on the new <var>NextPosition</var> value. | ||
<li>The FindToken and [[PeekToken (StringTokenizer function)|PeekToken]] | <li>The <var>FindToken</var> and <var>[[PeekToken (StringTokenizer function)|PeekToken]]</var> | ||
functions also return the next token. | functions also return the next token. | ||
Like <var>NextToken</var>, FindToken | Like <var>NextToken</var>, <var>FindToken</var> | ||
advances the tokenizing position to the position following that token. | advances the tokenizing position to the position following that token. | ||
PeekToken, however, does not advance the tokenizing position. | <var>PeekToken</var>, however, does not advance the tokenizing position. | ||
</ul> | </ul> | ||
==Examples== | ==Examples== | ||
<ol> | <ol> | ||
<li>In the following request fragment, the printText statements display, | <li>In the following request fragment, the printText statements display, | ||
respectively, < | respectively, <code>example</code>, <code>of</code>, and <code>nextToken</code>. | ||
<p class="code">%tok is object stringTokenizer | <p class="code">%tok is object stringTokenizer | ||
%tok = new | %tok = new | ||
| Line 62: | Line 62: | ||
<br> | <br> | ||
<div id="skipEx"></div> | <div id="skipEx"></div> | ||
<li>The following request fragment shows the effect of the Skip | <li>The following request fragment shows the effect of the <var>Skip</var> argument: | ||
<p class="code">%tok:string = ' * 0 * 1 * 2 * 3 * 4 * 5 * 6 * 7' | <p class="code">%tok:string = ' * 0 * 1 * 2 * 3 * 4 * 5 * 6 * 7' | ||
repeat while %tok:notAtEnd | repeat while %tok:notAtEnd | ||
Revision as of 17:02, 8 February 2012
Next token forward in current tokenizer string (StringTokenizer class)
This method returns the string value of
the next token forward in the current tokenizer string.
The next token parsing begins from the tokenizing position,
which is the value returned by NextPosition.
When the next token is determined, NextToken advances the tokenizing position to the position following that token, then returns the token value.
The delimiters that are involved in determining the tokens are set initially by the New method that instantiates the StringTokenizer object. The delimiters can be modified by the Spaces, Quotes, and TokenChars properties.
Syntax
[%string =] stringTokenizer:NextToken[( [Skip= number], [Default= string])] Throws MismatchedQuote, OutOfBounds
Syntax terms
| %string | A string variable to receive the value of the current token. |
|---|---|
| stringTokenizer | A StringTokenizer object expression. |
| Skip | This name required numeric argument specifies a number of tokens to skip before returning the next token. The value of number must be greater than 0, and it must be be less than the number of tokens remaining in the tokenizer string. See the example below. The Skip argument is available as of version 7.7 of the Sirius Mods. |
Usage notes
- If the value of AtEnd is
True, issuing NextToken is invalid and cancels the request. - For a new StringTokenizer instance, issuing NextToken, FindToken, or SkipTokens is required before CurrentToken may be issued without error.
- NextPosition returns the position of the character after the NextToken value. Conversely, the determination of the NextToken value always starts at the position given by NextPosition. If you reset NextPosition, the value returned by your next NextToken call will be based on the new NextPosition value.
- The FindToken and PeekToken functions also return the next token. Like NextToken, FindToken advances the tokenizing position to the position following that token. PeekToken, however, does not advance the tokenizing position.
Examples
- In the following request fragment, the printText statements display,
respectively,
example,of, andnextToken.%tok is object stringTokenizer %tok = new %tok:string = 'example of nextToken' printText {%tok:nextToken} printText {%tok:nextToken} printText {%tok:nextToken}
- The following request fragment shows the effect of the Skip argument:
%tok:string = ' * 0 * 1 * 2 * 3 * 4 * 5 * 6 * 7' repeat while %tok:notAtEnd printtext {~} = '{%tok:nextToken(skip=1)}' end repeat
The result is:
%tok:nextToken(skip=1) = '0' %tok:nextToken(skip=1) = '1' %tok:nextToken(skip=1) = '2' %tok:nextToken(skip=1) = '3' %tok:nextToken(skip=1) = '4' %tok:nextToken(skip=1) = '5' %tok:nextToken(skip=1) = '6' %tok:nextToken(skip=1) = '7'