NextToken (StringTokenizer function)

Next token forward in current tokenizer string (StringTokenizer class)

This method returns the string value of the next token forward in the current tokenizer string. The next token parsing begins from the tokenizing position, which is the value returned by NextPosition.

When the next token is determined, NextToken advances the tokenizing position to the position following that token, then returns the token value.

The delimiters that are involved in determining the tokens are set initially by the New method that instantiates the StringTokenizer object. The delimiters can be modified by the Spaces, Quotes, and TokenChars properties.

Syntax

[%string =] stringTokenizer:NextToken[( [Skip= number], [Default= string])] Throws MismatchedQuote, OutOfBounds

Syntax terms

%string	A string variable to receive the value of the current token.
stringTokenizer	A StringTokenizer object expression.
Skip	This name required numeric argument specifies a number of tokens to skip before returning the next token. The value of number must be greater than 0, and it must be be less than the number of tokens remaining in the tokenizer string. See the example below. The Skip argument is available as of version 7.7 of the Sirius Mods.

Usage notes

• If the value of AtEnd is True, issuing NextToken is invalid and cancels the request.
• For a new StringTokenizer instance, issuing NextToken, FindToken, or SkipTokens is required before CurrentToken may be issued without error.
• NextPosition returns the position of the character after the NextToken value. Conversely, the determination of the NextToken value always starts at the position given by NextPosition. If you reset NextPosition, the value returned by your next NextToken call will be based on the new NextPosition value.
• The FindToken and PeekToken functions also return the next token. Like NextToken, FindToken advances the tokenizing position to the position following that token. PeekToken, however, does not advance the tokenizing position.

Examples

1. In the following request fragment, the printText statements display, respectively, example, of, and nextToken.

   %tok is object stringTokenizer %tok = new %tok:string = 'example of nextToken' PrintText {%tok:nextToken} printText {%tok:nextToken} printText {%tok:nextToken}

   
   
2. The following request fragment shows the effect of the Skip argument:

   %tok:string = ' * 0 * 1 * 2 * 3 * 4 * 5 * 6 * 7' repeat while %tok:notAtEnd printtext {~} = '{%tok:nextToken(skip=1)}' end repeat

   The result is:

   %tok:nextToken(skip=1) = '0' %tok:nextToken(skip=1) = '1' %tok:nextToken(skip=1) = '2' %tok:nextToken(skip=1) = '3' %tok:nextToken(skip=1) = '4' %tok:nextToken(skip=1) = '5' %tok:nextToken(skip=1) = '6' %tok:nextToken(skip=1) = '7'

See also

StringTokenizer class	List of StringTokenizer methods	StringTokenizer methods syntax
Notation conventions for methods