NextToken (StringTokenizer function)

Next token forward in current tokenizer string

NextToken is a member of the StringTokenizer class.

This method returns the string value of the next token forward in the current tokenizer string. The next token parsing begins from the tokenizing position, which is the value returned by NextPosition.

When the next token is determined, NextToken advances the tokenizing position to the position following that token, then returns the token value.

The delimiters that are involved in determining the tokens are set initially by the New method that instantiates the StringTokenizer object. The delimiters can be modified by the Spaces, Quotes, and TokenChars properties.

Syntax

  %string = %tok:NextToken([Skip=num])

Syntax terms

%string

A string variable to receive the value of the current token.

%tok

A StringTokenizer object variable.

Skip= num

This name required numeric argument specifies a number of tokens to skip before returning the next token. The value of num must be greater than 0, and it must be be less than the number of tokens remaining in the tokenizer string. For an example, see item ?? refid=skipxmp.. The Skip parameter is available as of version 7.7 of the Sirius Mods.

Usage Notes

• If the value of AtEnd is True, issuing NextToken is invalid and cancels the request.
• For a new StringTokenizer instance, issuing NextToken, FindToken, or SkipTokens is required before CurrentToken may be issued without error.
• The NextPosition NextPosition returns the position of the character after the NextToken value. Conversely, the determination of the NextToken value always starts at the position given by NextPosition. If you reset NextPosition, the value returned by your next NextToken call will be based on the new NextPosition value.
• The FindToken and PeekToken functions also return the next token. Like NextToken, FindToken advances the tokenizing position to the position following that token. PeekToken, however, does not advance the tokenizing position.

Examples

1. In the following request fragment, the printText statements display, respectively, example, of, and nextToken.

       %tok is object stringTokenizer
       %tok = new
       %tok:string = 'example of nextToken'
       printText {%tok:nextToken}
       printText {%tok:nextToken}
       printText {%tok:nextToken}
   

   
   

2. The following request fragment shows the effect of the Skip parameter:

       %tok:string = ' * 0 * 1 * 2 * 3 * 4 * 5 * 6 * 7'
       repeat while %tok:notAtEnd
         printtext {~} = '{%tok:nextToken(skip=1)}'
       end repeat
   

   The result is:

       %tok:nextToken(skip=1) = '0'
       %tok:nextToken(skip=1) = '1'
       %tok:nextToken(skip=1) = '2'
       %tok:nextToken(skip=1) = '3'
       %tok:nextToken(skip=1) = '4'
       %tok:nextToken(skip=1) = '5'
       %tok:nextToken(skip=1) = '6'
       %tok:nextToken(skip=1) = '7'