CurrentTokenPosition (StringTokenizer property)

Position of first character in the current token

CurrentTokenPosition is a member of the StringTokenizer class.

This readWrite property returns or sets the position of the first character in the current token.

If you change the current token (by invoking NextToken, FindToken, or SkipTokens), CurrentTokenPosition is updated with a new value. If you explicitly set CurrentTokenPosition, the value returned by CurrentToken is adjusted accordingly.

Syntax

  %pos = %tok:CurrentTokenPosition

  %tok:CurrentTokenPosition = %pos

Syntax terms

%pos

A numeric variable to set or contain the returned value of the current token's first character.

%tok

A StringTokenizer object variable.

Usage Notes

• Issuing CurrentTokenPosition for a new StringTokenizer object is invalid and cancels the request until a NextToken or FindToken call establishes an initial token.
• The NextPosition property returns the position of the character from which the tokenizer starts parsing for the next token (often this is the position that follows the last character in the current token after the most recent call of NextToken). The NextPosition value is entirely independent from the CurrentTokenPosition value, and vice versa. Both properties can be set to arbitrary locations in the tokenization string. However, if these properties are never explicitly set, they will always have the relationship stated in the previous paragraph.

Examples

In the following request fragment, the string's first token is selected and immediately becomes the current token with current token position 1. The current token position is then explicitly moved, so the subsequent current token starts at that new position. The subsequent NextToken call, however, ignores the current token position and selects a token based on the position established by the previous NextToken call:

    %tok = new(spaces=',')
    %tok:string = 'start,one,two,three,end'
    printText {~} is {%tok:nextToken}
    printText {~} is {%tok:currentToken}
    printText {~} is {%tok:currentTokenPosition}
    %tok:currentTokenPosition = 15
    printText {~} is {%tok:currentTokenPosition}
    printText {~} is {%tok:currentToken}
    printText {~} is {%tok:nextToken}

The result is:

    %tok:nextToken is start
    %tok:currentToken is start
    %tok:currentTokenPosition is 1
    %tok:currentTokenPosition is 15
    %tok:currentToken is three
    %tok:nextToken is one