TokensToLower (StringTokenizer property)

Convert returned tokens to all-lowercase (StringTokenizer class)

This readWrite property returns or sets the Boolean value that indicates whether to convert returned, non-quoted, tokens to all-lowercase characters. The default value for a new tokenizer is False.

Syntax

%currentBoolean = stringTokenizer:TokensToLower stringTokenizer:TokensToLower = newBoolean

Syntax terms

%currentBoolean	A Boolean enumeration object to contain the returned value of TokensToLower.
stringTokenizer	A StringTokenizer object.
newBoolean	The Boolean value to assign to stringTokenizer's TokensToLower property.

Usage notes

• TokensToLower has no effect on non-alphabetic characters.
• The TokensToUpper property specifies or reports whether returned tokens are to be converted to uppercase characters. Note that if TokensToUpper is true, then TokensToLower is ignored.

Examples

Returning lowercased tokens

In the following example, the input string's tokens are returned in sequence under the default (False) setting of TokensToLower:

%tok is object stringtokenizer %tok = new %tok:string = 'Bad,F ?3mO ,7,{x Z}' repeat while not %tok:atEnd printText {~} is {%tok:nextToken} end repeat

The resulting tokens are:

%tok:nextToken is Bad,F %tok:nextToken is ?3mO %tok:nextToken is ,7,{x %tok:nextToken is Z}

If %tok:TokensToLower = True is specified for the instantiated object, the resulting tokens are:

%tok:nextToken is bad,f %tok:nextToken is ?3mo %tok:nextToken is ,7,{x %tok:nextToken is z}

Preserving case setting

Since the case which is returned by tokenization is determined by both TokensToUpper and TokensToLower, if you wish to save the previous state and then force lower case tokens, you can use a technique such as the following:

%oldUp = %tok:tokensToUpper %oldLow = %tok:tokensToLower %tok:tokensToUpper = false %tok:tokensToLower = true ... use tokenizer, lower case tokens returned %tok:tokensToUpper = %oldUp %tok:tokensToLower = %oldLow

See also

StringTokenizer class	List of StringTokenizer methods	StringTokenizer methods syntax
Notation conventions for methods