TokensToLower (StringTokenizer property): Difference between revisions
mNo edit summary |
|||
| (6 intermediate revisions by 2 users not shown) | |||
| Line 17: | Line 17: | ||
<tr><th>newBoolean</th> | <tr><th>newBoolean</th> | ||
<td>The <var>Boolean</var> value to assign to <var class="term">stringTokenizer</var>'s <var>TokensToLower</var> property.</td></tr> | <td>The <var>Boolean</var> value to assign to <var class="term">stringTokenizer</var>'s <var>TokensToLower</var> property. Note that if the <var>TokensToUpper</var> property is <code>true</code>, the <var>TokensToLower</var> property is ignored, so if you want to force tokens to be returned in lower case, the <var>TokensToUpper</var> property must be <code>false</code> (which is its default value). If you need to save the tokenization case setting and then force lower case returned tokens, you can use a technique such as described in [[#Preserving case setting|"Preserving case setting"]]. </td></tr> | ||
</table> | </table> | ||
| Line 23: | Line 23: | ||
<ul> | <ul> | ||
<li><var>TokensToLower</var> has no effect on non-alphabetic characters. | <li><var>TokensToLower</var> has no effect on non-alphabetic characters. | ||
<li>The <var>[[TokensToUpper (StringTokenizer property)|TokensToUpper]]</var> property specifies or reports whether | <li>The <var>[[TokensToUpper (StringTokenizer property)|TokensToUpper]]</var> property specifies or reports whether returned tokens are to be converted to uppercase characters. Note that if <var>TokensToUpper</var> is <code>true</code>, then <var>TokensToLower</var> is ignored. | ||
returned tokens are to be converted to uppercase characters. | |||
</ul> | </ul> | ||
==Examples== | ==Examples== | ||
===Returning lowercased tokens=== | |||
In the following example, the input string's tokens are returned in sequence | In the following example, the input string's tokens are returned in sequence | ||
under the default (< | under the default (<var>False</var>) setting of <var>TokensToLower</var>: | ||
<p class="code">%tok is object stringtokenizer | <p class="code">%tok is object stringtokenizer | ||
%tok = new | %tok = new | ||
%tok:string = 'Bad,F ?3mO ,7,{x Z}' | %tok:string = 'Bad,F ?3mO ,7,{x Z}' | ||
repeat while not %tok:atEnd | repeat while not %tok:atEnd | ||
printText {~} is {%tok:nextToken} | [[Targeted Text statements#AuditText, PrintText, and TraceText|printText]] {~} is {%tok:nextToken} | ||
end repeat | end repeat | ||
</p> | </p> | ||
| Line 51: | Line 51: | ||
%tok:nextToken is ,7,{x | %tok:nextToken is ,7,{x | ||
%tok:nextToken is z} | %tok:nextToken is z} | ||
</p> | |||
===Preserving case setting=== | |||
Since the case which is returned by tokenization is determined by both <var>TokensToUpper</var> and <var>TokensToLower</var>, if you wish to save the previous state and then force lower case tokens, you can use a technique such as the following: | |||
<p class="code">%oldUp = %tok:tokensToUpper | |||
%oldLow = %tok:tokensToLower | |||
%tok:tokensToUpper = false | |||
%tok:tokensToLower = true | |||
... | |||
... use tokenizer, lower case tokens returned | |||
... | |||
%tok:tokensToUpper = %oldUp | |||
%tok:tokensToLower = %oldLow | |||
</p> | </p> | ||
==See also== | ==See also== | ||
{{Template:StringTokenizer:TokensToLower footer}} | {{Template:StringTokenizer:TokensToLower footer}} | ||
Latest revision as of 20:59, 15 November 2012
Convert returned tokens to all-lowercase (StringTokenizer class)
This readWrite property returns or sets the Boolean value that indicates
whether to convert returned, non-quoted, tokens to all-lowercase characters.
The default value for a new tokenizer is False.
Syntax
%currentBoolean = stringTokenizer:TokensToLower stringTokenizer:TokensToLower = newBoolean
Syntax terms
| %currentBoolean | A Boolean enumeration object to contain the returned value of TokensToLower. |
|---|---|
| stringTokenizer | A StringTokenizer object. |
| newBoolean | The Boolean value to assign to stringTokenizer's TokensToLower property. Note that if the TokensToUpper property is true, the TokensToLower property is ignored, so if you want to force tokens to be returned in lower case, the TokensToUpper property must be false (which is its default value). If you need to save the tokenization case setting and then force lower case returned tokens, you can use a technique such as described in "Preserving case setting". |
Usage notes
- TokensToLower has no effect on non-alphabetic characters.
- The TokensToUpper property specifies or reports whether returned tokens are to be converted to uppercase characters. Note that if TokensToUpper is
true, then TokensToLower is ignored.
Examples
Returning lowercased tokens
In the following example, the input string's tokens are returned in sequence under the default (False) setting of TokensToLower:
%tok is object stringtokenizer %tok = new %tok:string = 'Bad,F ?3mO ,7,{x Z}' repeat while not %tok:atEnd printText {~} is {%tok:nextToken} end repeat
The resulting tokens are:
%tok:nextToken is Bad,F %tok:nextToken is ?3mO %tok:nextToken is ,7,{x %tok:nextToken is Z}
If %tok:TokensToLower = True is specified for the instantiated object,
the resulting tokens are:
%tok:nextToken is bad,f %tok:nextToken is ?3mo %tok:nextToken is ,7,{x %tok:nextToken is z}
Preserving case setting
Since the case which is returned by tokenization is determined by both TokensToUpper and TokensToLower, if you wish to save the previous state and then force lower case tokens, you can use a technique such as the following:
%oldUp = %tok:tokensToUpper %oldLow = %tok:tokensToLower %tok:tokensToUpper = false %tok:tokensToLower = true ... ... use tokenizer, lower case tokens returned ... %tok:tokensToUpper = %oldUp %tok:tokensToLower = %oldLow