RegexSplit (String function)

Split string using regex, creating new Stringlist (String class)

RegexSplit repeatedly applies a regular expression, or "regex," to the method object string until it has tested the entire string. This splits the string into the substrings that are matched by the regex (the "separators") and the substrings that are not matched. By default, the method saves each unmatched substring as a separate item in the Stringlist result object. The leftmost unmatched substring is the first item in the Stringlist, the next leftmost is the second item, and so on.

A simple application of RegexSplit is to extract only the data items from a string of comma-separated data items. If the specified regex is a comma, each of the resulting Stringlist items will contain one of the data items.

The Stringlist that is returned by a default invocation of RegexSplit will contain at least as many items as there are instances of matched substrings. Upon each match, the input string characters preceding the matched ones (and since the previous matched ones) are saved as a Stringlist item. If there are consecutive matching substrings (no unmatched characters between the matching ones), the corresponding Stringlist item for the second matching substring is empty.

RegexSplit also has non-default alternatives that let you save the following in the Stringlist:

• only the matched substrings
• both the matched and unmatched substrings
• only the substrings that are matched by capturing groups in the specified regex
• the unmatched substrings and the substrings matched by capturing groups

Within a regex, characters enclosed by a pair of unescaped parentheses form a "subexpression". A subexpression is a capturing group if the opening parenthesis is not followed by a question mark (?).

RegexSplit uses the Regex rules of regular expression matching.

Syntax

%sl = string:RegexSplit( regex, [Options= string], - [Add= regexSplitOutputOptions]) Throws InvalidRegex

Syntax terms

%sl	A Stringlist object created and returned by the RegexSplit function.
string	The input string, to which the regular expression regex is applied.
regex	A string that is interpreted as a regular expression and is applied in a matching operation to the method object string.
Options	This is an optional, but name required, parameter supplying a string of single letter options, which may be specified in uppercase or lowercase, in any combination, and blank separated or not as you prefer. For more information about these options, see "Common regex options". I Do case-insensitive matching between string and regex. S Dot-All mode: a period (.) can match any character, including carriage return and linefeed. M Multi-line mode: let anchor characters match end-of-line indicators wherever the indicator appears in the input string.
Add	The Add argument (name required) is a RegexSplitOutputOptions enumeration value (see below) that specifies what substrings of the input string string to store in the output Stringlist %outList.

RegexSplitOutputOptions enumeration

The values of this enumeration are the following:

Unmatched	Store only each unmatched substring and any empty substrings due to adjacent separators (consecutive matching substrings). For example, if the value of the RegexSplit parameter regex is #, and string is C###D, the UnMatched option adds four Stringlist items: C, two empty items, then D. Unmatched is the default.
Matched	Store each matched substring only. Include those characters matched by capturing or non-capturing groups.
MatchedAndUnmatched	Store each matched and each unmatched substring in alternating Stringlist items. The first item contains the first unmatched substring, the second item contains the first matched substring, and so on, ending with the last matched substring and the last unmatched substring.
Captured	Store only those substrings matched by capturing groups in the regex argument — as if RegexCapture were applied repeatedly.
CapturedAndUnmatched	Store the following in alternating Stringlist items: Those substrings matched by capturing groups in regex Each unmatched substring The first Stringlist item contains the first unmatched substring, if any; otherwise, it contains the substring captured by the first capturing group. The next item contains the substring captured by the next, if any, capturing group; otherwise, it contains the next unmatched string. And so on.

Note: As with all enumerations, the ToString method implicitly converts an enumeration value to a character string whose value is the name of the enumeration value. For more information about methods available to all enumerations, see "Common enumeration methods".

Exceptions

RegexSplit can throw the following exception:

InvalidRegex

If the regex parameter does not contain a valid regular expression. The exception object indicates the position of the character in the regex parameter where it was determined that the regular expression is invalid, and a description of the nature of the error.

Usage notes

• It is strongly recommended that you protect your environment from regex processing demands on PDL and STBL space by setting, say, UTABLE LPDLST 3000 and UTABLE LSTBL 9000. See SOUL programming considerations.
• The String intrinsic class RegexSplit function performs the same processing as the Stringlist class RegexSplit function. The only differences are:
  1. The target Stringlist for the Stringlist class method is its method object, whereas it is the function output for the String class method.
  2. The Stringlist class method appends to the target stringlist, the String class method creates a new Stringlist.
  3. The String class method has no status parameter, the Stringlist class method does.
  4. The only way the String class method has of indicating an error is via an exception.
• Basically, RegexSplit divides a string into pieces, some of which match a regular expression and some of which don't. Although, you may be more interested in the pieces that are matched than the pieces that are unmatched, this documentation often refers to the matched pieces as "separators," which displays a bias towards the default paradigm of a comma-delimited list. In this case (which is equivalent to the method's Add=Unmatched option), the regex identifies the "commas," and the method extracts the "list elements" that remain. In the algorithm for extracting the list items in the default case:
  1. The regex makes its initial match in the input string; this is the first "comma" separator.
  2. The characters to the left of the matched substring (if none, then the null string) are the first "list element" unmatched piece.
  3. The regex finds its next match, the next separator. The substring between the final character in the previous match and the first character in the current match (if none, then the null string) becomes the second unmatched piece.
  4. When the regex fails to make a next match, the entire substring remaining after the last match (if none, then the null string) becomes the last unmatched piece.

  Worth noting about this algorithm:

  • The matching by the regex is not affected by any capturing strings in the regex. Capturing, per se, is not involved in the default case.
  • A "comma" (separator) is always assumed to be preceded and followed by a "list element" (unmatched piece). Consequently, the extracted Stringlist often contains at least one null item. Only a comma-delimited list of the form 'a,b,c' (where there are no repeated, list-starting, or list-ending commas) results in a Stringlist with no nulls.
  • There is always one more unmatched piece than matched, because the pieces must alternate (consecutive matched pieces are separated by an unmatched null), and they begin and end with an unmatched piece.
• If a RegexSplit regex contains multiple capturing groups and the Add=Captured option is used, each time the regex matches in the input string, the number of Stringlist items added is the number of capturing groups. For example, if you have three capturing groups and you specify an Add=Captured option, the Stringlist item numbers always line up as follows:

  1. First capturing group 2. Second capturing group 3. Third capturing group 4. First capturing group 5. Second capturing group 6. Third capturing group 7. First capturing group 8. Second capturing group 9. Third capturing group ...

  If you are also capturing the non-matched pieces (Add=CapturedAndUnmatched), here is the item order:

  1. First non-matched piece 2. First capturing group 3. Second capturing group 4. Third capturing group 5. Second non-matched piece 6. First capturing group 7. Second capturing group 8. Third capturing group 9. Third non-matched piece 10. First capturing group 11. Second capturing group 12. Third capturing group ...

• An empty item in the output Stringlist may represent consecutive separators in the input string (with Add=Unmatched).

Examples

1. This example demonstrates how RegexSplit operating in default mode against a simple comma-delimited list assigns items to the result Stringlist.

   UTABLE LPDLST 3000 begin %str longstring %regex longstring %sl object stringlist %i is float %str = 'Barry,Mildred' %regex = ',' %sl = %str:regexSplit(%regex) printText {~} is: {%sl:count} for %i from 1 to %sl:count printText %sl({%i}) is: {%sl(%i)} end for end

   For the input string Barry,Mildred and for a comma (,) as the regex, the result is:

   %sl:Count is 2 %sl(1) is: Barry %sl(2) is: Mildred

   For the input string ,Barry,Mildred and the same regex, the result includes a null first item:

   %sl:Count is 3 %sl(1) is: %sl(2) is: Barry %sl(3) is: Mildred

   And similarly, the input string Barry,Mildred, and the same regex produce a null third item; and the input string ,Barry,Mildred, and the same regex produces a null first item and fourth items.

2. This example shows the utility of an Add=Matched option.

   ... %str = ' Barry Mildred Jack Faust ' %sl = %str:regexSplit(' +') printText ---------- Unmatched: %sl:Print %c is string len 10 %c = '[' with '5F':hexToString With ' ]+' %sl = %str:regexSplit(%c, add=matched) printText ---------- Matched: %sl:Print printText ----------* ...

   The result shows that using an Add=Matched option along with a regex that matches directly the data you want to extract is a successful alternative that also lets you avoid the nulls in the Stringlist output:

   ---------- Unmatched: Barry Mildred Jack Faust ---------- Matched: Barry Mildred Jack Faust ----------*

3. In the following example, an Add=Captured option is used with a regex that includes multiple capture groups to strip the labels but capture the data values of an input string. Using the CreateLines method as shown is a way to use RegexSplit against a Stringlist.

   begin %troops is object stringList %tokens is object stringList %troops = new text to %troops Name: Clegg Rank: Corporal Missing: Leg Name: Ryan Rank: Private Missing: Brothers Name: Bilko Rank: Sergeant Missing: Discipline end text %tokens= %troops:createLines:regexSplit( - name: *(\S+)\s*Rank: (\S+)\s*Missing: *(\S+)', - add=captured) %tokens:print(,3) end

   This is the result:

   5 Clegg 8 Corporal 3 Leg 4 Ryan 7 Private 8 Brothers 5 Bilko 8 Sergeant 10 Discipline

See also

• For information about additional methods that support regular expressions, see "Regex Processing".

Float class String class Unicode class	List of Float methods List of String methods List of Unicode methods List of Intrinsic methods	Float methods syntax String methods syntax Unicode methods syntax
Notation conventions for methods