RegexSplit (String function)

Split string using regex, creating new Stringlist (String class)

This method repeatedly applies a regular expression, or "regex," to the method object string until it has tested the entire string. This splits the string into the substrings that are matched by the regex (the "separators") and the substrings that are not matched. By default, the method saves each unmatched substring as a separate item in the Stringlist result object. The leftmost unmatched substring is the first item in the Stringlist, the next leftmost is the second item, and so on.

A simple application of the method is to extract only the data items from a string of comma-separated data items. If the specified regex is a comma, each of the resulting Stringlist items will contain one of the data items.

The Stringlist that is returned by a default invocation of RegexSplit will contain at least as many items as there are instances of matched substrings. Upon each match, the input string characters preceding the matched ones (and since the previous matched ones) are saved as a Stringlist item. If there are consecutive matching substrings (no unmatched characters between the matching ones), the corresponding Stringlist item for the second matching substring is empty.

RegexSplit also has non-default alternatives that let you save the following in the Stringlist:

• only the matched substrings
• both the matched and unmatched substrings
• only the substrings that are matched by capturing groups in the specified regex
• the unmatched substrings and the substrings matched by capturing groups

Within a regex, characters enclosed by a pair of unescaped parentheses form a "subexpression." A subexpression is a capturing group if the opening parenthesis is not followed by a question mark (?).

RegexSplit uses the rules of regular expression matching.

RegexSplit is available as of version 7.2 of the Sirius Mods.

Syntax

%sl = string:RegexSplit( regex, [Options= string], - [Add= regexSplitOutputOptions]) Throws InvalidRegex

Syntax terms

%outList	A Stringlist object created and returned by the RegexSplit function.
string	The input string, to which the regular expression regex is applied.
regex	A string that is interpreted as a regular expression and is applied in a matching operation to the method object string.
Options=opts	The Options argument (name required) is an optional string of options. The options are single letters, which may be specified in uppercase or lowercase, in any combination, and separated by blanks or not separated. For more information about these options, see Common regex options. I Do case-insensitive matching between string and regex. S Dot-All mode: a dot (.) can match any character, including carriage return and linefeed. M Multi-line mode: let anchor characters match end-of-line indicators wherever the indicator appears in the input string. M mode is ignored if C (XML Schema) mode is specified.
Add=output	The Add argument (name required) is one of the following RegexSplitOutputOptions enumeration values, which specify what substrings of the input string string to store in the output Stringlist %outList: Unmatched Store only each unmatched substring and any empty substrings due to adjacent separators (consecutive matching substrings). For example, if the value of regex is '#', and string is 'C###D', the UnMatched option adds four Stringlist items: 'C', two empty items, then 'D'. Unmatched is the default. Matched Store each matched substring only. Include those characters matched by capturing or non-capturing groups. MatchedAndUnmatched Store each matched and each unmatched substring in alternating Stringlist items. The first item contains the first unmatched substring, the second item The first item contains the first unmatched substring, the second item contains the first matched substring, and so on, ending with the last matched substring and the last unmatched substring. Captured Store only those substrings matched by capturing groups in regex — as if RegexCapture were applied repeatedly. CapturedAndUnmatched Store in alternating Stringlist items a) those substrings matched by capturing groups in regex, and b) each unmatched substring. The first item contains the first unmatched substring, if any; otherwise, it contains the substring captured by the first capturing group. The next item contains the substring captured by the next, if any, capturing group; otherwise, it contains the next unmatched string. And so on.

Exceptions

This intrinsic function can throw the following exceptions:

InvalidRegex

If the regex parameter does not contain a valid regular expression. The exception object indicates the position of the character in the regex parameter where it was determined that the regular expression is invalid, and a description of the nature of the error.

Usage notes

• It is strongly recommended that you protect your environment from regex processing demands on PDL and STBL space by setting, say, 'UTABLE LPDLST 3000' and 'UTABLE LSTBL 9000'. See User Language programming considerations.
• The String intrinsic class RegexSplit function performs the same processing as the Stringlist class RegexSplit function. The only differences are:
  1. The target Stringlist for the Stringlist class method is its method object, whereas it is the function output for the String class method.
  2. The Stringlist class method appends to the target stringlist, the String class method creates a new Stringlist.
  3. The String class method has no status parameter, the Stringlist class method does. The only way the String class method has of indicating an error is via an exception.
• Basically, RegexSplit divides a string into pieces, some of which match a regular expression and some of which don't. Although, you may be more interested in the pieces that are matched than the pieces that are unmatched, this documentation often refers to the matched pieces as "separators," which displays a bias towards the default paradigm of a comma-delimited list. In this case (which is equivalent to the method's Add=Unmatched option), the regex identifies the "commas," and the method extracts the "list elements" that remain. In the algorithm for extracting the list items in the default case:
  1. The regex makes its initial match in the input string; this is the first "comma" separator.
  2. The characters to the left of the matched substring (if none, then the null string) are the first "list element" unmatched piece.
  3. The regex finds its next match, the next separator. The substring between the final character in the previous match and the first character in the current match (if none, then the null string) becomes the second unmatched piece.
  4. When the regex fails to make a next match, the entire substring remaining after the last match (if none, then the null string) becomes the last unmatched piece.

  Worth noting about this algorithm:

  • The matching by the regex is not affected by any capturing strings in the regex. Capturing, per se, is not involved in the default case.
  • A "comma" (separator) is always assumed to be preceded and followed by a "list element" (unmatched piece). Consequently, the extracted Stringlist often contains at least one null item. Only a comma-delimited list of the form 'a,b,c' (where there are no repeated, list-starting, or list-ending commas) results in a Stringlist with no nulls.
  • There is always one more unmatched piece than matched, because the pieces must alternate (consecutive matched pieces are separated by an unmatched null), and they begin and end with an unmatched piece.
• If a RegexSplit regex contains multiple capturing groups and the Add=Capture option is used, each time the regex matches in the input string, the number of Stringlist items added is the number of capturing groups. For example, if you have three capturing groups and you specify the Add=Captured option, the Stringlist item numbers always line up as follows:

      1. First capturing group
      2. Second capturing group
      3. Third capturing group
      4. First capturing group
      5. Second capturing group
      6. Third capturing group
      7. First capturing group
      8. Second capturing group
      9. Third capturing group
       ...
  

  If you are also capturing the non-matched pieces (Add=CapturedAndUnmatched), here is the item order:

       1. First non-matched piece
       2. First capturing group
       3. Second capturing group
       4. Third capturing group
       5. Second non-matched piece
       6. First capturing group
       7. Second capturing group
       8. Third capturing group
       9. Third non-matched piece
      10. First capturing group
      11. Second capturing group
      12. Third capturing group
        ...
  

• An empty item in the output Stringlist may represent consecutive separators in the input string (with Add=Unmatched).
• For information about additional methods that support regular expressions, see Regex Processing.

Examples

1. This example demonstrates how RegexSplit operating in default mode against a simple comma-delimited list assigns items to the result Stringlist.

       UTABLE LPDLST 3000
       begin
         %str longstring
         %regex <var>Longstring</var>
         %sl object <var>Stringlist</var>
         %i is float
   
         %str = 'Barry,Mildred'
         %regex = ','
         %sl = %str:regexSplit(%regex)
         printText {~} is: {%sl:count}
         for %i from 1 to %sl:count
            printText %sl({%i}) is: {%sl(%i)}
         end for
       end
    

   For the input string Barry,Mildred and for a comma (',') as the regex, the result is:

       %sl:Count is 2
       %sl(1) is: Barry
       %sl(2) is: Mildred
   

   For the input string ',Barry,Mildred' and the same regex, the result includes a null first item:

       %sl:Count is 3
       %sl(1) is:
       %sl(2) is: Barry
       %sl(3) is: Mildred
   

   And similarly, the input string 'Barry,Mildred,' and the same regex produce a null third item; and the input string ',Barry,Mildred,' and the same regex produce a null first item and a null fourth item.

2. This example shows the utility of the Add=Matched option.

        ...
       %str = ' Barry  Mildred    Jack     Faust  '
       %sl = %str:regexSplit(' +')
       printText ---------- Unmatched:
       %sl:Print
   
       %c is string len 10
       %c = '[' with '5F':hexToString With ' ]+'
       %sl = %str:regexSplit(%c, add=matched)
       printText ---------- Matched:
       %sl:Print
       printText ----------*
        ...
   

   The result shows that using the Add=Matched option along with a regex that matches directly the data you want to extract is a successful alternative that also lets you avoid the nulls in the Stringlist output:

      ---------- Unmatched:
   

      Barry
      Mildred
      Jack
      Faust
   

      ---------- Matched:
      Barry
      Mildred
      Jack
      Faust
      ----------*
   

3. In the following example, the Add=Capture option is used with a regex that includes multiple capture groups to strip the labels but capture the data values of an input string. Using the CreateLines method as shown is a way to use RegexSplit against a Stringlist.

       b
       %troops is object stringList
       %tokens is object stringList
   
       %troops = new
       text to %troops
          Name: Clegg
          Rank: Corporal
          Missing: Leg
   
          Name: Ryan
          Rank: Private
          Missing: Brothers
   
          Name: Bilko
          Rank: Sergeant
          Missing: Discipline
       end text
   
       %tokens= %troops:createLines:regexSplit(      -
               'Name: *(\S+)\s*Rank: (\S+)\s*Missing: *(\S+)', -
               add=captured)
       %tokens:print(,3)
       end
   

   This is the result:

       5 Clegg
       8 Corporal
       3 Leg
       4 Ryan
       7 Private
       8 Brothers
       5 Bilko
       8 Sergeant
      10 Discipline
   

See also

List of intrinsic String methods