RegexSplit (Stringlist function)

Split a string onto a Stringlist using regex (Stringlist class)

This callable method repeatedly applies a regular expression, or "regex," to a given input string until it has tested the entire string. This splits the string into the substrings that are matched by the regex (the "separators") and the substrings that are not matched.

By default, the method saves each unmatched substring as a separate item in the Stringlist method object. The leftmost unmatched substring is the first item in the Stringlist, the next leftmost is the second item, and so on. A simple application of the method is to extract only the data items from a string of comma-separated data items. If the specified regex is a comma, each of the resulting Stringlist items will contain one of the data items. The Stringlist that is returned by a default invocation of RegexSplit will contain at least as many items as there are instances of matched substrings. Upon each match, the input string characters preceding the matched ones (and since the previous matched ones) are saved as a Stringlist item. If there are consecutive matching substrings (no unmatched characters between the matching ones), the corresponding Stringlist item for the second matching substring is empty. RegexSplit also has non-default alternatives that let you save the following in the Stringlist:

• only the matched substrings
• both the matched and unmatched substrings
• only the substrings that are matched by capturing groups in the specified regex
• the unmatched substrings and the substrings matched by capturing groups

Within a regex, characters enclosed by a pair of unescaped parentheses form a "subexpression". A subexpression is a capturing group if the opening parenthesis is not followed by a question mark (?). RegexSplit uses the rules of regular expression matching (information about which is provided in "Regex processing rules").

Specifying an invalid argument results in request cancellation.

Syntax

[%number =] sl:RegexSplit( inString, regex, [Options= string], - [Status= %output], [Add= regexSplitOutputOptions]) Throws InvalidRegex

Syntax terms

%number	If specified, a numeric variable that is set to 0 if the regular expression was invalid or no match was found or some error occurred, or it is the number of items added to the method Stringlist sl.
sl	A Stringlist object.
inString	The input string, to which the regular expression regex is applied.
regex	A string that is interpreted as a regular expression and is applied in a matching operation to the inString argument.
Options	This is an optional, name required, parameter supplying a string of single letter options, which may be specified in uppercase or lowercase, in any combination, and blank separated or not as you prefer. For more information about these options, see Common regex options.
Status	The Status argument (name required) is optional; if specified, it is set to an integer code. These values are possible: n A successful match was obtained. The (positive) number of items that are added to the method Stringlist sl. 0 No match (inString not matched by regex), and no error. -6 The regex produced a 0-byte match. This may be the result of metacharacters like ? or *, which by definition can "succeed" without matching a character. In these cases, %number is set to zero (although items may have been appended to the method Stringlist). -1nnn The pattern in regex is invalid. nnn (the absolute value of the return minus 1000) gives the 1-based position of the character being scanned when the error was discovered. The value for an error occurring at end-of-string is the length of the string + 1. If you omit this argument and an invalid regex pattern was specified, an InvalidRegex exception is thrown under Sirius Mods 7.2 and later, and the request is cancelled; under earlier versions of the Sirius Mods. If you omit this argument and another negative Status value is to be returned, the request is cancelled.
Add	The Add argument (name required) is one of the following RegexSplitOutputOptions enumeration values, which specify what substrings of the input string inString to store in the method Stringlist sl: Unmatched Store only each unmatched substring and any empty substrings due to adjacent separators (consecutive matching substrings). For example, if the value of regex is #, and inString is C###D, the UnMatched option adds four Stringlist items: C, two empty items, then D. Unmatched is the default. Matched Store each matched substring only. Include those characters matched by capturing or non-capturing groups. MatchedAndUnmatched Store each matched and each unmatched substring in alternating Stringlist items. The first item contains the first unmatched substring, the second item contains the first matched substring, and so on, ending with the last matched substring and the last unmatched substring. Captured Store only those substrings matched by capturing groups in regex — as if RegexCapture were applied repeatedly. CapturedAndUnmatched Store in alternating Stringlist items a) those substrings matched by capturing groups in regex, and b) each unmatched substring. The first item contains the first unmatched substring, if any; otherwise, it contains the substring captured by the first capturing group. The next item contains the substring captured by the next, if any, capturing group; otherwise, it contains the next unmatched string. And so on. For additional explanation, see this item in "Usage notes" below.

Exceptions

RegexSplit can throw the following exceptions under Sirius Mods 7.2 and later.

InvalidRegex

If the regex parameter does not contain a valid regular expression. The exception object indicates the position of the character in the regex parameter where it was determined that the regular expression is invalid, and a description of the nature of the error.

Usage notes

• RegexSplit is available in Sirius Mods Version 7.0 and later.
• It is strongly recommended that you protect your environment from regular expression processing demands on PDL and STBL space by setting, say, UTABLE LPDLST 3000 and UTABLE LSTBL 9000. See SOUL programming considerations.
• The Stringlist class RegexSplit function performs the same processing as the intrinsic String class RegexSplit function. The only differences are:
  1. The target Stringlist for the Stringlist class method is its method object, whereas it's the function output for the String class method.
  2. The Stringlist class method appends to the target Stringlist, the String class method creates a new Stringlist.
  3. The String class method has no Status parameter, the Stringlist class method does. The only way the String class method has of indicating an error is via an exception.
• Basically, RegexSplit divides a string into pieces, some of which match a regular expression and some of which don't. Although, you may be more interested in the pieces that are matched than the pieces that are unmatched, this documentation often refers to the matched pieces as "separators," which displays a bias towards the default paradigm of a comma-delimited list. In this case (which is equivalent to the method's Add=Unmatched option), the regex identifies the "commas," and the method extracts the "list elements" that remain. In the algorithm for extracting the list items in the default case:
  1. The regex makes its initial match in the input string; this is the first "comma" / separator.
  2. The characters to the left of the matched substring (if none, then the null string) are the first "list element" unmatched piece.
  3. The regex finds its next match, the next separator. The substring between the final character in the previous match and the first character in the current match (if none, then the null string) becomes the second unmatched piece.
  4. When the regex fails to make a next match, the entire substring remaining after the last match (if none, then the null string) becomes the last unmatched piece.

  Worth noting about this algorithm:

  • The matching by the regex is not affected by any capturing strings in the regex. Capturing, per se, is not involved in the default case.
  • A "comma" (separator) is always assumed to be preceded and followed by a "list element" (unmatched piece). Consequently, the extracted Stringlist often contains at least one null item. Only a comma-delimited list of the form a,b,c (where there are no repeated, list-starting, or list-ending commas) results in a Stringlist with no nulls. For an example, see Example 1 below.
  • There is always one more unmatched piece than matched, because the pieces must alternate (consecutive matched pieces are separated by an unmatched null), and they begin and end with an unmatched piece.
• If a RegexSplit regex contains multiple capturing groups and the Add=Capture option is used, each time the regex matches in the input string, the number of Stringlist items added is the number of capturing groups. For example, if you have three capturing groups and you specify the Add=Captured option, the Stringlist item numbers always line up as follows:

  1. First capturing group 2. Second capturing group 3. Third capturing group 4. First capturing group 5. Second capturing group 6. Third capturing group 7. First capturing group 8. Second capturing group 9. Third capturing group ...

  If you are also capturing the non-matched pieces (Add=CapturedAndUnmatched), here is the item order:

  1. First non-matched piece 2. First capturing group 3. Second capturing group 4. Third capturing group 5. Second non-matched piece 6. First capturing group 7. Second capturing group 8. Third capturing group 9. Third non-matched piece 10. First capturing group 11. Second capturing group 12. Third capturing group ...

  For a code example, see Example 3, below.

• If %number is 0, either regex did not match inString, or there was an error in the regex. The Status argument returns additional information: If it is negative, it indicates an error. If it is zero, it indicates there was no error, but the regex did not match.
• An empty item in the output Stringlist may represent consecutive separators in the input string (with Add=Unmatched).
• RegexSplit might add items to the method Stringlist, then subsequently encounter a zero-length match (which is treated as an error; status is set to -6). In this case, the modified Stringlist is not returned to its former state. You are responsible for preserving the unmodified state of the Stringlist if you want to restore that state when handling the error case.
• For information about additional methods and $functions that support regular expressions, see "Regex Processing".

Examples

1. This example demonstrates how RegexSplit operating in default mode against a simple comma-delimited list assigns items to the result Stringlist.

   UTABLE LPDLST 3000 Begin %inStr longstring %regex longstring %sl object stringlist %i is float %sl = new %str = 'Barry,Mildred' %regex = ',' %sl:RegexSplit (%str, %regex) Print '%sl:Count is ' %sl:Count For %i from 1 to %sl:Count Print '%sl(item' With %i With ') is: ' %sl:Item(%i) End For End

   For the input string Barry,Mildred and for a comma (,) as the regex, the result is:

   %sl:Count is 2 %sl(item1) is: Barry %sl(item2) is: Mildred

   For the input string ,Barry,Mildred and the same regex, the result includes a null first item:

   %sl:Count is 3 %sl(item1) is: %sl(item2) is: Barry %sl(item3) is: Mildred

   And similarly, the input string Barry,Mildred, and the same regex produce a null third item; and the input string ,Barry,Mildred, and the same regex produce a null first item and a null fourth item.

2. This example shows the utility of the Add=Matched option.

   ... %str = ' Barry Mildred Jack Faust ' %sl:RegexSplit(%str, ' +') Print '---------- Unmatched: ' %sl:Print %sl = New %c String Len 10 %c = '[' With $X2C('5F') With ' ]+' %sl:RegexSplit(%str, %c, Add=Matched) Print '---------- Matched: ' %sl:Print Print '----------' ...

   The result shows that using the Add=Matched option along with a regex that matches directly the data you want to extract is a successful alternative that also lets you avoid the nulls in the Stringlist output:

   ---------- Unmatched: Barry Mildred Jack Faust ---------- Matched: Barry Mildred Jack Faust ----------

3. In the following example, the Add=Capture option is used with a regex that includes multiple capture groups to strip the labels but capture the data values of an input string. Using the CreateLines method as shown is a way to use RegexSplit against a Stringlist.

   b %troops is object stringList %tokens is object stringList %troops = new text to %troops Name: Clegg Rank: Corporal Missing: Leg Name: Ryan Rank: Private Missing: Brothers Name: Bilko Rank: Sergeant Missing: Discipline end text %tokens = new %tokens:regexSplit(%troops:createLines, - 'Name: *(\S+)\s*Rank: (\S+)\s*Missing: *(\S+)', - add=captured) %tokens:print(,3) end

   This is the result:

   5 Clegg 8 Corporal 3 Leg 4 Ryan 7 Private 8 Brothers 5 Bilko 8 Sergeant 10 Discipline

See also

Stringlist class	List of Stringlist methods	Stringlist methods syntax
Notation conventions for methods