RegEx: Fix handling of unset/unknown capture groups

[dalexeev]

Add the PCRE2_SUBSTITUTE_UNSET_EMPTY flag so that unset capture groups do not clear the line.

• Closes Add option to allow RegEx substitution empty replacement groups godot-proposals#6114.
• See also Fix wrong OS regex in project converter #73954.

https://www.pcre.org/current/doc/html/pcre2api.html

PCRE2_SUBSTITUTE_UNKNOWN_UNSET causes references to capture groups that do not appear in the pattern to be treated as unset groups. This option should be used with care, because it means that a typo in a group name or number no longer causes the PCRE2_ERROR_NOSUBSTRING error.

PCRE2_SUBSTITUTE_UNSET_EMPTY causes unset capture groups (including unknown groups when PCRE2_SUBSTITUTE_UNKNOWN_UNSET is set) to be treated as empty strings when inserted as described above. If this option is not set, an attempt to insert an unset group causes the PCRE2_ERROR_UNSET error. This option does not influence the extended substitution syntax described below.

• Add error message output.

Example:

var line := "property get_property set_property"

var re1 := RegEx.new()
re1.compile("((get_|set_)?)property")

var re2 := RegEx.new()
re2.compile("(get_|set_)?property")

print("|", re1.sub(line, "$1new_property", true), "|")
print("|", re2.sub(line, "$1new_property", true), "|")
print("|", re2.sub(line, "$5new_property", true), "|")

Before:

|new_property get_new_property set_new_property|
||
||

After:

|new_property get_new_property set_new_property|
|new_property get_new_property set_new_property|
  modules/regex/regex.cpp:332 - PCRE2 Error: unknown substring
|new_property new_property new_property|

[akien-mga]

This makes sense to me. It makes the behavior consistent with Python 3's re.sub().

$ python3
Python 3.13.0 (main, Oct  8 2024, 00:00:00) [GCC 14.2.1 20240912 (Red Hat 14.2.1-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import re
>>> line = "property get_property set_property"
>>> res1 = re.sub(r"((get_|set_)?)property", r"\1new_property", line)
>>> res1
'new_property get_new_property set_new_property'
>>> res2 = re.sub(r"(get_|set_)?property", r"\1new_property", line)
>>> res2
'new_property get_new_property set_new_property'
>>> res3 = re.sub(r"(get_|set_)?property", r"\5new_property", line)
Traceback (most recent call last):
  File "<python-input-8>", line 1, in <module>
    res3 = re.sub(r"(get_|set_)?property", r"\5new_property", line)
  File "/usr/lib64/python3.13/re/__init__.py", line 208, in sub
    return _compile(pattern, flags).sub(repl, string, count)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.13/re/__init__.py", line 377, in _compile_template
    return _sre.template(pattern, _parser.parse_template(repl, pattern))
                                  ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.13/re/_parser.py", line 1070, in parse_template
    addgroup(int(this[1:]), len(this) - 1)
    ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.13/re/_parser.py", line 1015, in addgroup
    raise s.error("invalid group reference %d" % index, pos)
re.PatternError: invalid group reference 5 at position 1
>>> res3
Traceback (most recent call last):
  File "<python-input-9>", line 1, in <module>
    res3
NameError: name 'res3' is not defined. Did you mean: 'res1'?

Also checked sed:

$ echo "property get_property set_property" | sed 's/\(\(get_\|set_\)?\)property/\1new_property/g'
property get_property set_property
$ echo "property get_property set_property" | sed 's/\(get_\|set_\)?property/\1new_property/g'
property get_property set_property
$ echo "property get_property set_property" | sed 's/\(get_\|set_\)?property/\5new_property/g'
sed: -e expression #1, char 42: invalid reference \5 on `s' command's RHS

And perl:

$ perl -pe 's/((get_|set_)?)property/\1new_property/g' <<< "property get_property set_property"
new_property get_new_property set_new_property
$ perl -pe 's/(get_|set_)?property/\1new_property/g' <<< "property get_property set_property"
new_property get_new_property set_new_property
$ perl -pe 's/(get_|set_)?property/\5new_property/g' <<< "property get_property set_property"
new_property new_property new_property

[akien-mga]

I think this could be merged as is, but indeed there's a case to be made about what the return value should be in case of failure.

[akien-mga]

I've just been thinking that it would be good to add a unit test for those changes.

[akien-mga]

Thanks!