Reorder reverse caps characters table for string lower case conversion

[Xrayez]

The binary search algorithm used to lookup character codes in the table relies that the data must be ordered. This fixes to_lower() string method to convert upper case to lower case properly, so that the
algorithm doesn't terminate prematurely.

Fixes #26744 and potentially more languages which are not Latin.

[bruvzg]

Case mapping in core/ucaps.h is definitely incomplete, quick Unicode Character Database gives 1407 lower->upper and 1390 upper->lower one-to-one records, plus there are one-to-many special mappings.

Unicode case mapping FAQ: https://unicode.org/faq/casemap_charprop.html

[Xrayez]

@bruvzg so you say that we should create a script to generate those mappings from that database? That's a ton of mappings and might decrease performance as well, hmm.

I managed to finish creating a unit test for this case, for some reason a simple == operator wouldn't do it. Probably because it converts both strings with c_str which is not unicode? (which is yet another bug to fix?) found solution.

[breakmt]

@Xrayez I'm sorry, but I think unit test is a little bit incorrect - doesn't your cycle check just 1 letter?

[bruvzg]

@bruvzg so you say that we should create a script to generate those mappings from that database? That's a ton of mappings and might decrease performance as well, hmm.

Probably. Mappings should have many long runs, we could compact each run into a single record.
For example something like Go unicode package do (same binary search but with a bit more complex table):

Remap function:
https://github.com/golang/go/blob/50bd1c4d4eb4fac8ddeb5f063c099daccfb71b26/src/unicode/letter.go#L210

Table:
https://github.com/golang/go/blob/master/src/unicode/tables.go#L6670

[Xrayez]

@breakmt thanks for pointing out, a mistake from copy pasting... And surely the test doesn't pass again!

It does work in GDScript, can't understand why it doesn't work in C++, should I use unicode literals like \u0444 perhaps?

EDIT: no, using Cyrillic unicode literals doesn't work either.
EDIT2: interestingly, with one Cyrillic character it worked fine.

[Xrayez]

Ah yes, forgot about different types of string literals in C++:

L"АБВГДЕЁЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯ"

would be the correct unicode one that handles wchar_t type, the test passes!

[reduz]

is this the right PR, or the other one? :)

[Xrayez]

@reduz @akien-mga oh so my PR has unit test covered for this bug which #27598 doesn't:
https://github.com/godotengine/godot/blob/4858d2cd9b14249cbbdbc9fcf62c8e6561d8f292/main/tests/test_string.cpp#L1056-L1067

(also reordered commented out "dotless I" the same way)

Either way I added @AndreevAndrei as a co-author, the only problem is that he isn't detected by GitHub (git credentials are not matching) so his contribution will not be registered...

If @AndreevAndrei wants his contribution registered, he could add me as a co-author instead with these changes, or fix his git credentials first to match GitHub (email and username). 😃

[akien-mga]

Thanks!

[akien-mga]

Cherry-picked for 3.1.1.