Confusing error message in the presence of unicode combining characters

[Mandragorian]

Given the following code:

fn main() {
    println!("hello {}"̣, 1);
}

Playground link

The current output is:

error: unknown start of token: \u{323}
 --> src/main.rs:2:23
  |
2 |     println!("hello{}"̣, 1);
  |                       ^

error: could not compile `playground` due to previous error

Ideally the output should look like:

error: unknown start of token: \u{323}
 --> src/main.rs:2:23
  |
2 |     println!("hello{}"\u{323}, 1);
  |                       ^
help: Unicode character '\u{323}' might not be visible when rendered
error: could not compile `playground` due to previous error

[chenyukang]

I think the current output is already helpful enough to point out the issue.
In my opinion, the source code shown in disgnostics should keep same format with original code.

[CAD97]

That said, it could be useful to print a help message for any Bidi_Class=Nonspacing_Mark characters (as an approximation for characters that could be difficult to spot)

[Mandragorian]

Just letting the user know that there might be "invisible" characters would help.

[chenyukang]

Just letting the user know that there might be "invisible" characters would help.

Agree.
@rustbot claim

[chenyukang]

@CAD97
Do you know is there any simple way in Rust to test whether a char is in range of Nonspacing_Mark?
Currently, we have a https://github.com/rust-lang/rust/blob/master/compiler/rustc_parse/src/lexer/unicode_chars.rs#L10 to give suggestion for some special Unicode char, but it's not completed.

I found there is a crate: https://github.com/swgillespie/unicode-categories
I think it may too heavy to add a new crate for this? 😁

Maybe give a help message for !c.is_ascii() is also good?

Update: we also have a function in UI

rust/src/test/ui/issues/issue-29227.rs

Line 15 in b998821

fn is_unicode_nonspacing_mark(self) -> bool { false }

[CAD97]

The function in test/ui is part of a minimized repro of #29227, so not relevant. I just picked Nonspacing_Mark as a likely candidate for a property to check, not because the compiler already has access to it.

You'll need to use the unicode table generator to generate a table of the characters with General Category=Nonspacing Mark and/or Bidi_Class=Nonspacing_Mark for which to warn on. Adding the table to rustc is probably fine, adding it to std is nondesirable.