Do not remove tokens before AST json serialization

[petrochenkov]

TokenStripper is error-prone and introduces one more use of MutVisitor.
It's much simpler to treat serialization as just one more place that wants lazy token stream to turn into a real token stream.
Also, no code is better than more code, in general.
r? @Aaron1011

(I also merged tests for TokenStripper ICEs into one.)

[Aaron1011]

I wanted to avoid making this serializable, to guarantee that we properly 'lower' a TokenStream when serializing a macro_rules! definition. However, I agree that the use of MutVisitor isn't ideal.

[petrochenkov]

It doesn't matter much when exactly we force the real token stream creation?
We could delay it until metadata encoding (serializing a macro_rules! definition) and that wouldn't be a problem, I think.

Lazy token streams just never survive until that time because all of them are in nonterminals where you have to do other things forcing token creation, like print-reparse checking, anyway.

[Aaron1011]

The LazyTokenStream can be coverted into a TokenStream multiple times, though. Outside of AST JSON serializition, we should never be attempting to serialize a LazyTokenStream - with this PR, we'll get a (small) performance hit instead of an ICE.

I don't think there's a particularly clean way to only panic if we're encoding the LazyTokenStream for something other than AST JSON. If you can think of one, I think it would be a nice sanity check. However, I don't feel very strongly about this, so r=me otherwise.

[petrochenkov]

I think removing the sanity check is a lesser evil than keeping some global variables to check whether ast-json is enabled, or avoiding it with TokenStripper.

Besides, #77271 removes some visitor APIs related to tokens including those used by TokenStripper.

@bors r=Aaron1011 rollup

[bors]

📌 Commit 6b63e9b has been approved by Aaron1011