Support for incremental tokenization

[fabiospampinato]

It'd be very interesting for some applications if Shiki supported tokenizing incrementally, basically something that editors like vscode might want to use.

[muenzpraeger]

Out of curiosity: what would be the use case for incremental tokenization?

[fabiospampinato]

My use case would be building a new editor, I can't stop the world when tokenizing a document, and I can't restart tokenization from scratch when a new line is appended to the document, for example.

[muenzpraeger]

So your use case is around general tokenization of a given string/line of code based on a tmLanguage definition, and not (static) syntax highlighting?

[fabiospampinato]

Yes.

[orta]

Realistically shiki is too high level of an abstraction for you to do that, you're probably wanting to use the APIs from vscode-oniguruma & vscode-textmate yourself

[fabiospampinato]

Makes sense, thanks 👍

[octref]

I'd say look into tree-sitter.

[fabiospampinato]

Tree-sitter is interesting, but it doesn't support as many grammars as TextMate and the syntax highlighting would be slightly different compared to vscode, which I guess would be undesirable, even if it's slightly better.

I'd like to add support for both eventually though, for the moment I've only wired textmate grammars in.

[tats-u]

@fabiospampinato VS Code started to implement this feature: microsoft/vscode#210475
You do not have to throw away the current TextMate syntax assets; only supported languages are fine.
Could you reconsider this feature in shiki?