Explore using tree sitter for syntax highlighting

[alexr00]

Goals of the explorations:

• Get Aiday's exploration up and running again: [Exploration] Tree-sitter tokenization exploration (Fixes #161256) #161479
• Come up with ideas for a contribution model
• Confirm performance findings from [Exploration] Tree-sitter tokenization exploration (Fixes #161256) #161479 and understand incremental highlighting

[alexr00]

Some notes on incremental parsing:

• Per Can web-tree-sitter actually do async parsing? tree-sitter/tree-sitter#2118, it looks like we can set a timeout so that each call to parse times-out, with the parsing progress preserved for the next call to parse.
• I've confirmed that this is the case; however, it appears to be broken in the latest version of web-tree-sitter. Going back to 0.22.2 works (latest is 0.22.5).
• There does not appear to be any significant time penalty for setting a timeout.

@hediet FYI.

[alexr00]

Opened tree-sitter/tree-sitter#3341 for the above issue.

[alexr00]

After some discussion, we've decided to first understand the performance of using tree-sitter to parse files. #213565 contains that work.

[alexr00]

We also need new internal editor API to write a more optimal parse function, which @rebornix may help with.

[alexr00]

Progress in July:

• Created a build pipeline for the relevant wasm files (tree-sitter and typescript right now): https://github.com/microsoft/vscode-tree-sitter-wasm
• Published first npm package with said wasm files: https://www.npmjs.com/package/@vscode/tree-sitter-wasm
• Incorporated that package in Add telemtry for how long it takes to parse files with tree-sitter #213565, which is almost ready to merge, but we'll hold off until after endgame as there's no need to rush it in before release.

[usernamehw]

Will using Tree Sitter make it easier to implement this (10yo) extension api feature #580 - get scopes at Position/Range?

[RedCMD]

@usernamehw
yes

@alexr00
is there any time frame for giving extensions access?
proposed api?

[alexr00]

No time frame at the moment.

[alexr00]

End of Febuary updates:

• Viewport is colored first
• Existing bracket features enabled
• Themes that target multiple scopes enabled
• We're rolling out tree sitter based syntax highlighting for ini files in insiders
• Improvements to typescript highlighting

[usernamehw]

Is it possible for color themes to use scopes provided by tree-sitter?

[alexr00]

Currently, only scopes listed in the highligths file are available, though I'm ading more to that as we find things missing that existing themes need.

Once we have a contribution story, I think we would allow extensions to contribute an scm file like that.

[tats-u]

If the current language lacks the tree-sitter syntax, VS Code should fall back to the current TextMate highlight syntax.

[RedCMD]

@tats-u that is the current plan