Skip to content

Commit bf4eb7e

Browse files
committed
Move vscode builtin production from yarn to npm
Also updates builtin to 1.94.2 fixes #131 contributed on behalf of STMicroelectronics Signed-off-by: Remi Schnekenburger <rschnekenburger@eclipsesource.com>
1 parent 5586e40 commit bf4eb7e

10 files changed

+73
-96
lines changed

README.md

+2-3
Original file line numberDiff line numberDiff line change
@@ -10,8 +10,7 @@ Every sub-folder of vscode/extensions/ is one built-in vscode extension (with a
1010

1111
## Getting started
1212

13-
Building the built-in `*.vsix` files locally is described in [Building.md](./doc/Building.md). If you need to publish a new version of the built-ins for use with Theia, please follow the
14-
process described in [Publishing.md](./doc/Publishing.md).
13+
Building the built-in `*.vsix` files locally is described in [Building.md](./doc/Building.md). If you need to publish a new version of the built-ins for use with Theia, please follow the process described in [Publishing.md](./doc/Publishing.md).
1514

1615
## License
1716

@@ -21,4 +20,4 @@ process described in [Publishing.md](./doc/Publishing.md).
2120
## Trademark
2221

2322
"Theia" is a trademark of the Eclipse Foundation
24-
https://www.eclipse.org/theia
23+
<https://www.eclipse.org/theia>

doc/Building.md

+16-13
Original file line numberDiff line numberDiff line change
@@ -11,21 +11,26 @@
1111
cd vscode
1212
git checkout <git tag or branch>
1313
5. Install project dependencies
14-
yarn
15-
14+
npm install
15+
6. Get back to the repository root folder
16+
cd ..
17+
1618
## Building
17-
Building the exensions from VS Code is done simply with
19+
20+
Building the exensions from VS Code is done simply from this repository root folder with
1821

1922
yarn build:extensions
2023

2124
This will compile a production ("minified") version of the built-in extensions into the `vscode/.build` folder. In order to produce unminified versions for debugging,
2225
you will need to edit the build script at `vscode/build/lib/extensions.js`. Find the line that creates the webpack config. It should look like this:
23-
```
24-
const webpackConfig = {
25-
...config,
26-
...{ mode: 'production' }
27-
};
28-
```
26+
27+
```javascript
28+
const webpackConfig = {
29+
...config,
30+
...{ mode: 'production' }
31+
};
32+
```
33+
2934
Remove part saying `mode: production` and redo the build
3035

3136
## Packaging
@@ -39,15 +44,13 @@ Once we have built our extensions, we can packge them into `*.vsix`-files using
3944
The script will produce `*.vsix` files in a folder called `./dist`. The vsix files will be named like `<name>-<vscode-version>.vsix`. Note that the publisher (msvscode)
4045
is not included.
4146

42-
If you want to create a prerelease version, you can do so by invoking
47+
If you want to create a prerelease version, you can do so by invoking
4348

4449
yarn package-vsix:next
4550

4651
This will generate `*.vsix` files of the form `<name>-<vscode-version>-next.<commit hash>.vsix`
4752

48-
**Implementation Note:** the VS Code build process puts some shared depenencies in a `node_modules` folder which is located in the "extensions" folder at run time. In order to
49-
produce self-contained extensions, we need to include those modules (at the time, it's the typescript language server) into the packaged extensions (currently for `typescript-language-features` and `html-language-features`). The code doing this is located in `src/package-vsix.js`. We also need to patch the `typescript-language-features`
50-
extension because it contains a hard-code reference to `../node_modules`.
53+
**Implementation Note:** the VS Code build process puts some shared dependencies in a `node_modules` folder which is located in the "extensions" folder at run time. In order to produce self-contained extensions, we need to include those modules (at the time, it's the typescript language server) into the packaged extensions (currently for `typescript-language-features` and `html-language-features`). The code doing this is located in `src/package-vsix.js`. We also need to patch the `typescript-language-features` extension because it contains a hard-code reference to `../node_modules`.
5154

5255
### Creating the built-ins extension-pack
5356

doc/Publishing.md

+41-38
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
# Publishing VS Code built-in Extensions for a given VS Code Version
2+
23
Publishing the VS Code built-in extensions for a given relase of VS Code entails multiple steps (in order)
34

45
1. Perform IP-checks with the Eclipse foundation for the extensions included in the VS Code repo ("builtin")
@@ -7,8 +8,9 @@ Publishing the VS Code built-in extensions for a given relase of VS Code entails
78
4. Publish the extensions from the VS Code repo to open-vsx.org
89

910
## IP checks for VS Code built-ins
10-
To prepare for the IP checks, you'll have to perform the setup steps from [Building.md](./Building.md#setup). Now we need
11-
to first run the [dash-licenses](https://github.com/eclipse/dash-licenses) tool to check the dependencies of the bulit-in
11+
12+
To prepare for the IP checks, you'll have to perform the setup steps from [Building.md](./Building.md#setup). Now we need
13+
to first run the [dash-licenses](https://github.com/eclipse/dash-licenses) tool to check the dependencies of the built-in
1214
extensions for compatibility with the Theia license. There are a couple of package scripts helping with this: the following sequence downloads the dash-licenses jar to the current directory and then runs the `dash-licenses` for all relevant extensions in the `vscode/extensions` directory.
1315

1416
yarn download:dash-licenses
@@ -18,8 +20,8 @@ This will run the dash-licenses tool an all extensions in the VS Code repo. To a
1820

1921
yarn ip-check:builtin --token <your gitlab token>
2022

21-
Any issues will show up as opened by you (or the account owning the token) at https://gitlab.eclipse.org/eclipsefdn/emo-team/iplab. In general, it's a good idea to wait for the
22-
IP tickets to be closed before publishing the built-in. Technically, this restriction applies to publishing the built-ins as part of an Eclipse project artifact like Theia IDE.
23+
Any issues will show up as opened by you (or the account owning the token) at <https://gitlab.eclipse.org/eclipsefdn/emo-team/iplab>. In general, it's a good idea to wait for the
24+
IP tickets to be closed before publishing the built-in. Technically, this restriction applies to publishing the built-ins as part of an Eclipse project artifact like Theia IDE.
2325
Now it's time to open an ip-ticket for the source of the VS Code built-ins themselves.
2426

2527
Generate a source zip of the extensions folder. You can use a package script that will prune test extensions and test folders from the source:
@@ -28,67 +30,68 @@ Generate a source zip of the extensions folder. You can use a package script tha
2830

2931
This will `git clean` all extension directories and generate a zip file named like so: `vscode-builtins-<version>.src.zip`
3032

31-
Open an issue that looks like this: https://gitlab.eclipse.org/eclipsefdn/emo-team/iplab/-/issues/11676. Use the template "vet third party" on the new issue and fill in the templata liek in the example issue. Attach the source file generated in step one as "source". Since there is no real "clearlydefined id" for the built-ins, we set the title of the issue to "project/ecd.theia/-/vscode-builtin-extensions/<VS Code version>"
33+
Open an issue that looks like this: <https://gitlab.eclipse.org/eclipsefdn/emo-team/iplab/-/issues/11676>. Use the template "vet third party" on the new issue and fill in the templata liek in the example issue. Attach the source file generated in step one as "source". Since there is no real "clearlydefined id" for the built-ins, we set the title of the issue to "project/ecd.theia/-/vscode-builtin-extensions/<VS Code version>"
3234

3335
## IP checks for external VS Code built-ins
36+
3437
We now have to perform the IP checks for the "external builtins". These are extensions which are not developed as part of the VS code repository, but which are still included as part of the
3538
VS Code product. They are described in the `product.json` file which lives at the root of the VS Code repository. There is a package script which will clone the relevant repos and check out
3639
the correct tag into a folder named `external-builtins`.
3740

38-
yarn get-external-builtin
41+
yarn get-external-builtins
3942

4043
We now have to run the checks for the dependencies of those extensions:
4144

4245
yarn ip-check:external --token <your gitlab token>
4346

4447
Again, this will open issues with the Eclipse IP issue tracker. Once this is done, it's time to open an ip-check issue for the content of each of the external built ins.
45-
For extensions from github, it's usually enough to open a "vet third party" issue with just the project in the details, like this one: https://gitlab.eclipse.org/eclipsefdn/emo-team/iplab/-/issues/14430. The title should be the clearlydefined id of the form `git/github/<github org>/<project>/v<version>`. The IP-check bot is usually able to download the source from the github release page on its own. In the issue template, just fill in the "project" field.
46-
If the IP-check bot cannot figure out the source (it will ask for source in a comment on the issue), you can zip up the source of all external built-ins into files of the form `<publisher>.<name>-<version>.src.zip>` with a package script:
48+
For extensions from github, it's usually enough to open a "vet third party" issue with just the project in the details, like this one: <https://gitlab.eclipse.org/eclipsefdn/emo-team/iplab/-/issues/14430>. The title should be the clearlydefined id of the form `git/github/<github org>/<project>/v<version>`. The IP-check bot is usually able to download the source from the github release page on its own. In the issue template, just fill in the "project" field.
49+
If the IP-check bot cannot figure out the source (it will ask for source in a comment on the issue), you can zip up the source of all external built-ins into files of the form `<publisher>.<name>-<version>.src.zip>` with a package script:
4750

4851
yarn archive:external
4952

5053
You can then drag the relevant zip into the gitlab issue.
5154

5255
## Produce the VS Code built-ins
5356

54-
Building and packaging the built-ins is described in [Building.md](./Building.md).
57+
Building and packaging the built-ins is described in [Building.md](./Building.md).
5558

5659
## Testing
5760

58-
This section assumes you have a local clone of the [main Theia repo](https://github.com/eclipse-theia/theia). Please refer to the Theia documentation for instructions on how to build and
61+
This section assumes you have a local clone of the [main Theia repo](https://github.com/eclipse-theia/theia). Please refer to the Theia documentation for instructions on how to build and
5962
run Theia. Some built-ins may refuse to run if the VS Code API version reported by Theia is lower that what they require. If Theia's default API verison has not been updated yet, you can
6063
force a newer version by either setting the `VSCODE_API_VERSION` environment variable or by passing the option `--vscode-api-version <major>.<minor>.<patch>`
6164

6265
If already present, delete folder `plugins` in your local Theia repo folder. We will instead use the built-ins we previously built
6366

64-
```bash
65-
rm -rf plugins
66-
mkdir plugins
67-
```
67+
```bash
68+
rm -rf plugins
69+
mkdir plugins
70+
```
6871

6972
Copy the builtin extension `*.vsix` files built above to Theia's `extensions` folder (typically `~/.theia/extensions`)
7073

71-
```bash
72-
cp -a dist/* ~/.theia/extensions # adjust according to where your .theia folder resides
73-
```
74+
```bash
75+
cp -a dist/* ~/.theia/extensions # adjust according to where your .theia folder resides
76+
```
7477

7578
Get rid of a few builtins that will interfere with testing (note: we keep these extensions where they were generated, but remove them from our test Theia application):
7679

77-
```bash
78-
cd theia # back to theia repo
79-
rm -rf plugins/ipynb-*
80-
rm -rf plugins/extension-editing-*
81-
```
80+
```bash
81+
cd theia # back to theia repo
82+
rm -rf plugins/ipynb-*
83+
rm -rf plugins/extension-editing-*
84+
```
8285

83-
To test vscode builtin git, we need to remove the Theia-specific git extension from the example application, for this, remove the line referring to
86+
To test vscode builtin git, we need to remove the Theia-specific git extension from the example application, for this, remove the line referring to
8487
`"@theia/git": "<Theia version>"` from the `package.json` of the Theia example you use for testing.
8588

8689
Rebuild the example and start Theia:
8790

88-
```bash
89-
yarn && yarn browser build
90-
yarn browser start
91-
```
91+
```bash
92+
yarn && yarn browser build
93+
yarn browser start
94+
```
9295

9396
Note that startup will take a bit longer than usual while Theia unzips the *.vsix files to `~/.theia/deployedPlugins`.
9497

@@ -113,26 +116,26 @@ Now it's time to open a PR against master. The convention is to create a branch
113116

114117
git checkout -b <major>.<minor>.<patch> # replace the version here wiht the VS Code version, for example "1.88.1"
115118

116-
Now commit all changes you had to make to get the built-ins to correctly build, **including the `vscode` folder**. Adding `vscode` will update the submodule configuration n this repo to
119+
Now commit all changes you had to make to get the built-ins to correctly build, **including the `vscode` folder**. Adding `vscode` will update the submodule configuration n this repo to
117120
check out the correct version of VS Code upon `git submodule update`. The convention is to make a single commit named `changes for builtins v<major>.<minor>.<patch>`. Open a PR and have it reviewed as usual.
118121

119-
# Publishing to openvsx.org
122+
## Publishing to openvsx.org
120123

121-
**Before publishing to open-vsx, all issues opened in [internal](#ip-checks-for-vs-code-built-ins) and [external](#ip-checks-for-external-vs-code-built-ins) should be closed.**
124+
**Before publishing to open-vsx, all issues opened in [internal](#ip-checks-for-vs-code-built-ins) and [external](#ip-checks-for-external-vs-code-built-ins) should be closed.**
122125
Please work with the Eclipse Foundation staff and the Theia community if there are problems!
123126

124-
Publishing is done using GitHub Actions. In the vscode-builtin-extensions repo, a publish token for open-vsx.org has been set, that can be used to publish under the identity of the openvsx publish bot.
127+
Publishing is done using GitHub Actions. In the vscode-builtin-extensions repo, a publish token for open-vsx.org has been set, that can be used to publish under the identity of the openvsx publish bot.
125128

126-
There are four workflows in this repo.
129+
There are four workflows in this repo.
127130

128-
* **publish-vsx-latest.yml:** Will check out the latest tagged version of VS Code and builds and packages a release version of the extensions and extension pack
129-
* **publish-vsix-next.yml:** Will check out the VS Code `main branch` and build a prerelase version.
131+
- **publish-vsx-latest.yml:** Will check out the latest tagged version of VS Code and builds and packages a release version of the extensions and extension pack
132+
- **publish-vsix-next.yml:** Will check out the VS Code `main branch` and build a prerelase version.
130133

131134
both these workflows are triggered on a regular schedule and upon push to the master branch
132135

133-
* **publish-vsx-specific-latest:** This action is triggered upon pushes to the branch `ovsx-publish`. It checks out the version of VS Code that is checked in as a submodule
136+
- **publish-vsx-specific-latest:** This action is triggered upon pushes to the branch `ovsx-publish`. It checks out the version of VS Code that is checked in as a submodule
134137
on the branch and creates and packages a release version of the built-ins. It then publishes the built-ins and extension pack to the open-vsx registry.
135-
* **publish-vsx-specific-next:** This action is triggered upon pushes to the branch `ovsx-publish-next`. It checks out the version of VS Code that is checked in as a submodule
138+
- **publish-vsx-specific-next:** This action is triggered upon pushes to the branch `ovsx-publish-next`. It checks out the version of VS Code that is checked in as a submodule
136139
on the branch and creates and packages a prerelease version of the built-ins. It then publishes the built-ins and extension pack to the open-vsx registry.
137140

138141
For "regular" vs. "prerelease" vesions see [Building.md](./Building.md))
@@ -147,10 +150,10 @@ In order to publish updated built-ins, we replace the contents of the `osvx-publ
147150
Go in the [Actions](https://github.com/eclipse-theia/vscode-builtin-extensions/actions) tab to observe the publishing progress.
148151

149152
The publish workflow may fail, usually because the prerequisites for building the built-ins have changed. In this case, make the necessary change in the relevant workflows. In general, the setup should be aligned with the CI setup of VS Code:
150-
https://github.com/microsoft/vscode/blob/main/.github/workflows/ci.yml#L107. **Make sure you updated all the workflows that may be affected.**
153+
<https://github.com/microsoft/vscode/blob/main/.github/workflows/ci.yml#L107>. **Make sure you updated all the workflows that may be affected.**
151154
Now push to `osvx-publish` again. Repeat until the publish succeeds.
152155

153-
Now we make a copy of the publish branch "for the record":
156+
Now we make a copy of the publish branch "for the record":
154157

155158
git checkout -b old-ovsx-publish-<major>.<minor>.<patch>
156159
git push origin

package.json

+2-1
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,9 @@
11
{
2+
"name": "vscode-builtin-extensions",
23
"private": true,
34
"license": "EPL-2.0 OR GPL-2.0 WITH Classpath-exception-2.0",
45
"scripts": {
5-
"build:extensions": "yarn --cwd vscode && yarn compile:extensions && yarn bundle:extensions",
6+
"build:extensions": "npm --prefix vscode install && yarn compile:extensions && yarn bundle:extensions",
67
"compile:extensions": "cross-env NODE_OPTIONS=--max-old-space-size=8192 node ./src/compile.js",
78
"bundle:extensions": "cross-env NODE_OPTIONS=--max-old-space-size=8192 node ./src/bundle.js",
89
"publish:vsix": "node ./src/publish-vsix.js",

src/compile.js

+3-3
Original file line numberDiff line numberDiff line change
@@ -41,9 +41,9 @@ async function createMissingLockFiles(extensionsPath) {
4141

4242
for (let subFolderName of subFolderNames) {
4343
let subFolderPath = path.join(extensionsPath, subFolderName);
44-
let yarnLockExists = fs.existsSync(path.join(subFolderPath, 'yarn.lock'));
45-
if (!yarnLockExists) {
46-
await run('yarn', ['install'], subFolderPath );
44+
let packageLockExists = fs.existsSync(path.join(subFolderPath, 'package-lock.json'));
45+
if (!packageLockExists) {
46+
await run('npm', ['install'], subFolderPath );
4747
}
4848
}
4949
}

src/create-extension-pack.js

+4-4
Original file line numberDiff line numberDiff line change
@@ -98,11 +98,11 @@ const externalBuiltins = ['ms-vscode.js-debug-companion', 'ms-vscode.js-debug'];
9898
fs.writeFileSync(licensePath, generateLicense());
9999
fs.writeFileSync(readmePath, generateReadme());
100100

101-
await createYarnLock(packFolderPath);
101+
await createPackageLock(packFolderPath);
102102
await vsce.createVSIX({
103103
'cwd': packFolderPath,
104104
'packagePath': dist(),
105-
'useYarn': true
105+
'useYarn': false
106106
});
107107

108108
async function resolveExtensions(extensionsArr) {
@@ -163,8 +163,8 @@ const externalBuiltins = ['ms-vscode.js-debug-companion', 'ms-vscode.js-debug'];
163163
/**
164164
* @param {string} folderPath
165165
*/
166-
async function createYarnLock(folderPath) {
167-
await run('yarn', ['install'], folderPath);
166+
async function createPackageLock(folderPath) {
167+
await run('npm', ['install'], folderPath);
168168
}
169169

170170
function generateLicense() {

src/package-vsix.js

+1-1
Original file line numberDiff line numberDiff line change
@@ -170,7 +170,7 @@ const repository = {
170170
await vsce.createVSIX({
171171
'cwd': extensions(extension),
172172
'packagePath': dist(),
173-
'useYarn': true,
173+
'useYarn': false,
174174
'allowStarActivation': true
175175
});
176176
result.push('successfully packaged: ' + pck.name);

src/publish-vsix.js

+1-1
Original file line numberDiff line numberDiff line change
@@ -83,7 +83,7 @@ async function publishExtension(vsix) {
8383
console.log(`Extension ${extension} v${version} is already published - skipping!`);
8484
} else {
8585
console.log('Publishing: ', dist(vsix), ' ...');
86-
const results = await ovsx.publish({ extensionFile: dist(vsix), yarn: true });
86+
const results = await ovsx.publish({ extensionFile: dist(vsix), yarn: false });
8787
for (const result of results) {
8888
if (result.status === 'rejected') {
8989
message = `Error(s) Generated when publishing ${extension} v${version}!`;

0 commit comments

Comments
 (0)