Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

modify WEBPACKER PARALLEL environment variable. #2850

Closed

Conversation

masato-hi
Copy link
Contributor

Will be possible to specify the WEBPACKER PARALLEL environment variable from the command line.

We also added a document about WEBPACKER_PARALLEL.

The master branch is currently working on 6.x, so we need to merge after merging this pull request.
#2839

I'm not good at English, so I'm looking for opinions on better documentation.

ref: #2093

@@ -37,11 +38,24 @@ const canProcess = (rule, fn) => {
return null
}

const asBoolean = (str) => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @masato-hi Could you please use the above function isBoolean (line 2)?

Copy link
Contributor Author

@masato-hi masato-hi Jan 5, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for review.
The isBoolean function cannot handle "false" correctly

@pedrofurtado
Copy link
Member

Hey, @masato-hi ! 👋

Thanks for your contribution!

For this purpose (modify WEBPACKER_PARALLEL environment variable for a specific environment, for example), this works for your scenario?

  • Edit your config/webpack/[ENVIRONMENT].js and put at the top the desired value of WEBPACKER_PARALLEL, like this snippet code below:
process.env.WEBPACKER_PARALLEL = 'desired-value'
// ... existing code of file ...

Or this not works for you, and the PR in this case solves some problem related to it?

Let me know what you think about 🤝 🍻

@masato-hi masato-hi force-pushed the modify-webpacker-parallel-env branch from 549ec62 to 7ec0be8 Compare February 13, 2021 10:26
@masato-hi
Copy link
Contributor Author

@pedrofurtado Thanks for confirming.

It looks like it's working fine in my environment. 🤔 (The sed command installed on MacOSX may not execute the following code successfully.)

rails new myapp --webpack
cd myapp
sed -i -e "s/gem 'webpacker', '~> 5.0'/gem 'webpacker', github: 'masato-hi\/webpacker', branch: 'modify-webpacker-parallel-env'/" Gemfile
bundle install
yes | bundle exec rake webpacker:install
sed -i -e 's/"@rails\/webpacker": ".*"/"@rails\/webpacker": "masato-hi\/webpacker#modify-webpacker-parallel-env"/' package.json
yarn
echo "
process.env.NODE_ENV = process.env.NODE_ENV || 'production'
process.env.WEBPACKER_PARALLEL = '2'

const webpackConfig = require('./base')

console.log(webpackConfig.optimization.minimizer[0])
module.exports = webpackConfig
" > config/webpack/production.js
rm .browserslistrc # cheap tricks for "Error [BrowserslistError]: contains both .browserslistrc and package.json with browsers"
RAILS_ENV=production bundle exec rake webpacker:clean webpacker:compile

Output:

Compiling...
Compiled all packs in /tmp/myapp/public/packs
TerserPlugin {
  options: {
    test: /\.[cm]?js(\?.*)?$/i,
    extractComments: true,
    parallel: 2,
    include: undefined,
    exclude: undefined,
    minify: undefined,
    terserOptions: {
      parse: [Object],
      compress: [Object],
      mangle: [Object],
      output: [Object]
    }
  }
}
asset manifest.json 23 bytes [emitted]
webpack 5.21.2 compiled successfully in 26 ms

The value of "parallel" is 2.

The configuration only affects production builds.
You also need to set process.env.WEBPACKER_PARALLEL before require('@rails/webpacker') / require('./ base') is executed.

@masato-hi
Copy link
Contributor Author

@gauravtiwari
The changes in docs/troubleshooting.md have been merged into master and are now reflected.
What do you think?

Copy link
Contributor

@olleolleolle olleolleolle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I made some inline suggestions, hope this helps.

Comment on lines +124 to +126
Check the link for details.

https://webpack.js.org/plugins/terser-webpack-plugin/#parallel
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Check the link for details.
https://webpack.js.org/plugins/terser-webpack-plugin/#parallel
[`terser-webpack-plugin` documentation](https://webpack.js.org/plugins/terser-webpack-plugin/) has more details.

@@ -111,6 +111,19 @@ mv $HOME/your_rails_app/node_modules/.bin/elm-make $HOME/your_rails_app/node_mod
printf "#\041/bin/bash\n\necho \"Running elm-make with sysconfcpus -n 2\"\n\n$HOME/dependencies/sysconfcpus/bin/sysconfcpus -n 2 $HOME/your_rails_app/node_modules/.bin/elm-make-old \"\$@\"" > $HOME/your_rails_app/node_modules/.bin/elm-make
chmod +x $HOME/your_rails_app/node_modules/.bin/elm-make
```
## If an error occurs or times out during a production build on Continuous Integration (CI) services such as CircleCI, CodeShip, Travis CI

The parallel option of Tersor-Plugin is enabled in the production build, which may cause memory shortage.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The parallel option of Tersor-Plugin is enabled in the production build, which may cause memory shortage.
The parallel option of TerserWebpackPlugin is enabled in the production build, which may cause memory shortage.

@@ -111,6 +111,19 @@ mv $HOME/your_rails_app/node_modules/.bin/elm-make $HOME/your_rails_app/node_mod
printf "#\041/bin/bash\n\necho \"Running elm-make with sysconfcpus -n 2\"\n\n$HOME/dependencies/sysconfcpus/bin/sysconfcpus -n 2 $HOME/your_rails_app/node_modules/.bin/elm-make-old \"\$@\"" > $HOME/your_rails_app/node_modules/.bin/elm-make
chmod +x $HOME/your_rails_app/node_modules/.bin/elm-make
```
## If an error occurs or times out during a production build on Continuous Integration (CI) services such as CircleCI, CodeShip, Travis CI
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
## If an error occurs or times out during a production build on Continuous Integration (CI) services such as CircleCI, CodeShip, Travis CI
## If an error occurs or times out during a production build

@justin808
Copy link
Contributor

@guillaumebriday, For sure, we should do something with PR. Running out of memory causes a very confusing build just seems to hang when it's swapping memory.

I don't like the current default:

   parallel: Number.parseInt(process.env.WEBPACKER_PARALLEL, 10) || true,

Defaults should be defaults, which is true for this option.

How about this:

  1. default to true
  2. all env override of process.env.WEBPACKER_PARALLEL to either be false or some lower number to set on CI machines?

https://webpack.js.org/plugins/terser-webpack-plugin/#parallel

parallel
Type: Boolean|Number Default: true

Use multi-process parallel running to improve the build speed. Default number of concurrent runs: os.cpus().length - 1.

ℹ️ Parallelization can speedup your build significantly and is therefore highly recommended.

⚠️ If you use Circle CI or any other environment that doesn't provide real available count of CPUs then you need to setup explicitly number of CPUs to avoid Error: Call retries were exceeded (see #143, #202).

@dhh dhh closed this Jan 19, 2022
@justin808
Copy link
Contributor

Feel free to reopen this PR here:
https://github.com/shakacode/shakapacker/pulls

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants