Use Circle CI container parallelization to speed up building/deployment

[kba]

I just came across https://circleci.com/docs/2.0/parallelism-faster-jobs IIUC we could spend part of the 2500 credits in the free tier (which is basically equivalent to 2500 minutes of computing) to parallelize some steps in our CI pipeline. Esp. when we start working on integration tests and for building at least the git/non-git variants, this might be worth investigating.

[stweil]

GitHub actions also allows parallel actions. I use that to build 32 and 64 bit Tesseract installers at the same time.

[bertsky]

IIUC we could spend part of the 2500 credits in the free tier (which is basically equivalent to 2500 minutes of computing) to parallelize some steps in our CI pipeline.

So is this about getting us faster builds, or investing credits better? (I didn't read anything about a credit-parallelism relationship.) Doesn't it save us credits that we fail earlier due to sequential build (if CI fails)?

[kba]

GitHub actions also allows parallel actions. I use that to build 32 and 64 bit Tesseract installers at the same time.

Nice, I'm also open to GitHub actions, I'll look into it.

I didn't read anything about a credit-parallelism relationship

IIUC n parallelism of a build (as opposed to concurrent builds which is a different feature) results in n times as many credits (i.e. computing time in minutes). Saving credits was not my point, I'm not in the business of promoting CircleCI and if we reach a limit on the free tier, there are alternatives like GH action, Travis, Semaphore, Buddy or possibly mirroring to gitlab.com or bitbucket and use their jobs/pipelines systems...

[kba]

Pro GitHub actions: We could work together on the CI/CD pipeline. How Circle CI ACL maps to GitHub's ACL is intransparent and only I can currently manage environment variables and configuration outside the CI config YAML.

[cneud]

GitHub actions only offers 2000 minutes in the free tier in comparison though - otherwise it would seem very beneficial switching?

(LOL)

Jobs that run on Windows and macOS runners that GitHub hosts consume minutes at 2 and 10 times the rate that jobs on Linux runners consume.

[bertsky]

I don't think we still need parallelism after 9e1dd88 already makes the various variants build in parallel. We could use make -j for each job to harvest more of its CPU resources – but there's always a risk that subtle errors in the dependency declarations fail or race such parallel builds.

Build time for the largest variant is still under 1h now. I'd say we can close this.

[bertsky]

Since we crossed the 1h barrier recently and subsequently had to add -j as default, let's close.