test: rework E2Es

[goosewobbler]

Reworking the E2Es so that they use a matrix-based approach and the packaged version of the service.

Closes #955

TODO:

• Update for rebuilt CI (ci: improve performance of CI job #978)
• Ensure all CI jobs run tests
• Fix E2Es matrix (test:all)
• Fix test:window multiremote variant
• Clean up scripts
• Docs

[goosewobbler]

Targetting 8.0.2 for this.

[goosewobbler]

@mato533 Just to let you know I thought I'd do the yarn hack removal in this PR, and rework the E2Es at the same time. Basically deal with all the tech debt before getting some traction on v9. Let me know what you think of this matrix approach to the E2Es, and if you have any improvements.

I decided to ditch the JS variants of the E2Es for now in order to make the rework simpler, these variants were put in place for a specific bug that was masked by the use of tsx so I'm not sure how much value we are getting from them vs. maintenance cost and build slowdown. We can always reinstate them after the rework.

[mato533]

@goosewobbler
It is awesome idea to rework E2E before v9.

Basically, The matrix approach seems to be good idea.
This service should be tested to the forge and builder. (I agree that you have decided to remove the JS variants considering into historical background)

I've put together some points that I think need improvement. I hope it helps.

• improvements (May be)

  • Number of the Sample App
    Which test target electron app is correct?

    1. An electron app which build result is based on CJS/ESM code
    2. An electron app whose source code (which before build) is written in CJS/ESM and bundled with it
       If this is correct target, we have to check the current sourcecode... Although it's named cjs, it looks like there are sample apps written in ESM.

    If the a) is correct, we may be able to reduce the number of apps from 6 sample apps to 3 because we can build both CJS and ESM from ESM code by creating properly configurations for the Rollup.(When remove the JS variants, it become 4 sample apps to 2 apps)

  • E2E concurrency
    E2E needs to run 16 patterns of E2E, 4 (Lin/Mac/Mac-uni/win) multiply 2 (builder/forge) multiply 2 (esm/cjs).
    there is 2 approach in order to execute these efficiently,.

    1. Execute parallel in multiple container while taking into account the limitations of Github Actions (The sample of this approach is ci: improve performance of CI job #978)
    2. Execute parallel in a single container, taking into account the CPU/Memory consumption of the container
       This approach is similar to current E2E for Mac and Linux.
       But current E2E is executing sequentially the scenarios (no-binary-esm -> no-binaly-cjs -> builder-esm ->....)
       This approach is intend to running multiple scenarios at the same time.
       build no-binary-esm -> run no-binary-esm
       build no-binary-cjs -> run no-binary-cjs
       build builder-esm -> run builder-esm
       ....

    In practice, The approach a) seems to be the good approach, as there are variations in performance between operating systems and build tool combinations (Windows is generally slower, Mac-Universal is slower to build, etc.). This is only my opinion, I respect your thought.

[mato533]

Incidentally, it is assumed that the dependency between sample-apps and the service will completely disappear from v9. (because wdio-electron-service will have no more preload/main scripts to import due to the IPC-Bridge replacement).

This should allow e2e app builds to run in parallel with serivce builds, which may be able to improve things in some way when v9 is released.