Test runner matching every .ts and .js if glob is not provided

[marco-ippolito]

Version

Test on v23.6.0 and v22.10.0

Platform

All

Subsystem

test_runner

What steps will reproduce the bug?

Consider this folder structure:

└── test
    ├── fixtures
    │   ├── boom.js
    │   └── boom.ts
    └── index.test.js

When running node --test the test runner will execute ./test/fixtures/boom.js.
In v23 it will also execute ./test/fixtures/boom.ts, since --experimental-strip-types has been unflagged.

marcoippolito@marcos-MacBook-Pro test % node --test /Users/marcoippolito/Documents/projects/test/test/fixtures/boom.js:1 throw new Error('boom'); ^

Error: boom
at Object. (/Users/marcoippolito/Documents/projects/test/test/fixtures/boom.js:1:7)
at Module._compile (node:internal/modules/cjs/loader:1739:14)
at Object..js (node:internal/modules/cjs/loader:1904:10)
at Module.load (node:internal/modules/cjs/loader:1473:32)
at Function._load (node:internal/modules/cjs/loader:1285:12)
at TracingChannel.traceSync (node:diagnostics_channel:322:14)
at wrapModuleLoad (node:internal/modules/cjs/loader:234:24)
at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:151:5)
at node:internal/main/run_main_module:33:47

Node.js v23.6.0
✖ test/fixtures/boom.js (36.07275ms)
'test failed'

/Users/marcoippolito/Documents/projects/test/test/fixtures/boom.ts:1
throw new Error('boom TS');
^

Error: boom TS
at Object. (/Users/marcoippolito/Documents/projects/test/test/fixtures/boom.ts:1:7)
at Module._compile (node:internal/modules/cjs/loader:1739:14)
at Object.loadTS [as .ts] (node:internal/modules/cjs/loader:1831:10)
at Module.load (node:internal/modules/cjs/loader:1473:32)
at Function._load (node:internal/modules/cjs/loader:1285:12)
at TracingChannel.traceSync (node:diagnostics_channel:322:14)
at wrapModuleLoad (node:internal/modules/cjs/loader:234:24)
at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:151:5)
at node:internal/main/run_main_module:33:47

Node.js v23.6.0
✖ test/fixtures/boom.ts (62.136209ms)
'test failed'

✔ should return true (0.3725ms)
ℹ tests 3
ℹ suites 0
ℹ pass 1
ℹ fail 2
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 72.075583

✖ failing tests:

test at test/fixtures/boom.js:1:1
✖ test/fixtures/boom.js (36.07275ms)
'test failed'

test at test/fixtures/boom.ts:1:1
✖ test/fixtures/boom.ts (62.136209ms)
'test failed'

How often does it reproduce? Is there a required condition?

Always

What is the expected behavior? Why is that the expected behavior?

Maybe this is intended behavior but I'd would expect matching .test.js.
I think it's an undesired side effect to execute everything.
I immagine breakages due to a lot of .ts fixtures being executed.

What do you see instead?

Everything is executed.

Additional information

I know changing this would be a breaking change, but I dont think it's sane as it is.

[targos]

It matches the files because they are under a test folder.
The default matching behavior is documented here: https://nodejs.org/docs/latest/api/test.html#running-tests-from-the-command-line

[pmarchini]

As correctly pointed out by @targos, this behaviour is intended.
I believe this is mostly a logic inherited from tapjs.
Checking other tools, this behaviour is different, and only **/*.{test,spec}.?(c|m)[jt]s?(x) are being matched.

IMHO, it is common to have test helpers/fixtures under the test directory.
For this reason, I believe we should consider changing this behaviour, even if it means introducing a breaking change.

[cjihrig]

Mocha also checks for a test/ directory. Independent from that, I would be -1 to changing this default. It's not just a breaking change, but an aggressive breaking change because people often run their tests across multiple versions of Node. The introduction of globbing (while a great change) has caused many headaches between Node 20 and 22. This change would not have nearly the same upside.

[marco-ippolito]

I think executing only .test.js/ts would be a better default behavior.

[AugustinMauroy]

I agree that a huge breaking change. But I'm also convinced that **/**.test.{cjs,mjs,js} or **/**-test.{cjs,mjs,js} is the healthiest for the ecosystem. Knowing that the test runner has not yet been given too much attention

[p-mcgowan]

Without changing the default behaviour, perhaps a flag --test-file-pattern=dist/test/*.spec.js instead?

Normally the positional args would be fine, but if using with npm test you cannot default it in the package script and still pass in flags afterwards:

"scripts": {
  "test": "node --test 'dist/test/*.spec.js'"
}

npm test -- --test-only 'dist/test/some-folder/*.spec.js'

In this case, the --test-only flag is ignored as it's after positionals.

Alternatively, allowing flags after positionals would also work, but the default pattern flag would be nice if you could additionally pass in a test name to run only that file instead of .only'ing it.

eg.
if test folder contains:

• a.spec.js
• b.spec.js
• setup.js
• something.js

running node --test --test-file-pattern='*.spec.js' should run a and b, whereas node --test --test-file-pattern='*.spec.js' test/a.spec.js would only run a.

Not sure how feasible that is, but something like that behaviour would be nice.

Related: #51384

[marco-ippolito]

As supected unflagging experimental-strip-types in v22 is a breaking change because it will execute .ts if glob is not provided. Ideas?
We just disable .ts by default for just v22? Id do it for other lines too along with .js (but it would be a breaking change)
.spec.ts and .test.ts are fine imho

[cjihrig]

.spec.ts and .test.ts are fine imho

I don't think there is a scenario where you can execute any additional files and be able to guarantee it won't break anyone. IMO the benefit to unflagging in the same way as v23 is worth it. I think users will be far more annoyed by the lack of "easy" TypeScript support in the test runner for the rest of v22's lifetime than they will be over possibly having to rename fixtures one time.

[cjihrig]

FWIW, we didn't backport globs in the test runner to v20 because there was a remote chance that the glob would match some existing pattern. In retrospect, I think that was a massive mistake (which we could technically still rectify), as it introduced hard to work around changes between v20 and v22. I don't think we should make the same mistake here.

[marco-ippolito]

Possible options are:

• 👍keep as it is, and unflag (some users will experience a breaking change) (example ./test/fixture/foo.ts will be executed)

• ❤️ remove the most problematic glob */test/**/*.{cts,mts,ts} from v23 too, this should remove a big chunk but still some user might experience breaking changes if they use .test.ts file (super niche case imho)

• 🚀 not unflag

• 🎉 disable .ts .test.ts etc... matching completely

[cjihrig]

There are other options too (ignore a test/fixtures directory, only process .spec.ts, etc.), but I think we need to pick a general direction before nailing down specifics like that.

[marco-ippolito]

I thin the main problem I heard users talkabout are:

• TypeScript projects with .test.ts files in the src folder, and once transpiled the execute the .test.js files. Now node --test that will execute both.

• ./test/fixtures/.ts were previously ignored now executed

Another issue is that there is no way to run the same command to exclude .ts files in v20 and v22 (Immagine a library supporting v20 and v22 running tests, v22 enables typescript and they have to disable it, they cannot use the same command for v22 and v20 because requires glob and they cant --no-experimental-strip-types)

[marco-ippolito]

Or backport glob to v20 there is a special semver minor in v20 coming could be a good time 😃

[cjihrig]

+1000 to backporting globs to v20.

[marco-ippolito]

It needs a volunteer to do the manual backport
v20 special minor release is next week so we are on a tight schedule

[AugustinMauroy]

+1 to backborting globs to v20 I needed for my project. my package support v20 but isn't tested on it because we specify tests using glob

[cjihrig]

#47653 is the change that needs to be backported.

[marco-ippolito]

But its a semver major 🫨 I dont think its possible then

[cjihrig]

Yea, that's what I was referring to in #56546 (comment). If we are aiming for 100% guaranteed non-breaking behavior then we can't backport that or process .ts files in the test runner on v22.

However, I don't believe that option is what is best for end users. If one person has a filename containing an asterisk, I don't think that's worth keeping globs out of v20 🤷

[marco-ippolito]

Pinging @nodejs/tsc as this is a tricky situation
Maybe safest option is disable .ts .test.ts etc... matching completely

[cjihrig]

It's worth noting that --experimental-strip-types was unflagged in v23.6.0 (not a semver major) and it had the same implications as unflagging it in v22.

[remcohaszing]

The focus seems to be on test fixtures which happen to be TypeScript files. This is an edge case. I feel like this discussion is missing the most important context: The most basic TypeScript setup. An example of such a project is remark-mdx-frontmatter.

TypeScript has historically been and still is a compile-time language. You compile it to JavaScript, you then run the emitted JavaScript code. This is no different for tests.

├── dist  // This folder is generated
│   ├── some-pkg.test.d.ts
│   ├── some-pkg.test.d.ts.map
│   ├── some-pkg.test.js
│   ├── some-pkg.test.js.map
│   ├── some-pkg.d.ts
│   ├── some-pkg.d.ts.map
│   ├── some-pkg.js
│   └── some-pkg.js.map
├── src
│   ├── some-pkg.test.ts
│   └── some-pkg.ts
├── package.json
└── tsconfig.json

In this project package.json probably contains something like this:

{
  "scripts": {
    "prepack": "tsc --build",
    "pretest": "tsc --build",
    "test": "node --test --enable-source-maps"
  }
}

With the new glob pattern, not only dist/some-pkg.test.js is matched, but also src/some-pkg.test.ts. Depending on how some-pkg.test.ts is written, this may have the following implications:

1. If src/some-pkg.test.ts contains the following code, the tests run twice:

   import { someExport } from 'some-pkg'

2. If src/some-pkg.test.ts contains the following code, module resolution fails:

   import { someExport } from './some-pkg.js'

3. If src/some-pkg.test.ts contains TypeScript syntax Node.js doesn’t understand, loading the test module fails.

With Node.js supporting TypeScript files, TypeScript is no longer just a compile-time language. You need a different TypeScript configuration. You need to compile source files, but not tests. I imagine a project layout like this:

├── dist  // This folder is generated
│   ├── some-pkg.d.ts
│   ├── some-pkg.d.ts.map
│   ├── some-pkg.js
│   └── some-pkg.js.map
├── src
│   └── some-pkg.ts
├── test
│   └── some-pkg.test.ts
├── package.json
├── tsconfig.build.json  // Build the src into dist
└── tsconfig.json        // Type-check tests, noEmit. Use project references to reference tsconfig.build.json

The problem is, the first setup works with projects using Node.js versions up to 22. The second setup works with projects using Node.js 23 and greater. Libraries however, need to run tests against multiple Node.js versions. Many libraries are still even compatible with Node.js versions that are EOL. Breaking compatibility with a Node.js version, even if it’s EOL, would require a semver major release of the library.

Some vague ideas I have:

• Don’t support TypeScript test file discovery. I don’t like this. I really look forward to the point where we can just run TypeScript tests.
• Allow configuring the test runner to discover TypeScript files in a way that works with maintained Node.js versions, but also doesn’t break EOL Node.js versions. This means CLI flags are not an option, but maybe an environment variable or configuration file is.

[cjihrig]

Thanks for the input @remcohaszing!

Libraries however, need to run tests against multiple Node.js versions.

I agree that this is important and that we should minimize differences in test discovery between versions.

Allow configuring the test runner to discover TypeScript files in a way that works with maintained Node.js versions, but also doesn’t break EOL Node.js versions.

I think the easiest way to accomplish this and address the previous point is to just backport glob support to v20. Unfortunately, since it's semver major, the TSC would need to bless that.

[remcohaszing]

I definitely agree backporting glob support to v20 would be a great idea. However, this still means you can’t test a library against versions 16, 18, 20, 22, and 23 with the same setup. I personally test against even majors only, so I really read version 23 as some future version.

Although I guess it’s a somewhat acceptable middle ground to delay testing against Node.js 24 until I decide to drop support for Node.js 18. The odds of JavaScript-only packages breaking in newer Node.js versions are relatively small.

[marco-ippolito]

16 is EOL and afaik does not have the test runner and 18 is EOL in 1 month.

[remcohaszing]

Node.js 16 does have the test runner. I’m aware it’s EOL, that’s my point. I absolutely push for people to use modern Node.js versions. But for a library removing support for an EOL Node.js version is still a breaking change. Node.js going EOL shouldn’t mean all npm packages have to make a semver major release. This can be decided on a per-package basis.

Note that I’m not saying libraries should support every Node.js version indefinitely, but there’s a middle ground somewhere.

[marco-ippolito]

Node.js 16 does have the test runner. I’m aware it’s EOL, that’s my point. I absolutely push for people to use modern Node.js versions. But for a library removing support for an EOL Node.js version is still a breaking change. Node.js going EOL shouldn’t mean all npm packages have to make a semver major release. This can be decided on a per-package basis.

Note that I’m not saying libraries should support every Node.js version indefinitely, but there’s a middle ground somewhere.

v16 is experimental so it does really matter. I disagree we should care about EOL versions or libraries supporting EOL version. And honestly its a very niche issue.

[remcohaszing]

I would argue most packages support EOL Node.js versions during some points in their lifetime, often several versions. Of course you may draw your own conclusions from this, but I wouldn’t say this is a niche.

[p-mcgowan]

Would allowing flags to come after positional glob / file patterns in the test runner resolve these issues?
That would allow npm test scripts to simply pass in 'dist/test/*.spec.js' or 'test/**/*.spec.ts' in npm run scripts (since it's likely they will have a few other default flags).
Backporting a different glob strategy seems like it might help in some cases, but not resolve the issue entirely. Or am I missing something?

[mcollina]

This is the reason why I created https://www.npmjs.com/package/borp among others: standardizing the test runner across multiple Node.js versions, ts supports and a few other bits.

I'm happy to:

1. review and merge PRs that uses amaro/ts strip types by default
2. review and merge PRs that keep it in line with the current node --test flow
3. transfer it to the org

Or we could just use the npm test module for that (or even just use borp as a start for that).

[marco-ippolito]

I opened a PR to change the standard glob for .ts
#57359
I know its conservative but imho its better to have a conservative default than unexpected matching

[cjihrig]

Or we could just use the npm test module for that

It would be great to put the test module to use.

Backporting a different glob strategy seems like it might help in some cases, but not resolve the issue entirely. Or am I missing something?

The problem is that globs don't exist at all in v20. It wouldn't be a different glob strategy.

[AugustinMauroy]

Just something that we didn't take a look is what other runtime does.
Deno: https://docs.deno.com/runtime/reference/cli/test/
Bun: https://bun.sh/docs/cli/test#run-tests