The ESM move

[sindresorhus]

At the end of April, Node.js 10 goes out of LTS, which means I can target Node.js 12 in my packages. Node.js 12 has full ESM support!

Required reading: https://nodejs.org/api/esm.html

I plan to migrate most of my (1K+) packages to ESM within 2021.

At the start of April, I will begin with some less popular packages, just to test the waters.

I will use "type": "module" in package.json and not .mjs.

---

What to do:

• Add "type": "module", to package.json (below the author field)
• Add "exports": "./index.js", to package.json (below the type field)
• Update engines field in package.json to Node.js 12.
• Update index.d.ts to use ESM imports/exports.
• Update code examples in readme and index.d.ts to use ESM and also top-level await.
• Remove 'use strict';.
• Convert '.' to './index.js'.
• Use the node: protocol for imports.

Notes to self

• Start with the lowest level packages that most of my other packages depend on.
• Deprecate packages when needed.
• I also need to update the TypeScript types (index.d.ts) to use ESM syntax instead of CommonJS.
• Moving TypeScript packages to ESM is blocked by Support ESM test files avajs/typescript#5.
• Move from nyc to c8 for packages with code coverage?
• Wait with packages that require exports/imports package.json fields: https://twitter.com/bradleymeck/status/1346466891070664704
• Packages that export a JSON file directly must use fs.promises.readFile instead: https://twitter.com/MylesBorins/status/1346730287363985409
  • Docs: https://nodejs.org/api/esm.html#esm_no_json_module_loading
• Write a blog post encouraging other package maintainers to move their packages to ESM. (Done: https://blog.sindresorhus.com/get-ready-for-esm-aa53530b3f77)
• Update https://github.com/sindresorhus/typescript-definition-style-guide to use ESM export syntax.
• Update all my code examples to use ESM. (Code examples should also use top-level await and no async IIFE)

---

FAQ

Do you plan to include CommonJS fallback files?

No. Node.js 12 supports ESM natively. There's no point.

Can I help out?

Sure, but not yet. I need to figure out the best way to approach this first.

[Richienb]

Packages that export a JSON file directly must use fs.promises.readFile instead: twitter.com/MylesBorins/status/1346730287363985409

If a module only exports JSON files, using Node APIs to reexport them opens a whole new can of worms which include the best way to import a json file worm and the browser support worm.

We could try moving the JSON file to JavaScript but that might require us to add extra Typings to egg on the TypeScript compiler which can already understand static JSON files way easier. It might also make it more boilerplatey to create update scripts.

I think we should wait for JSON imports to be usable without flags before making the move.

[fregante]

I think we should wait for JSON imports to be usable

Therefore 2023 since they aren’t even finalized yet.

I agree with the concern, perhaps we can find an alternative solution here, but it won’t be a clean one.

Maybe via export map in package.json:

• exports.module = './index.js' with fs module
• exports.browser = './index.json' since bundlers support this, even if sometimes not by default

[bd82]

Do you plan to include CommonJS fallback files?

No. Node.js 12 supports ESM natively. There's no point.

If I understand correctly(?) requiring ESM modules from CJS is a tricky topic, see:

• https://redfin.engineering/node-modules-at-war-why-commonjs-and-es-modules-cant-get-along-9617135eeca1
• Open issues with require(esm) nodejs/modules#454

In this case what happens to existing code bases, which migrate to nodejs 12 on May
yet still use CJS syntax?
Would such projects be expected to migrate to ESM to be able to (easily) consume one of your packages?

It is perfectly fine to only support ESM from a library maintenance overhead perspective.
But if I understand the sources above correctly (?) , there may be a point for exporting both commonjs and ESM
variants from library.

[niksy]

The thing is, ESM is here to stay and it’s better than CommonJS in which was never standardized. Yes, we used it thinking it’s standard, but we might as well be better off using RequireJS. Instead of giving people hacky solutions to a problem, why not try teaching how to solve those problems?

[joepie91]

The thing is, ESM is here to stay and it’s better than CommonJS in which was never standardized.

This is wrong on multiple counts. Not only were CommonJS Modules, in fact, standardized all the way back in 2009, Node.js has also exhaustively documented its specific implementation (which deviates from the spec on some points), and this specific design has been implemented in hundreds of different tools in a compatible manner over the past decade. I see no reasonable way in which you could argue that this "was never standardized". It wasn't in the ECMAScript spec, but why would that be necessary for it to be a valid standard, especially considering that it far predates any attempts from ES to make this a language feature?

Additionally, as I have also addressed in my article, almost nobody is actually using ESM, because most people are 'using' it through Babel and ultimately dealing with the CommonJS-transformed version of it. By just about every metric, CommonJS usage is far more widespread than that of ESM, and to top that off, while all ESM library code is still in actively-maintained libraries, a lot of CommonJS code is in libraries that will never be changed.

If any module system is "here to stay", it would be CommonJS. No matter how strongly the ESM spec is worded, no matter how much people try to force people over to ESM, this is the reality of the situation. JS environments will need to support CommonJS perpetually. There exists no scenario in which CommonJS is fully migratable to ESM. There is a scenario in which ESM is abandoned and reversed, however.

Instead of giving people hacky solutions to a problem, why not try teaching how to solve those problems?

Because there are fundamental design problems with ESM, that no solutions exist for. The most obvious one is the spec-wise asynchronous nature; which sounds great if you buy into the idea of native browser module support... but once you realize that that fundamentally cannot work due to dependency graph traversal being inherently serial, what you're left with is a limitation that makes things like parametric modules fundamentally impossible to use ergonomically.

There are no solutions to the problems I am pointing out. They are design problems, fundamental incompatibilities between different paradigms. ESM is simply worse than CommonJS in these respects, in ways that cannot be fixed in the spec afterwards without compromising on the core design principles.

[niksy]

There is a scenario in which ESM is abandoned and reversed, however.

You seriously believe this? I see you’re very well versed in module spec definition and implementation, but I guess Sindre and everyone else is wrong in implementing something that is fundamentally wrong from your point of view? I think your efforts would be better spent in trying to adjust to ESM world than trying to push for CommonJS only. I refuse to believe that finalized spec is that bad that the whole ecosystem is blind to it’s pros and cons.

[joepie91]

You seriously believe this?

Yes. But only if people stop blindly defending ESM as the hip new thing and start actually looking at its technical merits (or lack thereof) and ecosystem implications, which very few people seem to have actually done beyond repeating the same few debunked claims of benefits.

The alternative is harm to the ecosystem and implementers until the end of time (or the end of JS, whichever comes first). I don't see that we have another option in the long term. And it's certainly much easier to undo the damage now that all the maintainers are still around.

I think your efforts would be better spent in trying to adjust to ESM world than trying to push for CommonJS only.

As stated, there are features that are needed that ESM fundamentally cannot support, and never will be able to. ESM is worse than CommonJS, it just looks more aesthetically pleasing to some people. The correct thing to do here is to prioritize and formalize CommonJS.

Also, if ESM really were totally fine and could replace CommonJS, wouldn't it have been much easier for me to just convert stuff over? Would've saved me a bunch of implementation work for fix-esm and the like. Instead, I'm here arguing about it. There's a reason for that...

I refuse to believe that finalized spec is that bad that the whole ecosystem is blind to it’s pros and cons.

Over the past several years, I have spoken to hundreds of people about ESM and its technical properties. Effectively none of them actually understood how it worked, or what the implications of its design choices were.

ESM gained popularity through hype and visual familiarity (due to being similar to Python syntax), not through technical approval. And yes, I feel comfortable stating that the ecosystem at large does not understand the full implications. If they did, wouldn't I have heard a good argument for its existence by now?

Again: if someone were to come up with an actual convincing technical argument that holds up to scrutiny, I would happily hear it. But after so many years and so many discussions, I have pretty much given up on that happening, and I can only realistically conclude that ESM was just a bad decision, and that it is a bad idea to keep throwing good money after bad.

I've provided all the technical rationale on why ESM is problematic above and in my article, and so you can assess for yourself whether these arguments make sense and whether ESM proponents have provided compelling counterarguments or not. That's probably a better approach than assuming I must be wrong just because it's not the popularly-espoused view or that of TC39 participants...

[privatenumber]

FYI For mixed ESM + CJS runtime, checkout esno/esmo:

• npx esno ./script converts all ESM to CJS
• npx esmo ./script converts all CJS to ESM

They use esbuild for instantaneous transpilation, and also works with TypeScript.

[astoilkov]

I tried making one of my packages use "type": "module" but it seems(for now) that Next.js doesn't work with ESM modules. See astoilkov/use-local-storage-state#20.

Hope this is useful information. I didn't know that.

[sindresorhus]

I'm personally not concerned with that. Build-tools have had plenty of time to support it. Moving packages to module type is a good way to force them to prioritize it.

[silverwind]

Update code examples in readme and index.d.ts to use ESM and also top-level await.

I thought TLA requires Node 14.3.0 or greater. Is it wise to suggest it when still targeting Node 12?

[jsumners]

Do you plan to include CommonJS fallback files?
No. Node.js 12 supports ESM natively. There's no point.

I think this is being purposefully blind to reality. Yes, Node.js 12 supports ESM natively. The key word is "supports." Node.js 12 does not deprecate CJS in favor of ESM, and to my knowledge no one has suggested that ever be the case. So you'll be breaking a lot of people who do not see any benefit in restructuring their whole code base to support a different syntax.

I see this plan as creating more fragmentation than cohesion.

[WebReflection]

@fregante to be honest, the only limitation in CommonJS from ESM is live bindings, which is both rare, but also easy to circumvent, by simply keeping the let reference in the module, and export getReference and/or setReference instead of live changing the reference. That is the only thing to consider, in my experience, everything else is a ascjs esm-folder cjs-folder away.

[fregante]

TLA is not coming to CJS either, is that correct?

I get it that moving to ESM might not be straightforward for all, but having a language-native module system means that we’re more likely to see development towards ESM rather than CJS. i.e. I expect most projects to be ESM-only eventually.

While sometimes packages can be easily converted between the two, they’re not equivalent, regardless of the effort they put in. You can see this in the unnatural require('abc').default you often find. Supporting both means at least extra documentation and a build step, plus additional headaches and inability to use future features.

If tools like ascjs are actually super simple and fast, I don’t see why a tool like that can’t be run on your own node_modules. The cool part of this is that it would transfer the maintenance burden from the package maintainers to the CJS users, which may be more fair.

[WebReflection]

I think this post from NodeJS repo itself explains it well: nodejs/node#33954 (comment)

in NodeJS, the language-native module system is both CommonJS and ESM

if modules are for browsers only, I agree there's no need to provide CJS too, as bundlers should work OK with ESM, but if that's not the case, it's a choice that might cause more troubles than it's aiming to solve for SSR/NodeJS related projects.

[jsumners]

TLA is not coming to CJS either, is that correct?

It's not possible. But I fail to see how that matters.

I get it that moving to ESM might not be straightforward for all, but having a language-native module system means that we’re more likely to see development towards ESM rather than CJS. i.e. I expect most projects to be ESM-only eventually.

Language support is nice. But your completely ignoring the framework support. Node.js isn't a language, it is a framework and they have clearly stated that they are not deprecating CJS. The modules converting to ESM only, that are under discussion here (and I argue there isn't much discussion happening), are Node.js targeted modules. The ramification of this change is that all of the dependents of these modules will be forced to choose one of:

1. Convert their code base to ESM: surely you can see how this is an expensive, in terms of real world currency, task
2. At a minimum refactor their code to wrap everything in an async function main() {} just so they can continue using these modules: again, expensive in terms of real world currency
3. Fork the old versions of the modules to maintain CJS compatibility: expensive and fractious
4. Abandon these modules completely: I get it, world's smallest 🎻 . But it is a consequence

🤷‍♂️ I know what I'll have to do. It disappoints me, but it is what it is given the positions expressed here.

[joepie91]

@jsumners I've published fix-esm yesterday, which hooks into the module system and makes it possible to require() ESM modules from CommonJS code. Hopefully this will help to reduce fragmentation and ecosystem misery.

[jaydenseric]

@sindresorhus

I will use "type": "module" in package.json and not .mjs.

Why? IMO .mjs is technically superior.

• It makes it easy for humans to tell what kind of module the file is.
• I could be mistaken about this, but could Node.js skip costly filesystem lookups for the nearest package.json if the import path ends in .mjs? Even if Node.js caches package types for certain directories, that still has a memory cost. The same concern applies to bundlers, or any other tool that needs to resolve an import. Perhaps both Node.js and bundlers have to resolve package.json files anyway, to figure out package exports so it's not much additional cost to read the type field at the same time 🤔
• ESLint configs can have overrides based on the file extension to enforce ESM vs CJS specific rules, without having to go to the filesystem to try to find the nearest ancestor package.json, read it, and figure out what the type is. E.g:
  • https://github.com/jaydenseric/eslint-config-env/blob/v18.0.1/index.js#L233-L246
  • https://github.com/mysticatea/eslint-plugin-node/blob/v11.1.0/lib/configs/recommended.js#L13-L16

My packages (~4.3m/wk installs) already support both CJS and native ESM, but on May 1, 2021 I plan to join you (and it looks like Babel too now) and publish them all as pure ESM. To avoid copping a lot of individual heat from CJS users that will no longer be able to require our packages, package maintainers need to collectively go hard with this at the same time in early May. Make it a movement!

Nothing, and I mean nothing has been a greater maintenance headache than trying to support dual ESM/CJS exports whilst avoiding the dual package hazard through all the iterations of Node.js --experimental-modules, and the ignorant mess that is the npm eccostem. Even now Next.js doesn't support native ESM and CJS interop properly (e.g. vercel/next.js#17806), and ESLint has refused (multiple times: eslint/eslint#9592 , eslint/eslint#13301) to process .mjs files by default, and if you do use the --ext flag, it won't automatically process .mjs/.cjs/.js files as ESM or CJS.

I can't wait to get this revolution started.

One other suggestion for the "What to do" list; now is a good opportunity for each package to:

• Add a package exports field to make sure only the public API can be imported.
• Structure the files in a way that every single export is available (and documented) as a deep import, so that people can import just the necessary code without loading unused functionality into memory or having to rely on a tree shaking bundler. Internally, every import should be deep; no imports should be via an index. In a perfect world index files would only be used as an entrypoint for bundle size analysis. The ecosystem would be so much better off if deep imports were common practice and index files rarely used.

[Richienb]

It makes it easy for humans to tell what kind of module the file is.

In an ideal world of only ESM, there is no other kind of module and .mjs and .cjs have both been phased out. Let's try to achieve that perfect world.

[sindresorhus]

I was personally always against .mjs. Adding a new extension caused much more problems and churn than it solved. I find the type field being a good solution. I'd rather optimize for an ESM-only future.

[sindresorhus]

Perhaps both Node.js and bundlers have to resolve package.json files anyway, to figure out package exports so it's not much additional cost to read the type field at the same time 🤔

Correct. Node.js and bundlers have to read package.json regardless.

[sindresorhus]

Add a package exports field to make sure only the public API can be imported.

This is already in the list.

[coderaiser]

Automatic migration from Commonjs to ESM

putout code transformer can be used for automatic migration to ESM with help of a bundled plugin convert-commonjs-to-esm.

It converts require/module.exports into import/export accordingly. It is enabled by default for *.mjs files, for regular *.js files can be enabled with config file .putout.json:

{
    "rules": {
        "convert-commonjs-to-esm": "on"
    }
}

Using import as require

When things can't be solved with automatic migration, simport can help, solving a couple edge cases. One of them is import by absolute path in windows (usually after finding a plugin that should be loaded dynamically in tools like ESLint). Also it gives ability to use import just as require (no extensions, support of json files, passing dynamic import function as an argument etc).

Also you can use plain old __dirname, __filename and require in ESM with help of:

import {createSimport, createCommons} from 'simport';

const simport = createSimport(import.meta.url);
const {
    __filename,
    __dirname,
   require
} = createCommons(import.meta.url);
// now you can use require, __filename and __dirname in ESM

await simport('./package');
// will read `package.json

await simport('c:\\some-js-file');
// will also works

Mocking of modules for testing purpose in ESM

With ESM you cannot use mock-require. It supports only commonjs packages.
Also cache of imported modules cannot be accessed from JavaScript. That's why loaders was introduced.

You can mock imports using mock-import. Which is based on putout and gives ability to dynamically mock imported modules.

Testing of ESM modules

tape doesn't supports ESM, to use it keeping assertions API, supertape can be used. It has a similar API, very fast, and has a native support of ESM without transpiling.

[silverwind]

For __dirname, I just use:

import {dirname} from "path";
import {fileURLToPath} from "url";

const __dirname = dirname(fileURLToPath(import.meta.url));

[WebReflection]

FWIW there's a module too 👋

const {dirName, fileName, require} = umeta(import.meta);

[Richienb]

Start with the lowest level packages that most of my other packages depend on.

Is it possible to make an ordered list of exactly what packages need to be done first?

[revelt]

Practically, this means, if anybody has TS packages, until microsoft/TypeScript#33079 is solved, we won't be able to update the consumed Sindre's packages. I tried this morning to wire the exports, set the engine to >=12 etc. in my monorepo, the definitions are not found. I hope after 2021-04-30 MS will start moving faster. No deps bumping until then!

[mitschabaude]

@lockHrt It seems to me that you can easily avoid this complexity by not setting the target to commonJS?

I am using ESM-only packages successfully, across the whole stack - node (ESM) and web, JS and TS

[lockHrt]

@lockHrt It seems to me that you can easily avoid this complexity by not setting the target to commonJS?

I am using ESM-only packages successfully, across the whole stack - node (ESM) and web, JS and TS

What's the use of that if many developer dependencies stop working? Jest, pkg etc are crucial for me to work.

There are many more such dependencies are used in ci/cd pipeline which completely break the project if they smell even a single ESM only package.

Avoiding 'ESM only' is the best option as of now.

[mitschabaude]

I think the best option always depends on your specific situation. But ESM is very obviously the future.
I just want to mention that my coworker got jest to work in a TS/ESM project. But it seems to me you're not really interested in getting it working anyway?

[lockHrt]

If companies like MS, Facebook, Vercel don't see ESM as future then my personal opinion doesn't count. There's no benifit swimming against the flow. TS has effectively solved this mess and today more and more people are maintaining their packages in TS. So there won't be many 'esm only' packages in the future.

I have wasted days implementing my own solution to get 'esm only' working in TS only to realise a lot of developer dependencies straight up don't work.

Company may not want to restructure 100s of thousands of lines of project just to include a small 'esm only' package where there are many alternatives are available. I think your use case is small. You should visit issues of these big packages to see companies like MS, Facebook don't care that ESM is future.

[OmgImAlexis]

I have wasted days implementing my own solution to get 'esm only' working in TS only to realise a lot of developer dependencies straight up don't work.

What in the world are you on about? If you're using esm and ts then commonjs will work, esm will work, where's the issue?
I keep seeing people say this kind of stuff yet none of them actually say what didn't work or the errors, etc. they're getting. I have a feeling a lot of you guys would have working repos if instead of complaining you provided an example of what's not working and asked for some help fixing it.

[ezzatron]

Another roadblock here is extension-less executable files using a shebang like #!/usr/bin/env node. AFAIK, there's no way to get Node.js to treat these files as ESM, which makes it basically impossible to use any ESM-only module from within them. Unless I'm missing something, that means that most CLIs basically can't use your modules now.

[jaydenseric]

I'm not following. Why don't you use a file extension? I've published quite a few CLI's and always used file extensions. For example:

https://github.com/jaydenseric/find-unused-exports/blob/0f2a2bcb9c3cb6c389e06a21a33115b6f4693830/cli/find-unused-exports.js#L1

[ezzatron]

Hmm. I don't know why I guess, it just seems weird to use file extensions for executables that also have shebangs. Like, if I was writing an executable shell script I would leave the extension off so that people can invoke it like foo instead of foo.sh. Mostly because it's marginally nicer to use.

But now that I think about it, that mostly only affects me while developing the library and running the executable directly, instead of via node_modules/.bin once it's installed as a dependency. And NPM sort of "hides" the extension when it creates the symlink, so I suppose it's all good.

Thanks for the reply.

[WebReflection]

AFAIK, there's no way to get Node.js to treat these files as ESM

there is one

[ezzatron]

AFAIK, there's no way to get Node.js to treat these files as ESM

there is one

Useful to know. Unfortunate that import.meta.url can't work via STDIN. Hopefully Node can provide a better solution in future releases.

[LinusU]

The engine field

Currently, I'm leaning towards the following engine field for my packages:

  "engines": {
    "node": "^12.20.0 || ^14.13.1 || >=16.0.0"
  },

12.20.0 and 14.13.1 were the first versions in their respective release line to support the node: protocol for importing built-in modules.

@sindresorhus you have the following point in your list:

Update engines field in package.json to Node.js 12.

Would you mind sharing the exact string you intended to use? ☺️

[sonicdoe]

Similarly, v14.13.1 is required for node: as well. Do you prefer >=12.20 to keep the engines field simpler (even though it allows v14.0.0, for example)?

[sindresorhus]

I prefer just >=12, but I'm tired of people asking me to do something more specific, so I'm caving. I still feel like it adds very little as npm doesn't enforce it, and from experience, users open issue about it anyway without reading the install log. But yes, if we add specific for 12, we should do it for 14 too.

[sindresorhus]

Updated the gist: https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3ecc99d99c

[sonicdoe]

Sorry for being one of those people then! I see the appeal in >=12 myself but wondered about only v12 being more specific now. In any case, we can all look forward to April 2023, when v14 is scheduled to be marked as end-of-life 😉

[sindresorhus]

@sonicdoe No, you're right. I guess I was just frustrated that there's no perfect solution to this and a lot of churn.

[LinusU]

Native modules

Native modules cannot (yet?) be loaded using import, and instead we must use module.createRequire() or process.dlopen.

ref: https://nodejs.org/api/esm.html#esm_no_native_module_loading

No Native Module Loading

Native modules are not currently supported with ES module imports.

They can instead be loaded with module.createRequire() or process.dlopen.

@sindresorhus will you be using module.createRequire() or process.dlopen?

I'm personally leaning towards process.dlopen to move as far away from require as possible hehe, but I need to do some more investigation before I'm totally decided...

[sindresorhus]

Probably module.createRequire() since process.dlopen is discouraged. Although, I have very few native addons. Should not matter though. Eventually, we'll be able to use import.

[LinusU]

Sounds good 👍

process.dlopen doesn't seem to support a URL as the path argument, which was the small thing (together with having to specify RTLD_NOW) that tipped me over as well...

import os from 'node:os'
import process from 'node:process'

const addon = { exports: {} }
process.dlopen(addon, new URL('./build/Release/xattr.node', import.meta.url).pathname, os.constants.dlopen.RTLD_NOW)

vs.

import { createRequire } from 'node:module'

const addon = createRequire(import.meta.url)('./build/Release/xattr')

[LinusU]

Just converted my first package, which also happened to be a native module! 🚀

Thank you so much @sindresorhus for putting together the checklist! ❤️

LinusU/fs-xattr#41 -> 🚢 0.4.0 / 2021-05-11

[wKovacs64]

Can someone explain this one to me like I'm 5?

• Wait with packages that require exports/imports package.json fields: https://twitter.com/bradleymeck/status/1346466891070664704

I've read the Twitter thread and other related material but I'm still not grokking it. 🙁 Surely exports in package.json is fine, as adding "exports": "./index.js" is part of the steps. Is the problem with a more complicated exports map, like this, as subpath imports are still experimental?

"exports": {
  ".": "./index.js",
  "mod1": "./foo/bar/mod1.js",
  "mod2": "./foo/bar/baz/mod2.js"
}

Or am I still way off?

[sindresorhus]

Node.js handles that fine, but bundlers are lagging behind (even though they had lots of time to prepare for this).

[wKovacs64]

Ahh ok, so the bundlers can handle single file "exports": "./index.js" but they're behind on subpath support. Thank you!

[typhonrt]

I just want to drop a tooling related ESM success story. Often tooling lags behind major tech transitions and Oclif was the main framework that kept me working with CJS. I'm aggressively moving to native ESM for all my current and future dev efforts on Node. I recently had the opportunity as an external developer to add comprehensive ESM support to Oclif v2 which is the next generation of the Salesforce / Heroku CLI framework launching later this summer.

You can read more about the process and how things went down here:
oclif/core#130

I'll make another post when all of my open source repos are converted to native ESM and available on Node as well. Let's go ESM!

[yinzara]

I cannot tell you the hell I've gone through trying to support using the updated ESM only packages you've published.

Every framework I use from TypeScript itself, to Jest, to Ts-node, to Yarn 2.0 all have issues with ESM support. Even NodeJS still considers parts of it experimental.

jestjs/jest#9430
microsoft/TypeScript#43329
TypeStrong/ts-node#1007
yarnpkg/berry#638
https://nodejs.org/api/vm.html#vm_class_vm_module

While I appreciate the reasoning behind why you're doing this, the cut over was done prematurely before the ecosystem was ready IMO. Without supporting both CJS and MJS while packages correct issues means I as a user of published packages have literally spent 100s of hours trying to make what you've done work to be constantly thwarted at every corner and had to swap back to your prior versions meaning if there is ever a vulnerability found in them, I won't be able to upgrade. The work arounds that you've stated simply don't work when the libraries we use all break.

Do you plan to include CommonJS fallback files?
No. Node.js 12 supports ESM natively. There's no point.

"There's no point" is a pretty strong statement.

There is a major point. Not breaking the entirety of the dependency ecosystem while your attempt to force a paradigm change. All of the above mentioned projects were already attempting ESM support before you strong armed your 1000+ libraries to only ESM but there are still serious problems across the entirety of the ecosystem. Trying to state that the only way the ecosystem will move to ESM is by package maintainers forcing people to use ESM is simply untrue. The whole reason the dual syntax support exists is so packages can support both while the ecosystem changes.

As you update your packages one by one, my RenovateBot attempts to upgrade and breaks every time and I have to exclude that package from dependency upgrade management or I have to disable Renovate attempting to upgrade any major version of any package in my monorepo of more than 7000+ dependencies. Since you don't have a package prefix on all your packages (i.e. no @sindresorhus/...) there is no way for me to even exclude all of them from upgrades. If any libraries that I use that transitively depend on any of your packages happens to update to your new ESM only versions (even if that package supports both), my whole build now breaks because of the above Yarn 2.0 issue. There is no work around for this, I have to exclude that package as well. I'm stuck in a place where now I have the potential of a whole set of dependencies (everything you've changed and every library that transitively depends on your ESM packages) that I can no longer update and I have no work arounds until all those above libraries fix their issues.

Very :-( 👎

[jimmywarting]

there are also flaws with require('./foo') path resolver that simply just don't work in browsers / networks - it can't guess if you mean ./foo/index.js or ./foo.js
import enforce exact path and makes it better in a way

[privatenumber]

@joepie91

I think you raise good points about ESM, but again, I'm afraid your arguments will be lost in this thread.

(Although for me, the benefit of standardization and native/syntax-level support for modules is a great and good enough win.)

I would much rather read your ideas in a more structured and cohesive manner (eg. blog post). Posting something like that on Hacker News or Reddit could reach a much broader audience, and hopefully be taken seriously by TC39.

I'm still wondering "so what?" though. Convincing Sindre to stop converting to ESM won't stop ESM.
What's your end goal? And how can others help you achieve it?

[joepie91]

@jimmywarting:

And now thanks to more modern QUIC, http2 (or what we have right now - not up to date)... pushing multiple files to the client isn't a as much as an issue anymore with now also faster internet speed.

You are misunderstanding the problem. The issue is not in loading things in parallel; that has been possible prior to HTTP/2 already. The problem is specifically traversing the depth of a dependency tree over the network, which is not possible to parallelize. Because you only know which other modules are required by a given module once you have parsed that module, you will need to make at least as many roundtrips serially (not in parallel!) as there are levels of nesting in your dependency tree.

You can't fix this with network-level parallelism. It's a fundamental impossibility to know what other modules a module requires without either a) parsing the module itself, or b) generating a manifest upfront. Option B is, for all intents and purposes, the same thing as a bundler, just with a different output format.

If you have, say, 30ms of roundtrip network latency to a server - and that is incredibly optimistic, pretty much "the server is near me geographically" - then a dependency tree with 6 levels of nesting will unavoidable require at least 180ms (6 * 30ms) to traverse, excluding parsing time, excluding the actual download of the code, just the latency of running these requests serially. That's a lot of latency! And it gets far, far worse when you look at more realistic latency like 100-150ms, where your 6 levels of deps will net you 600-900ms (nearly a second!) just to traverse the dependencies.

That's not workable. Which means that in practice, everybody will use a bundler (or bundler-like development tool) anyway. ESM does not and cannot solve this. We're running up against the laws of physics here, not a limitation of the module system or the network layer.

no build processing, faster load time, no garbage output, no annoying sourcemap, easier to debug, fewer files, fewer config, faster/rapid prototyping.

None of this other than maybe a very limited sourcemap has anything to do with whether you use a bundler or not, as I've explained earlier. It's completely possible to have a dev setup using CommonJS that does not do minifying (therefore no real need for sourcemaps, and you have readable output), that hot-reloads instantly and only what has changed, that doesn't transpile anything, and so on. This is possible today, with CommonJS. Hell, both Browserify and Parcel can effectively run with zero configuration. It's really just Webpack that has a terrible configuration system.

You do not need ESM for any of this. The idea that these features are brought to you by ESM is a widespread but ultimately false belief. None of this requires breaking the ecosystem.

The real issue is the amount of things we bundle into one single file and all the extra overhead (wrapper) around existing stuff like using Buffers instead of plain uint8array in browsers. there is also a huge amount of unused code being used. Writing good performant code is better than compressing and minifying code. not only that but that you also use code splitting to and lazy import is good too - for that we need a proper module loader.

All of this is, again, completely possible with CommonJS today. These are not benefits of ESM.

the way i see it is that it was fragmented before ESM came to be. ppl couldn't agree upon something before cuz there where no standard for it, goog.loader, reacts own stuff, typescript, commonjs, UMD, requireJS, systemJS, <script>, importScript ajax+eval and more had there own way of importing modules

Let's take these one by one:

• Google always reinvents wheels, and they had already moved most things to CommonJS years ago anyway.
• RequireJS, SystemJS, etc. have been dead for years - people switched to CommonJS instead, as was the intention behind the CommonJS standardization effort.
• UMD is a compatibility mechanism, not its own module system, as explained above.
• React doesn't have its own module system at all - not sure what you're referring to there.
• CommonJS was already usable in TS just fine - and they have their own module syntax for definition imports anyway, which ESM does not address either - TS is just a different language entirely.
• <script> is going to stick around for the reasons I've already explained, ESM or not.
• importScripts is something I've never seen used as anything other than an implementation detail within bundlers, so a completely different category of thing than CommonJS and ESM in practice.
• Likewise for async HTTP requests + eval, I've only seen it used as an implementation detail in bundlers, not as an actual module system.

So where's the fragmentation here? I don't see anything in that list that is both in common use and that can be replaced by ESM.

If you can't see the benefits of what ESM brings then you maybe don't see things that others do

I've been discussing this topic with hundreds of people for years now, including TC39 people, precisely to ensure that I wasn't just missing something here. Yet nobody, including vocal ESM proponents, has been able to come up with a good reason for its existence that actually holds up to scrutiny. I feel pretty confident in concluding that there is none, by this point.

[joepie91]

@privatenumber:

I would much rather read your ideas in a more structured and cohesive manner (eg. blog post). Posting something like that on Hacker News or Reddit could reach a much broader audience, and hopefully be taken seriously by TC39.

I tried that. That's what this article and the preceding Twitter thread are (and which I am mostly just repeating in these comments here). Unfortunately, it completely drowned in the ESM hype. While I can speak with a louder voice than most people in JS-land, it is nowhere near enough to compete with that of TC39 and the hype train following ESM.

If you feel you can help with this, I'd be happy to hear it. But right now, talking about this at the current 'center' of the breakage is the best option I see myself having to raise awareness of these issues.

What's your end goal?

To stop or at least reduce the harm that is coming to the ecosystem as a result of this push for ESM, which will very likely continue for many years for developers, and in perpetuity for runtime maintainers (due to having to support two module systems). Same reason I built fix-esm as a temporary workaround.

I expect a lot of developers to only find out about the consequences of the ESM decisions over the next couple of years, as more and more stuff breaks for them and they are suddenly having to scramble and fix it all. I would like to pre-empt that, if possible, and salvage the situation without long-term harm while it is still possible.

And how can others help you achieve it?

That's a fair question.

Honestly? By talking about it, discussing it, bringing it up to others, and looking more critically at the claims about ESM. By cutting through the hype, seeing how much ESM really improves and whether it wouldn't have been possible to get there with CommonJS. And considering how absolutely massive the breakage caused by this change is, there's a high bar to be met there.

By looking at the articles extolling the virtues of ESM, and seeing whether any of them can actually come up with anything that isn't already proven false, or whether they're all just blindly parroting the same few claims without verifying them.

I believe that my arguments can stand up against scrutiny by themselves, that they can be judged on their merit in a more scientific approach to the situation - and I would ask everybody to look for counterpoints in good faith, if you're not convinced! My goal here is to make a logically sound argument. But having a good argument is only half the battle, and it's not going to gain much currency if it drowns in the hype.

Outside of some very specific circles, I'm not some "thought leader", not some famous author, not somebody who can tweet a thing and have the entire JS ecosystem pick it up. I'm just a guy who has spent way too much time diving into the topic of dependency management as a whole, and who has done a lot of volunteer IRC support, and come up with some issues in a widely-poorly-understood system as a result.

So what I need is people who look into this seriously, who are willing to dig into the topic and assess what the right answers here really are, who are willing to accept that maybe popularity doesn't mean it's a good idea, and who can help talk to others about it, and generally raise awareness about the long-term consequences of this decision that aren't quite that obvious yet unless you know what to look for.

[lockHrt]

Add https://github.com/vercel/pkg to your list. Another big library in CI pipeline that breaks.

[OmgImAlexis]

Just a tip, if you're using tsup and install ESM modules as dev deps they'll be rolled up into your final build.
When I see a dep move to ESM I simply uninstall it, reinstall as a dev dep and that's it.

No extra step needed.

https://github.com/egoist/tsup

[coderbyheart]

Very happy to see that other popular packages are also making the shift: node-fetch v3 is ESM only.

[loynoir]

@yinzara
Hey, bro. Good news for you. Check out this.
Seems all tests passed.
https://github.com/loynoir/example-jest-config-esm/blob/master/test.sh#L39

PS: Of course, if not specific with experimental vm, some hard core ESM feature not work, see test folder.

[yinzara]

I think you're completely missing my point. I said
< node-fetch is one of the most used mocked libraries that exists. There are tons of projects that the entire goal is just to enable mocking of node-fetch.
<
< Mocking with ESM is HORRIBLE because of the immutability of the module system.

Your example doesn't attempt to mock anything and even if it did:
jestjs/jest#10025
https://github.com/yurijmikhalevich/esm-project-with-working-jest-mock/blob/main/package.json

Your solution is exactly why the above jest issues was filed. Every ESM only package must be listed in your exclude list and unfortunately with a monorepo, the exclude list is different, difficult to figure out (if at all possible) and can't have and even then you must use Babel to transpile everything. Every user must have a very complex configuration using experimental features of NodeJS. Mocking still is horrible (a complete change even with the PR in progress).

How am I as the package maintainer for jest-fetch-mock supposed to deal with trying to get everyone who uses my package to do that!?

The whole argument in this discussion was:
< Do you plan to include CommonJS fallback files?
< No. Node.js 12 supports ESM natively. There's no point.

There is a VERY good point. Right now the ecosystem is not ready. It simply isn't and all the workarounds in the world isn't going to convince me of that.

[loynoir]

@yinzara
Yes, current best workaround I found is

• hardcode every ESM-only pattern if omit --experimental-vm-modules
• Or just accept experimental :(

That's also why I wish @sindresorhus on .mjs side.
In this real world case, jest.config.XXX config context, only file pattern matching is accepted.
Not many context allow user doing, or themself doing AST level matching.
If everyone using .mjs, the hard code pattern A|B|C|D|E|F|G can simplify to mjs, can reduce a lot of hardcode.
Really hope @sindresorhus on .mjs side. 😢

[loynoir]

Hi, @yinzara .
I think, I didn't miss your point.

I'm just a normal passer-by, upstream decision is out of my scope.

I'm just a normal passer-by, willing to help. By sharing my not perfect, but tested working jest config, under 2 different flags, with 4 PM situations, with limitation stated for each flag. Hope it can help you find some workaround clue, by yourself.

I'm just a normal passer-by, not using jest-fetch-mock. Although I'm willing to help you, I have no obligation to offer you full workaround, or solution.

Edit: Agreed with not ready. Maybe you can create your jest preset. Or...

[yinzara]

Yeah you clearly did. I didn't ask for help coming up with a workaround. As a package maintainer trying to get everyone who uses my package to use some workaround isn't helpful especially if that work around varies per package manager and requires you use a transpilation step. All I wanted to do was show that the current JS ecosystem isn't ready for people to force ESM only which I think I did.

[loynoir]

@Richienb

Eventually, Python 2 development ended, 14 years later.

I'm sure that in 14 years, the JavaScript community is going to reach the same place as Python did.

I'm more positive about the moving speed.

Way more less than 14 years! 😃

• More solid AST tools (babel, typescript...) compared to python 2to3 / 3to2

JS community is such a uniq community, everyone seems kind of using AST tools somehow. 😂

• Driven by Internet power

I can't image a world without Internet! 😂

• CJS->ESM move not as code breaking as Python 2 to 3, the later linter not report error.

@Richienb

In an ideal world of only ESM, there is no other kind of module and .mjs and .cjs have both been phased out. Let's try to achieve that perfect world.

@sindresorhus

I was personally always against .mjs. Adding a new extension caused much more problems and churn than it solved.

As I'm personally on .mjs, kind of sad. 😂

• Compared to python shift, but less hashbang concept, although there is a stage3 proposal.
• Personally, new thing prefer new extension, using own name, instead of using the name of its senior.
• .mjs is more identify-able than .js. At filename level matching, instead of full AST level matching, and unstable RegExp full content matching. No hashbang matching yet.
• JS community is a large AST tool user community, somehow. Codeshift is peaceful. But filename is outside AST. I guess .mjs is more peaceful than "type": "module"

But, I guess .mjs is loosing. 😂

• At least 1K+ packages on the way to .js
• typescript haven't support .mjs well.

ESM-only decrease library maintainence.

But moving too fast, I think might be roadblock to let people adapt it, compared to dual-module.

Anyway, glad to see ESM movement! 😃

[vjpr]

Idea: Could we have dist-tags pointing to the last cjs version?

So we can do: pnpm add indent-string@cjs, instead of having to look up which version cjs is supported in?

[WebReflection]

out of curiosity: isn't that a slippery slope to embrace unmaintained code and security issues that could be part of it?

[vjpr]

Of course, but as this discussion clearly demonstrates, there are many people who cannot simply upgrade to ESM overnight.

[joepie91]

For all of those who are unable or unwilling to change their codebase to ESM, fix-esm makes it possible to load ESM modules from CJS (require/module.exports) code in Node.js.

(Disclosure: I'm the author of that module. Already mentioned it in some subthreads, but posting it top-level now for visibility.)

[WebReflection]

ascjs, ucjs, are there since about ever too (author here) but ESM to CJS is never 100% possible, for various reasons. If ESM modules don't use live bindings, or abuse the default export, then yes, it's easy to transform those, yet dual modules are a thing, which is what I publish these days, but here, for some, arguably valid, reason, is not considered ... this move is to "let it go" and move forward packages, and I sympathize with that.

[joepie91]

Unfortunately, as I have also addressed in another subthread, "let's do a clean move" is a practical impossibility due to the size of the CJS ecosystem. A "clean break" simply is not an option, regardless of whether certain maintainers try to force people to use ESM (despite its shortcomings and missing features, which have already been addressed in various other subthreads by various people). So all it does is in practice cause misery for people.

[WebReflection]

I guess that's one side-effect when it comes to Open Source: you don't get the lifetime support a paid software would give you ... I honestly think this was an opportunity for the OP to open a CJS support channel with an Open Collective goal, considering how many developers chimed in here, but again, I sympathize with the "let it go" message, and all miserable developers might be better off talking to NodeJS people, as there are valid CJS points that cannot be addressed via current ESM, but none of these point are a module's authors responsibility, imho.

[joepie91]

"You're not paying so you should expect things to break" is an extremely poor argument to justify intentionally and unnecessarily(!) breaking large swathes of the public commons. There's no good reason for this change other than to try and force people over to ESM "in one go", which is not viable to begin with for aforementioned reasons.

I'm not unfamiliar with coordinating mass community moves, see eg. my involvement in the Freenode/Libera debacle, so I understand the intention here and the rationale behind the plan. But that's something you only do when 1) it's necessary and 2) the alternative is a sufficient replacement to facilitate the move. Neither thresholds are met here, and this remains a very bad idea under the circumstances.

"Let it go" is an easy argument to make when you are in a position where you can afford it. But it should be clear by now that this is far from universally true for other people, and in fact there are cases which can never be ported over to ESM due to fundamental design problems, as explained here. And that makes this entire campaign simply irresponsible.

[WebReflection]

Dunno @joepie91, I am on Arch Linux and in the last 7 years, for a couple of times, also ages ago, they did some fundamental breaking change under the hood, on a rolling release, hence shipped to any user out of the box, without the versioning guarantees that npm with package-lock.json has, and never complained upstream, just updated some script and move along ... I could surely have spent days ranting on the forum, but I didn't care, spent maybe a couple of hours to fix, and then I just let it go and moved along.

Others maybe changed distro, or kept it calm, and moved forward. That is: I don't care about this move because I am not affected at all. I write software I need, usually, and update, change, software I depend on when needed.

I don't get any of these arguments, all I see is frustration behind laziness:

• fork these modules and keep them updated when needed, option 1
• stick in package-lock.json the version that works for you, and get time to plan a migration
• help NodeJS bringing in features missing in ESM

What I keep reading ... is ranting that these modules' author doesn't even bother answering to, or even reading.

"let it go" is an easy argument in every occasion, specially when you have options, because these are still Open Source modules.

Take care 👋

[mitschabaude]

Hi, I want to tell a little story: A week ago, the move by node-fetch to ESM broke one of my projects. It actually broke in production because a part of our CI installed dependencies without a package.json to lock versions 😅.

Upon understanding the cause of this breakage, I didn't go hunt for github issues where I could post "+1" and hope maintainers would revert the change and fix my problem for me. Instead, I just upgraded my project to ESM. It was quick and easy. It will never be an issue again 🚀

I just came here to express my huge respect for @sindresorhus, @Richienb, @jimmywarting, @LinusU and others to have the balls for this move. A clear break is 1000x better than decade-long legacy-supporting maintenance hell. This thread demonstrates that without breaks like these, a large part of the community would prefer to cling to CJS for years and years to come, which would imply most JS devs would still have to deal with it in one way or the other (please don't). @sindresorhus & co, you may be the people that save JS from a python2-level mess.

I'm only a small fish in the package maintainer sea, but I want to say that I'm with you on this move and won't publish CJS to npm ever again. Thank you ❤️

[Jaid]

I made a list of ESLint rules that can help migrating projects from CommonJS to ESM:

https://gist.github.com/Jaid/164668c0151ae09d2bc81be78a203dd5

[jimmywarting]

Good news for typescript beta, https://devblogs.microsoft.com/typescript/announcing-typescript-4-5-beta

the ecosystems moves to ES-modules

[lockHrt]

the ecosystems moves to ES-modules

That's a good news however as I read, it's still limited by nodejs dynamic importing of ES modules. So, folks who are writing back end code, we still have to make our code asynchronous. I don't think it'll fix anything for many popular libraries which are used in CI.

[s97712]

beta

It is beta

[coderbyheart]

AWS lambda now supports ESM: https://aws.amazon.com/de/blogs/compute/using-node-js-es-modules-and-top-level-await-in-aws-lambda/

[yinzara]

Here we are more than a year in the future and the primary issues (nodejs/node#35889 and nodejs/node#37648) that keep the most important JavaScript testing framework from converting to ESM (jestjs/jest#9430) are still not resolved.

Even code coverage is not functioning when using ESM.
istanbuljs/nyc#1287

Good news for typescript beta, https://devblogs.microsoft.com/typescript/announcing-typescript-4-5-beta

the ecosystems moves to ES-modules

Too bad. Never happened. microsoft/TypeScript#46452

Even using a ESM module in TypeScript using await import() syntax STILL requires a hack/workaround (TypeStrong/ts-node#1290) every time you want to do it.

NodeJS has STILL not removed the VM modules for ESM from experimental nodejs/modules#507.

A full year yet the ecosystem is not any closer to resolving the major issues that prevent people from using ESM in server side NodeJS.

Thanks for causing every one of us a combined total of 1000s of hours (perhaps even more) trying to work around your forcing ESM for every tiny package you maintain that are used across the entire JS ecosystem long before the ecosystem as a whole could support it.

I'm going to put my money on this being one of the direct causes that forces people to move away from server-side JavaScript all together and is the eventually downfall of the entire community. Guess we'll see.

[OmgImAlexis]

Even using a ESM module in TypeScript

No.. this is ESM in ts-node which is quite different.

[Julusian]

Lack of support from jest, with it being blocked on those nodejs issues pretty much sums up why I havent switched fully to ESM.

I have some projects which don't have tests so they are happily ESM now, but most things which are using jest need to be able to use mocks.
I have had a note of future discussion topic to raise at work about switching some things to ESM, but it is rather dried onto the whiteboard now due to mock support still lacking. If it werent for this then I would have definitely been able to migrate software at work across.

I don't dare start to convert any library I maintain to esm, because I know that 99% of users of it won't be using esm with them (even I'm not most of the time), so progress on that front is also stuck.
Maybe I would have updated some libraries to be pure esm too if it werent for jest, but that is harder to say as cjs is working perfectly fine for them and they mostly wont get any benefit from esm.

All of this means that I'm stuck using old versions of every library sindresorhus maintains. I even have a pr which was merged over a year ago but Im still having to rely on my fork because it was released the same time as the switch to esm, and its being used in a plugin for a project which doesnt support esm (they are making slow progress in switching over)

By the looks of it, I am not alone in this. It massively varies by library, but some are as little as less than 1% of the downloads are the esm version with the majority being the latest cjs version. So forcing esm at this point in so many libraries is forcing many users to be using unmaintained versions of these libraries.
Which leaves an annoying view in yarn outdated and has resulted in a mental list of which packages I know aren't worth looking into updating without a good reason.

Honestly, I don't have a problem with the plan of forcing people to migrate, but nodejs itself clearly is not ready to support a full esm ecosystem, so its the timing I have a problem with.
Overall this won't push me away from nodejs, but will likely cause my apps to live with known security issues (its not a big deal as they arent internet facing). That situation is already bad enough with how many unmaintained dependencies libraries on npm have