bug: vite build fails due to out of memory error

[olehmisar]

Building this simple file with vite fails because of out of memory error:

import { createPXEClient } from "@aztec/aztec.js";
const pxe = createPXEClient("http://localhost:8080");

This issue first started happening in 0.35.0. It still persists in 0.40.1.

Full repro: https://github.com/ShieldSwap/aztec-build-out-of-memory-repro
Related vite issue: vitejs/vite#2433

Error logs

vite build

vite v5.2.11 building for production...
transforming (1193) node_modules/asn1.js/lib/asn1.js
<--- Last few GCs --->

[27246:0x120008000]    66734 ms: Scavenge 4031.0 (4095.6) -> 4030.7 (4106.6) MB, 10.58 / 0.00 ms  (average mu = 0.365, current mu = 0.399) allocation failure; 
[27246:0x120008000]    66753 ms: Scavenge 4037.7 (4106.6) -> 4038.4 (4107.4) MB, 14.38 / 0.00 ms  (average mu = 0.365, current mu = 0.399) allocation failure; 
[27246:0x120008000]    66773 ms: Scavenge 4038.5 (4107.4) -> 4037.6 (4130.4) MB, 20.54 / 0.00 ms  (average mu = 0.365, current mu = 0.399) allocation failure; 


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x102ae20f0 node::Abort() [~/.nvm/versions/node/v20.8.1/bin/node]
 2: 0x102ae22d8 node::ModifyCodeGenerationFromStrings(v8::Local<v8::Context>, v8::Local<v8::Value>, bool) [~/.nvm/versions/node/v20.8.1/bin/node]
 3: 0x102c68e98 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [~/.nvm/versions/node/v20.8.1/bin/node]
 4: 0x102e3d67c v8::internal::Heap::GarbageCollectionReasonToString(v8::internal::GarbageCollectionReason) [~/.nvm/versions/node/v20.8.1/bin/node]
 5: 0x102e41530 v8::internal::Heap::CollectGarbageShared(v8::internal::LocalHeap*, v8::internal::GarbageCollectionReason) [~/.nvm/versions/node/v20.8.1/bin/node]
 6: 0x102e3df94 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [~/.nvm/versions/node/v20.8.1/bin/node]
 7: 0x102e3bd1c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [~/.nvm/versions/node/v20.8.1/bin/node]
 8: 0x102e32970 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [~/.nvm/versions/node/v20.8.1/bin/node]
 9: 0x102e331d0 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [~/.nvm/versions/node/v20.8.1/bin/node]
10: 0x102e181d4 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [~/.nvm/versions/node/v20.8.1/bin/node]
11: 0x1031ffbe4 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [~/.nvm/versions/node/v20.8.1/bin/node]
12: 0x10355cc44 Builtins_CEntry_Return1_ArgvOnStack_NoBuiltinExit [~/.nvm/versions/node/v20.8.1/bin/node]
13: 0x108b63864 
14: 0x108995e5c 
15: 0x10896cd68 
16: 0x10898bdf4 
17: 0x108972a60 
18: 0x10896491c 
19: 0x10898eb98 
20: 0x1035b8fb8 Builtins_PromiseFulfillReactionJob [~/.nvm/versions/node/v20.8.1/bin/node]
21: 0x1034fab94 Builtins_RunMicrotasks [~/.nvm/versions/node/v20.8.1/bin/node]
22: 0x1034d23f4 Builtins_JSRunMicrotasksEntry [~/.nvm/versions/node/v20.8.1/bin/node]
23: 0x102daaae4 v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [~/.nvm/versions/node/v20.8.1/bin/node]
24: 0x102daafd0 v8::internal::(anonymous namespace)::InvokeWithTryCatch(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [~/.nvm/versions/node/v20.8.1/bin/node]
25: 0x102dab1ac v8::internal::Execution::TryRunMicrotasks(v8::internal::Isolate*, v8::internal::MicrotaskQueue*) [~/.nvm/versions/node/v20.8.1/bin/node]
26: 0x102dd2268 v8::internal::MicrotaskQueue::RunMicrotasks(v8::internal::Isolate*) [~/.nvm/versions/node/v20.8.1/bin/node]
27: 0x102dd2a04 v8::internal::MicrotaskQueue::PerformCheckpoint(v8::Isolate*) [~/.nvm/versions/node/v20.8.1/bin/node]
28: 0x102a14c64 node::InternalCallbackScope::Close() [~/.nvm/versions/node/v20.8.1/bin/node]
29: 0x102a1501c node::InternalMakeCallback(node::Environment*, v8::Local<v8::Object>, v8::Local<v8::Object>, v8::Local<v8::Function>, int, v8::Local<v8::Value>*, node::async_context) [~/.nvm/versions/node/v20.8.1/bin/node]
30: 0x102a2b48c node::AsyncWrap::MakeCallback(v8::Local<v8::Function>, int, v8::Local<v8::Value>*) [~/.nvm/versions/node/v20.8.1/bin/node]
31: 0x102ae7b48 node::fs::FSReqCallback::Resolve(v8::Local<v8::Value>) [~/.nvm/versions/node/v20.8.1/bin/node]
32: 0x102ae8298 node::fs::AfterNoArgs(uv_fs_s*) [~/.nvm/versions/node/v20.8.1/bin/node]
33: 0x102add1ec node::MakeLibuvRequestCallback<uv_fs_s, void (*)(uv_fs_s*)>::Wrapper(uv_fs_s*) [~/.nvm/versions/node/v20.8.1/bin/node]
34: 0x1034b0b9c uv__work_done [~/.nvm/versions/node/v20.8.1/bin/node]
35: 0x1034b45ec uv__async_io [~/.nvm/versions/node/v20.8.1/bin/node]
36: 0x1034c66c4 uv__io_poll [~/.nvm/versions/node/v20.8.1/bin/node]
37: 0x1034b4bb0 uv_run [~/.nvm/versions/node/v20.8.1/bin/node]
38: 0x102a15754 node::SpinEventLoopInternal(node::Environment*) [~/.nvm/versions/node/v20.8.1/bin/node]
39: 0x102b25114 node::NodeMainInstance::Run(node::ExitCode*, node::Environment*) [~/.nvm/versions/node/v20.8.1/bin/node]
40: 0x102b24eb0 node::NodeMainInstance::Run() [~/.nvm/versions/node/v20.8.1/bin/node]
41: 0x102aad60c node::Start(int, char**) [~/.nvm/versions/node/v20.8.1/bin/node]
42: 0x183eb50e0 start [/usr/lib/dyld]
fish: Job 1, 'npm run build' terminated by signal SIGABRT (Abort)

[olehmisar]

the cause is @aztec/bb.js package because it exports a 63.6mb file "dest/browser/index.js". It got bigger in version 0.35.0.

0.34.0 - 23.3mb
0.35.0 - 61mb
0.40.1 - 63.6mb

[ludamad]

Thanks, duly noted. This is indeed not ideal, we should put this larger debug-symbols-included version in another package but have not yet. Meanwhile, can you let us know how much RAM you have?

[olehmisar]

16gb. If i allow nodejs to use more memory via NODE_OPTIONS=--max_old_space_size=16384, it builds fine. The problem is that it cannot be build on servers like netlify, vercel etc.

[ludamad]

Thanks makes sense, helpful to know. I see someone was actually able to resolve such an issue on vercel sveltejs/kit#4795 wonder if you can set it in build step, but limited context so sorry if already tried

[ludamad]

Hi there! Just wanted to see if we had to do something here or if the setting could be applied in vercel (our preference for now). Were you able to try this?

[olehmisar]

@ludamad is it time to reduce bb.js size? More and more people are stumbling upon this issue

[ludamad]

It is now done, will be in next release

[ludamad]

... or it was in but strangely broke things, hopefully will be able to investigate why soon

[olehmisar]

btw, are you gonna keep BarretenbergSync around?

It does not play well with browsers because the whole aztec.js sdk depends on BarretenbergSync and something as simple as import { createPXEClient } from "@aztec/aztec.js" will block the whole execution graph until all wasm is fetched and executed. But it does not make sense because createPXEClient does not even use wasm.

[rahul-kothari]

@olehmisar as per our chat on discord DMs, could you confirm if increasing ram given to node in vercel fixed the build error

[olehmisar]

@rahul-kothari no, not fixed. As per comment above it still fails to build on vercel. Looks like vercel does not allow increasing RAM. Same thing with netlify. Increasing RAM only works locally.

#6527 (comment)

[ludamad]

Are you not able to change the build command Vercel runs? Does it not just do a yarn/npm build so you can fix like the command above that worked for svelte?

[olehmisar]

@ludamad

1. I can change the build command. But vercel/netlify just ignores NODE_OPTIONS=--max_old_space_size=16384 flag. Or vercel/netlify VPS just does not have 16gb of memory.
2. It's not sveltekit specific, it's vite specific.

You can try deploying this repro on vercel to see yourself https://github.com/ShieldSwap/aztec-build-out-of-memory-repro

[ludamad]

I didn't say it was svelte specific, I was giving an example that seemingly resolved it. We are not committed to supporting less than 16gb of RAM at this time so would appreciate if you could investigate more on vite side

…

On Mon, Sep 30, 2024, 4:39 PM oleh ***@***.***> wrote: @ludamad <https://github.com/ludamad> 1. I can change the build command. But vercel/netlify just ignores NODE_OPTIONS=--max_old_space_size=16384 flag. Or vercel/netlify VPS just does not have 16gb of memory. 2. It's not sveltekit specific, it's vite specific. You can try deploying this repro on vercel to see yourself https://github.com/ShieldSwap/aztec-build-out-of-memory-repro — Reply to this email directly, view it on GitHub <#6527 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABIBGJF5AMUVVVP764Z6LLZZGZIFAVCNFSM6AAAAABH5KEIN6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOBUGEYDIMZYG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[olehmisar]

@ludamad there is nothing to investigate. Vite uses rollup under the hood. Rollup crashes during build because it tries to load bb.js into RAM. Vercel/Netlify do not allow increasing memory during build time, so if you are not committed to supporting less than 16gb of RAM, then nobody will be able to deploy to Vercel/Netlify if they are using vite.

It affects not only Aztec but Noir as well(because Noir uses bb.js to prove).

[olehmisar]

The issue from svelte that you linked is marked as "completed" but it's not solving anything. They just marked it as duplicate of vitejs/vite#2433 (which is still open)

[ludamad]

Gotcha. It seems the hobby plans just have 1GB RAM, I will take a look sometime

[olehmisar]

@ludamad i believe you should avoid bundling WASM as an inline base64 url string. That's where the problem is. But not sure about Vite support of .wasm imports. It's not straightforward either. Though situation

[olehmisar]

Meanwhile, I created a vite plugin that fixes bb.js by importing it from unpkg in production builds.

https://github.com/ShieldSwap/vite-plugin-aztec

[olehmisar]

@ludamad could you please at least not minify the js bundle of bb.js? It would then be possible to patch bb.js in userland. Add this to webpack config of bb.js.

module.exports = {
  //...
  optimization: {
    minimize: false,
  },
};

[ludamad]

Sorry is this related to the memory bug?

[olehmisar]

@ludamad yes it is. If users can patch bb.js, vite will be able to build. A temporary workaround for a very cheap price

[pavitra-infocusp]

Agree with @olehmisar! The @aztec/bb.js package is huge!

[twt--]

@ludamad can this be prioritized, please? it's burning a lot of time for us devs building web stuff

[olehmisar]

Fixed in aztec.js 0.64.0

[rahul-kothari]

Appreciate you checking and closing all these Oleh :)