Terrible memory leak when importing a large git cloned repository.

[cryoboltinteractive]

Tested versions

Reproducible in - v4.3.stable.mono.official [77dcf97]

System information

Godot v4.3.stable.mono - Windows 10.0.22631 - Vulkan (Forward+) - dedicated NVIDIA GeForce RTX 2050 (NVIDIA; 32.0.15.5612) - 11th Gen Intel(R) Core(TM) i5-11260H @ 2.60GHz (12 Threads)

Issue description

A Large Godot Project hosted on a Git server (4 GB of Assets, >50,000 Files) when cloned and imported without the (.godot) folder, in a machine with 16 GB of RAM will use all system memory and will 100% of the time result in a full system crash, not just the godot Engine. This happens at around 96% to 97% of the import.
Tested on the following Machines
Macbook Pro 15 inch 2017
(Core i7 - 7820HQ, 16GB DDR4 RAM, Radeon Pro 560) (System Crashed) (5 Attempts)
Asus Zepherus G14
(AMD Ryzen 7 6800HS, 16GB DDR5, RTX 3060 ) (BSOD) (4 Attempts)
Asus TUF F15
(Intel Core i7 11260H, 16 GB DDR4, RTX 2050) (System Crashed) (2 Attempts)

Asus TUF F15 (After Upgrade)
(Intel Core i7 11260H, 32 GB DDR4, RTX 2050) (Import Success) (First Attempt)
Desktop 1
(AMD Ryzen 5 5600, 32 GB DDR4, RTX 3070) (Import Success) (First Attempt)
Desktop 2
(AMD Ryzen 5 3600, 32GB DDR4, RTX 2070) (Import Success) (First Attempt)

However even on 32GB of RAM it uses all resources and even after importing, it will continue to use in excess of 22GB RAM

but if you restart Godot after importing everything, it will not use more than 2GB of RAM

Steps to reproduce

We cannot unfortunately share steps as it is just importing an existing project

Minimal reproduction project (MRP)

We cannot provide an MRP as the licenses on certain assets do not allow us, how ever we are always available on discord call to talk and maybe help rectify the issue.

Equivalent MRP is provide below (Comment 5)

[Nazarwadim]

I can confirm the problem at https://github.com/godotengine/tps-demo. Appears in ResourceCache. Once exported, it has over 300 resources and uses 1.5 GB of RAM. After restarting the project there are only 6 resources and 0.6 GB memory usage.

[cryoboltinteractive]

@Nazarwadim Can you confirm if it was on import? Our issue was raised on importing a project, you have mentioned on export, so we're just trying to clarify.
On our internal test with the TPS demo, on import it uses 1.4 GB of RAM and on restart it uses ~750MB. Probably not a huge deal, but on large projects this issue is multiplied.

[Nazarwadim]

Yes on import. Made a typo

[cryoboltinteractive]

Project imported on the Asus G14 with 40GB of RAM, (8GB Soldered + 32GB DIMM), it also used 97% of total RAM to import.
So I guess it uses all the RAM Available? This is kinda concerning as the total project size is under 4GB.

[cryoboltinteractive]

ImportCrashTest.zip

Created an Equivalent MRP with the same issue, 100 MB project will use more that 6GB of RAM to import and stay at that usage until engine restart.

Credits Kenney.nl for the CC0 Assets.
The files are similar to how we have our files

The real issue happens when we set the Mesh Paths in the import settings, only then the Insane RAM usage happens.

Edit - It happens with individual files as well

[cryoboltinteractive]

not only is RAM getting maxed out even pagefile (Commited memory) is being hammered.
This is using 4 Synty packs only
even with ufbx, this issue will be a big no no for 3D game develpments.
again this only happens during import, not when using the engine.

[Nazarwadim]

@cryoboltinteractive If you open your large imported project and work in it for a certain time (say an hour), will the memory usage increase to the same level?

By the way, this code shows the same result 558 MB, despite the fact that one non-imported one takes 5GB, and the imported one takes 1GB.

@tool
extends Node

var value = 0
func _process(delta: float) -> void:
	value += 1
	if(value % 250):
		return
	print(OS.get_static_memory_usage() / 1000000, " MB")

[cryoboltinteractive]

@Nazarwadim , once imported it doesn't use much RAM. The memory usage happens during Import only.
About the the OS.get_static_memory_usage I'm not sure

@nikitalita , our issue happens even with other file formats like glb and fbx (using ufbx).
But thank you for the insights!

[Nazarwadim]

I FOUND IT. I will make a PR soon.

[cryoboltinteractive]

Congratulations!
@Nazarwadim You worked real hard on this
So this update will come in 4.4 I'm guessing?

[Nazarwadim]

So this update will come in 4.4 I'm guessing?

Yes