change_scene_to_file() does not work correctly in terms of VRAM

[KoBeWi]

Godot version

4.1 beta1

System information

Windows 10.0.19045 - Vulkan (Forward+) - dedicated NVIDIA GeForce GTX 1060 (NVIDIA; 30.0.15.1403) - Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz (8 Threads)

Issue description

The description of change_scene_to_file() says:

This ensures that both scenes are never loaded at the same time, which can exhaust system resources if the scenes are too large or if running in a memory constrained environment.

That's not really true. When you have 2 scenes with heavy textures, there's a visible VRAM spike when they are both loaded:

It kind of makes sense when looking at the implementation, but the same does not happen with RAM. When you have 2 heavy scenes (no textures), there is no spike. It's even more surprising that you can do:

var scene2 := load("res://HugeSceneRAM2.scn")
get_tree().change_scene_to_packed(scene2)

and there is still no RAM spike. There is some optimization behind the scenes here (if you use timer between load and change, the spike is visible).

Steps to reproduce

1. Load a heavy scene
2. Change to another heavy scene
3. Notice a VRAM jump while the scenes are switched

Minimal reproduction project

I prepared a project with both RAM and VRAM scenario:
HeavyScenes.zip

For the record, I tried fixing this by making the scene change more deferred, but it just results in empty frame. The VRAM spike disappears only when you defer the scene change with a timer. Here's my attempt:
0001-Fix-_change_scene_to_file.patch