Editor crash on project open with specific shader using varyings - NVIDIA GPU on Windows

[snowflower]

Tested versions

Not Reproducible in Godot 4.3 Stable

100% Reproducible in Godot 4.4-dev5

System information

Godot v4.4.dev5 - Windows 10.0.22631 - Multi-window, 2 monitors - Vulkan (Forward+) - dedicated NVIDIA GeForce RTX 3080 Ti Laptop GPU (NVIDIA; 32.0.15.6614) - 12th Gen Intel(R) Core(TM) i9-12900HK (20 threads)

Issue description

Attempted to migrate a functioning project from Godot 4.3 Stable to 4.4.5dev:

After opening project , the editor will crash before it loads the main scene file.

I have tried replacing the main scene file it loads with an empty one (by manually editing project.godot) and it still crashes.

I have deleted the .godot folder forcing a complete reimport of all assets, and updated graphics drivers to the latest version.
The only thing that has changed compared to an earlier reproduction test was the breadcrumb ID in the error output
this is the commandline I used to get this output logged to a text file, since running it in console mode by default gave no info and immediately closed upon crash:

Godot_v4.4-dev5_win64_console.exe --debug --accurate-breadcrumbs --verbose --log-file testlog.txt --editor --path C:\gunlab

relevant section at the end of the log:

ERROR: Printing last known breadcrumbs in reverse order (last executed first).
   at: print_lost_device_info (drivers/vulkan/rendering_device_driver_vulkan.cpp:5325)
ERROR: Searching last breadcrumb. We've sent up to ID: 118
ERROR: Last breadcrumb ID found: 114
ERROR: Last known breadcrumb: BLIT_PASS
ERROR: Last known breadcrumb: UI_PASS
ERROR: Last known breadcrumb: BLIT_PASS
ERROR: Last known breadcrumb: UI_PASS
ERROR: Last known breadcrumb: BLIT_PASS
ERROR: Last known breadcrumb: UI_PASS
ERROR: Last known breadcrumb: BLIT_PASS
ERROR: Last known breadcrumb: UI_PASS

================================================================
CrashHandlerException: Program crashed with signal 11
Engine version: Godot Engine v4.4.dev5.official (9e6098432aac35bae42c9089a29ba2a80320d823)
Dumping the backtrace. Please include this when reporting the bug to the project developer.
[1] error(-1): no debug info in PE/COFF executable
[2] error(-1): no debug info in PE/COFF executable
[3] error(-1): no debug info in PE/COFF executable
[4] error(-1): no debug info in PE/COFF executable
[5] error(-1): no debug info in PE/COFF executable
[6] error(-1): no debug info in PE/COFF executable
[7] error(-1): no debug info in PE/COFF executable
[8] error(-1): no debug info in PE/COFF executable
[9] error(-1): no debug info in PE/COFF executable
[10] error(-1): no debug info in PE/COFF executable
[11] error(-1): no debug info in PE/COFF executable
[12] error(-1): no debug info in PE/COFF executable
[13] error(-1): no debug info in PE/COFF executable
[14] error(-1): no debug info in PE/COFF executable
[15] error(-1): no debug info in PE/COFF executable
[16] error(-1): no debug info in PE/COFF executable
[17] error(-1): no debug info in PE/COFF executable
[18] error(-1): no debug info in PE/COFF executable
[19] error(-1): no debug info in PE/COFF executable
[20] error(-1): no debug info in PE/COFF executable
-- END OF BACKTRACE --
================================================================

Steps to reproduce

reproducible locally using the steps outlined in the issue description.

Minimal reproduction project (MRP)

No MRP available, I test ported some smaller standalone projects using some of the assets and shaders and did not encounter this crash.
I am still attempting to reproduce and find an MRP case to upload.
Attached is a full verbose output log of the crash.

testlog.txt

MRP: mrpVulkanCrash_4.4-dev6.zip

Bugsquad edit: added MRP from OP comment

[snowflower]

after a reboot, i'm still getting the vulkan error and editor crashes on opening the full project running the godot console.exe
--gpu-validation isnt giving me any extra information

C:\Godot_v4.4-dev5_win64\Godot_v4.4-dev5_win64_console.exe --gpu-validation --verbose --debug --accurate-breadcrumbs --log-file godotlog_4.4-dev5.txt

[caspercasanova]

I too get a vulkan error on starting up a new project. Here's a picture of the console when I start up a new project

This occurs for both 4.4 dev 4 & 4.4 dev 5 but not for 4.4 dev 3

[snowflower]

attempting to run my project that functions fine on 4.3stable, on 4.4-dev3 with no commandline options,
I get a new error about some RID , and the project fails to load and editor shuts down.

[snowflower]

migrating my 4.3stable project to Godot 4.4-dev1 does not give any warning/errors and loads the project successfully without the editor crashing like it does in 4.4-dev3 and 4.4-dev5

ran godot in console mode with no commandline

[snowflower]

tested my project on godot 4.4-dev2 and loads and runs without issue.
running console with no commandline

So there is something starting with godot 4.4-dev3 that causes my project to fail loading and crashes.

[balloonpopper]

Might be related to #99064

[adamscott]

Could you run without the godot_debug_draw_3d addon, @snowflower ?

[snowflower]

I removed the debug draw addon from the project since I was not using it, and I still have the crash.

I have tried using the latest version Godot 4.4-dev6
have had 1 instance of hard bluescreen crash and pc resetting trying the latest version, and another test run with the following errors in the outpout log before I get a vulkan crash output to the console, and then the editor and console window closes, similar to the posts above.

I have also updated the machine to the latest windows update and NVIDIA game ready drivers released on Dec5th (version 566.36 )

I have also deleted the .godot folder of the project to see if it was a problem with cached files and make it force reimport everything, but that does not fix the crash.
This is always the visible UI element with progress bar stuck, it hangs on this an extremely long time on before the crash

[snowflower]

Okay i have finally found the issue, after isolating a material in a scene, that uses a shader that caused the crash.
After incorrectly assuming it might be the complex fragment program causing the issue, it turns out to be the use of a varyings declared of different sizes in a particular ordering.

I systematically pruned the shader down to a minimal representation that still triggers the crash.
I assume causes some bad code generation/driver issue to make things go boom.

Reproduction steps:
Create a shader with the following code.
Create a shader material and assign this shader to it.
Click the material so it opens in the inspector panel or have an object in a scene that uses it and have that scene opened/load.

Editor will crash 100%
Tested on Godot 4.4-dev6

shader_type spatial;

varying vec3 v_objectscale;// commenting out all other varyings makes this stop crashing

#if 1
// dont change the order of varyings, this seems to be the cause of the problem
varying vec4 a; // commenting this one out, and leaving all the others will make it not crash
varying mat4 b;
varying mat4 c;
varying mat3 d;
varying vec3 e;
varying vec3 f;
varying mat4 g;
#endif

void vertex() {

#if 1
	// commenting this code out will cause the crash to go away,
	// since there is no generated code or reference to any varying
	//
	vec3 objScaleUnrotatedMtx;
	objScaleUnrotatedMtx.x = length(MODEL_MATRIX[0].xyz);
	objScaleUnrotatedMtx.y = length(MODEL_MATRIX[1].xyz);
	objScaleUnrotatedMtx.z = length(MODEL_MATRIX[2].xyz);
	v_objectscale = objScaleUnrotatedMtx;
#endif

}

void fragment() {
	ALBEDO = vec3(1,0,0);
}

[snowflower]

Attached is the MRP
Editor should crash just trying to open the project.

mrpVulkanCrash_4.4-dev6.zip

[akien-mga]

@smix8 was able to reproduce the crash on latest master (296de7d) and get a backtrace:

Vulkan 1.3.289 - Forward+ - Using Device #0: NVIDIA - NVIDIA GeForce RTX 3090 Ti

ERROR: Printing last known breadcrumbs in reverse order (last executed first).
Some of them might be inaccurate. Try running with --accurate-breadcrumbs for precise information.
   at: RenderingDeviceDriverVulkan::print_lost_device_info (drivers\vulkan\rendering_device_driver_vulkan.cpp:5557)
ERROR: Searching last breadcrumb. We've sent up to ID: 306
ERROR: Last breadcrumb ID found: 302
ERROR: Last known breadcrumb: SKY_PASS | User data: 5
ERROR: Last known breadcrumb: SKY_PASS | User data: 4
ERROR: Last known breadcrumb: SKY_PASS | User data: 3
ERROR: Last known breadcrumb: SKY_PASS | User data: 2
ERROR: Last known breadcrumb: SKY_PASS | User data: 1
ERROR: Last known breadcrumb: SKY_PASS
ERROR: Last known breadcrumb: BLIT_PASS
ERROR: Last known breadcrumb: UI_PASS

================================================================
CrashHandlerException: Program crashed
Engine version: Godot Engine v4.4.beta.custom_build (261e7d32d37445de3ef3a9804346552fdac6096e)
Dumping the backtrace. Please include this when reporting the bug to the project developer.
[0] <couldn't map PC to fn name>
[1] RenderingDeviceDriverVulkan::on_device_lost (H:\files\github\godot\godot_4.x_fork\drivers\vulkan\rendering_device_driver_vulkan.cpp:5481)
[2] RenderingDeviceDriverVulkan::print_lost_device_info (H:\files\github\godot\godot_4.x_fork\drivers\vulkan\rendering_device_driver_vulkan.cpp:5668)
[3] RenderingDeviceDriverVulkan::command_queue_execute_and_present (H:\files\github\godot\godot_4.x_fork\drivers\vulkan\rendering_device_driver_vulkan.cpp:2596)
[4] RenderingDevice::_submit_transfer_worker (H:\files\github\godot\godot_4.x_fork\servers\rendering\rendering_device.cpp:5689)
[5] RenderingDevice::_submit_transfer_workers (H:\files\github\godot\godot_4.x_fork\servers\rendering\rendering_device.cpp:5786)
[6] RenderingDevice::_end_frame (H:\files\github\godot\godot_4.x_fork\servers\rendering\rendering_device.cpp:6411)
[7] RenderingDevice::_flush_and_stall_for_all_frames (H:\files\github\godot\godot_4.x_fork\servers\rendering\rendering_device.cpp:6600)
[8] RenderingDevice::screen_prepare_for_drawing (H:\files\github\godot\godot_4.x_fork\servers\rendering\rendering_device.cpp:4144)
[9] RendererCompositorRD::blit_render_targets_to_screen (H:\files\github\godot\godot_4.x_fork\servers\rendering\renderer_rd\renderer_compositor_rd.cpp:40)
[10] RendererViewport::draw_viewports (H:\files\github\godot\godot_4.x_fork\servers\rendering\renderer_viewport.cpp:921)[11] RenderingServerDefault::_draw (H:\files\github\godot\godot_4.x_fork\servers\rendering\rendering_server_default.cpp:86)
[12] RenderingServerDefault::draw (H:\files\github\godot\godot_4.x_fork\servers\rendering\rendering_server_default.cpp:415)
[13] Main::iteration (H:\files\github\godot\godot_4.x_fork\main\main.cpp:4517)
[14] OS_Windows::run (H:\files\github\godot\godot_4.x_fork\platform\windows\os_windows.cpp:2062)
[15] widechar_main (H:\files\github\godot\godot_4.x_fork\platform\windows\godot_windows.cpp:96)
[16] _main (H:\files\github\godot\godot_4.x_fork\platform\windows\godot_windows.cpp:122)
[17] main (H:\files\github\godot\godot_4.x_fork\platform\windows\godot_windows.cpp:136)
[18] WinMain (H:\files\github\godot\godot_4.x_fork\platform\windows\godot_windows.cpp:150)
[19] __scrt_common_main_seh (D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)
[20] <couldn't map PC to fn name>
-- END OF BACKTRACE --

[akien-mga]

CC @godotengine/rendering @godotengine/shaders

According to #99587 (comment) the regression was introduced in 4.4.dev3.

There's a MRP linked in the OP.
And relevant reproduction steps in #99587 (comment) for a minimal reproducer. Sounds like a potential alignment/memory issue?

[smix8]

No crash with 4.4 dev2 but crashes with 4.4 dev3.

Bisected to #83360

[clayjohn]

I suspect that this is a duplicate of #76667. Basically the issue is that most GPUs only support 32 varyings. The shader from this issue uses 20 varyings and Godot reserves 14, so together they go over the limit.

I think that some drivers are able to repurpose the reserved but unused varyings so they don't have the same issue. As I commented on #76667, the solution will be for us to validate the number of varyings used in user shaders since right now users can exceed the limit and crash their systems without having any feedback that they did something wrong.

#64948 is the same issue as well

[clayjohn]

On an AMD system with validation layers enabled, I get the following:

ERROR: VALIDATION - Message Id Number: -1553903733 | Message Id Name: VUID-RuntimeSpirv-Location-06272
        Validation Error: [ VUID-RuntimeSpirv-Location-06272 ] Object 0: VK_NULL_HANDLE, type = VK_OBJECT_TYPE_PIPELINE; | MessageID = 0xa3614f8b | Invalid Pipeline CreateInfo State: Vertex shader output variable uses location that exceeds component limit VkPhysicalDeviceLimits::maxVertexOutputComponents (128) The Vulkan spec states: The sum of Location and the number of locations the variable it decorates consumes must be less than or equal to the value for the matching {ExecutionModel} defined in Shader Input and Output Locations (https://vulkan.lunarg.com/doc/view/1.3.224.1/windows/1.3-extensions/vkspec.html#VUID-RuntimeSpirv-Location-06272)
        Objects - 1
                Object[0] - VK_OBJECT_TYPE_PIPELINE, Handle 0