Add lightmap bake canceling

[badsectoracula]

This enables the "Cancel" button in the lightmap baking progress window and checks the cancel request whenever a progress is updated/reported during lightmap baking. Most of the functionality was already there, including a potential cancel request by the callback, but callback's return value was never checked.

This adds some checks where possible. A couple of progress updates wouldn't forward a cancellation result so i didn't add them but in practice i never saw them when testing cancelling (though i mainly tested with the GI demo). The main part that takes most of the time, indirect light integration, does get cancelled.

The patch just stops the lightmap calculation when it gets cancelled. I didn't notice this creating any leaks, but i might have missed them as this is the first time i saw the lightmapper code, so someone who is more familiar with this should review it.

This should address godotengine/godot-proposals#8181.

[fire]

The proposal enhancement makes sense to me.

I have yet to review the pull request, but this is good from a usability point of view. Cancelling a potentially 30-minute plus bake is essential rather than forcing quitting.

[Calinou]

Tested locally, it works as expected.

However, a memory leak occurs if I cancel baking while indirect light is being calculated:

lightmap_bake_leak.mp4

Every time I cancel baking, VRAM utilization grows by 600 MB without shrinking after.

No leak occurs if I cancel baking just after it starts (by pressing Space immediately after the progress dialog pops up, which presses the autofocused Cancel button).

[clayjohn]

However, a memory leak occurs if I cancel baking while indirect light is being calculated:

I'm not surprised, these early return error conditions will need to free the GPU resources before returning as we do with the current error handling.

See for example

godot/modules/lightmapper_rd/lightmapper_rd.cpp

Lines 1453 to 1465 in 9e60984

	if (err != OK) {
	FREE_TEXTURES
	FREE_BUFFERS
	FREE_RASTER_RESOURCES
	memdelete(rd);
	if (rcd != nullptr) {
	memdelete(rcd);
	}
	compute_shader->print_errors("compute_shader");
	}
	ERR_FAIL_COND_V(err != OK, BAKE_ERROR_LIGHTMAP_CANT_PRE_BAKE_MESHES);

[badsectoracula]

I updated the PR, it should release memory now (thanks to @Calinou 's video i also learned the AMD equivalent to the tool he uses - amdgpu_top - so i ran a bunch of bakes and cancelled them at various points and memory returned at the pre-bake numbers).

There were already some existing early exits in the function so i thought it was ok to exit, i guess they are also leaks. There are a lot of exit points in this so it should probably be refactored to have a single exit point with some sort of "if (error) goto fail" at the end of the function that cleans up everything, so perhaps someone more familiar with the code could clean it up (i could do it in the future if nobody does but i think some stuff may need to be moved around and at the moment i do not feel confident enough to touch anything that would affect the code's logic).

[clayjohn]

Looks good to me!

[Calinou]

Tested locally, there are no more memory leaks now, regardless of how many times I cancel the bake (at different stages).

[Repiteo]

Thanks! Congratulations on your first merged contribution! 🎉