Add support for per-vertex lighting functions in shaders

[bunabyt3]

Describe the project you are working on

I'm working on a game in the style of something from the era of Half-Life 1 or Unreal Tournament. These games didn't use shaders, but they did achieve some interesting effects using the fixed-function pipeline; since we have to use shaders in Godot, as fixed-function is considered obsolete, I figured I would try recreating those effects, as well as adding new ones specific to my game in the same style.

Describe the problem or limitation you are having in your project

Godot 4.4 provides the ability to enable vertex lighting, which I think is great. However, custom lighting functions are not supported in this mode, forcing us to use Lambertian diffuse instead of any custom lighting model. The shader used by the StandardMaterial is able to use a more advanced lighting model which includes specular lighting, but doesn't support things like Blinn-Phong specular that would work better in vertex-lit mode.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

I think that when vertex lighting is turned on in a project, the light() function inside user-created shaders should be used for shading vertices rather than pixels. Alternatively, a separate vertex_light() function could be defined in a shader to use a custom vertex lighting model. This would allow for fully custom lighting models in vertex-lit mode, expanding the possibilities for vertex lighting effects instead of forcing everything to use Lambertian diffuse.
Another nice thing to have would be a feature where the results of light() calculations can be accessed inside of the fragment shader; this would allow, for example, creating a toon shader without having to use unnecessarily expensive (or aesthetically inappropriate) per-pixel lighting.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

This seems like something fairly simple to implement, but since I don't quite know my way around the rendering code, I'm not sure if it is. However, if my assumptions are correct, then one could simply:

• Check if vertex lighting is enabled
• If so, run the light() function per-vertex instead of per-pixel

I wasn't able to implement this on my own, so feel free to tell me how realistic of a request it really is.

If this enhancement will not be used often, can it be worked around with a few lines of script?

As far as I'm aware, scripts can't be used to inject or modify render passes, but please correct me if I'm wrong.

Is there a reason why this should be core and not an add-on in the asset library?

I believe the limits of what's possible without modifying the engine code would prevent such an add-on from being created. Once again, please correct me if I'm wrong.

[Calinou]

cc @ywmaa, as they implemented vertex shading in godotengine/godot#83360.

I figured I would try recreating those effects, as well as adding new ones specific to my game in the same style.

Which effects are you thinking about creating? Note that it's possible to emulate the per-vertex look using fragment shaders. It's less efficient but more flexible. If the shader is to be used on a few surfaces only, it might be a good enough solution.

[bunabyt3]

@Calinou At the moment, I want to use Blinn-Phong specular lighting, which I've already implemented in the light() function of my shader. The important thing here is that the light() function should be per-vertex when vertex lighting is enabled, and per-pixel when it isn't, regardless of use case.