Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Metal: Add MetalFX upscaling support #99603

Merged
merged 1 commit into from
Jan 6, 2025
Merged

Metal: Add MetalFX upscaling support #99603

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions doc/classes/ProjectSettings.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3024,6 +3024,12 @@
Sets the scaling 3D mode. Bilinear scaling renders at different resolution to either undersample or supersample the viewport. FidelityFX Super Resolution 1.0, abbreviated to FSR, is an upscaling technology that produces high quality images at fast framerates by using a spatially-aware upscaling algorithm. FSR is slightly more expensive than bilinear, but it produces significantly higher image quality. On particularly low-end GPUs, the added cost of FSR may not be worth it (compared to using bilinear scaling with a slightly higher resolution scale to match performance).
[b]Note:[/b] FSR is only effective when using the Forward+ rendering method, not Mobile or Compatibility. If using an incompatible rendering method, FSR will fall back to bilinear scaling.
</member>
<member name="rendering/scaling_3d/mode.ios" type="int" setter="" getter="">
iOS override for [member rendering/scaling_3d/mode]. This allows selecting the MetalFX spatial and MetalFX temporal scaling modes, which are exclusive to platforms where the Metal rendering driver is used.
</member>
<member name="rendering/scaling_3d/mode.macos" type="int" setter="" getter="">
macOS override for [member rendering/scaling_3d/mode]. This allows selecting the MetalFX spatial and MetalFX temporal scaling modes, which are exclusive to platforms where the Metal rendering driver is used.
</member>
<member name="rendering/scaling_3d/scale" type="float" setter="" getter="" default="1.0">
Scales the 3D render buffer based on the viewport size uses an image filter specified in [member rendering/scaling_3d/mode] to scale the output image to the full viewport size. Values lower than [code]1.0[/code] can be used to speed up 3D rendering at the cost of quality (undersampling). Values greater than [code]1.0[/code] are only valid for bilinear mode and can be used to improve 3D rendering quality at a high performance cost (supersampling). See also [member rendering/anti_aliasing/quality/msaa_3d] for multi-sample antialiasing, which is significantly cheaper but only smooths the edges of polygons.
</member>
Expand Down
8 changes: 8 additions & 0 deletions doc/classes/RenderingDevice.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2529,6 +2529,14 @@
<constant name="LIMIT_MAX_VIEWPORT_DIMENSIONS_Y" value="36" enum="Limit">
Maximum viewport height (in pixels).
</constant>
<constant name="LIMIT_METALFX_TEMPORAL_SCALER_MIN_SCALE" value="46" enum="Limit">
Returns the smallest value for [member ProjectSettings.rendering/scaling_3d/scale] when using the MetalFX temporal upscaler.
[b]Note:[/b] The returned value is multiplied by a factor of [code]1000000[/code] to preserve 6 digits of precision. It must be divided by [code]1000000.0[/code] to convert the value to a floating point number.
</constant>
<constant name="LIMIT_METALFX_TEMPORAL_SCALER_MAX_SCALE" value="47" enum="Limit">
Returns the largest value for [member ProjectSettings.rendering/scaling_3d/scale] when using the MetalFX temporal upscaler.
[b]Note:[/b] The returned value is multiplied by a factor of [code]1000000[/code] to preserve 6 digits of precision. It must be divided by [code]1000000.0[/code] to convert the value to a floating point number.
</constant>
<constant name="MEMORY_TEXTURES" value="0" enum="MemoryType">
Memory taken by textures.
</constant>
Expand Down
10 changes: 9 additions & 1 deletion doc/classes/RenderingServer.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4986,7 +4986,15 @@
<constant name="VIEWPORT_SCALING_3D_MODE_FSR2" value="2" enum="ViewportScaling3DMode">
Use AMD FidelityFX Super Resolution 2.2 upscaling for the viewport's 3D buffer. The amount of scaling can be set using [member Viewport.scaling_3d_scale]. Values less than [code]1.0[/code] will be result in the viewport being upscaled using FSR2. Values greater than [code]1.0[/code] are not supported and bilinear downsampling will be used instead. A value of [code]1.0[/code] will use FSR2 at native resolution as a TAA solution.
</constant>
<constant name="VIEWPORT_SCALING_3D_MODE_MAX" value="3" enum="ViewportScaling3DMode">
<constant name="VIEWPORT_SCALING_3D_MODE_METALFX_SPATIAL" value="3" enum="ViewportScaling3DMode">
Use MetalFX spatial upscaling for the viewport's 3D buffer. The amount of scaling can be set using [member Viewport.scaling_3d_scale]. Values less than [code]1.0[/code] will be result in the viewport being upscaled using MetalFX. Values greater than [code]1.0[/code] are not supported and bilinear downsampling will be used instead. A value of [code]1.0[/code] disables scaling.
[b]Note:[/b] Only supported when the Metal rendering driver is in use, which limits this scaling mode to macOS and iOS.
</constant>
<constant name="VIEWPORT_SCALING_3D_MODE_METALFX_TEMPORAL" value="4" enum="ViewportScaling3DMode">
Use MetalFX temporal upscaling for the viewport's 3D buffer. The amount of scaling can be set using [member Viewport.scaling_3d_scale]. Values less than [code]1.0[/code] will be result in the viewport being upscaled using MetalFX. Values greater than [code]1.0[/code] are not supported and bilinear downsampling will be used instead. A value of [code]1.0[/code] will use MetalFX at native resolution as a TAA solution.
[b]Note:[/b] Only supported when the Metal rendering driver is in use, which limits this scaling mode to macOS and iOS.
</constant>
<constant name="VIEWPORT_SCALING_3D_MODE_MAX" value="5" enum="ViewportScaling3DMode">
Represents the size of the [enum ViewportScaling3DMode] enum.
</constant>
<constant name="VIEWPORT_UPDATE_DISABLED" value="0" enum="ViewportUpdateMode">
Expand Down
16 changes: 15 additions & 1 deletion doc/classes/Viewport.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -503,7 +503,21 @@
<constant name="SCALING_3D_MODE_FSR2" value="2" enum="Scaling3DMode">
Use AMD FidelityFX Super Resolution 2.2 upscaling for the viewport's 3D buffer. The amount of scaling can be set using [member Viewport.scaling_3d_scale]. Values less than [code]1.0[/code] will be result in the viewport being upscaled using FSR2. Values greater than [code]1.0[/code] are not supported and bilinear downsampling will be used instead. A value of [code]1.0[/code] will use FSR2 at native resolution as a TAA solution.
</constant>
<constant name="SCALING_3D_MODE_MAX" value="3" enum="Scaling3DMode">
<constant name="SCALING_3D_MODE_METALFX_SPATIAL" value="3" enum="Scaling3DMode">
Use the [url=https://developer.apple.com/documentation/metalfx/mtlfxspatialscaler#overview]MetalFX spatial upscaler[/url] for the viewport's 3D buffer.
The amount of scaling can be set using [member scaling_3d_scale].
Values less than [code]1.0[/code] will be result in the viewport being upscaled using MetalFX. Values greater than [code]1.0[/code] are not supported and bilinear downsampling will be used instead. A value of [code]1.0[/code] disables scaling.
More information: [url=https://developer.apple.com/documentation/metalfx]MetalFX[/url].
[b]Note:[/b] Only supported when the Metal rendering driver is in use, which limits this scaling mode to macOS and iOS.
</constant>
<constant name="SCALING_3D_MODE_METALFX_TEMPORAL" value="4" enum="Scaling3DMode">
Use the [url=https://developer.apple.com/documentation/metalfx/mtlfxtemporalscaler#overview]MetalFX temporal upscaler[/url] for the viewport's 3D buffer.
The amount of scaling can be set using [member scaling_3d_scale]. To determine the minimum input scale, use the [method RenderingDevice.limit_get] method with [constant RenderingDevice.LIMIT_METALFX_TEMPORAL_SCALER_MIN_SCALE].
Values less than [code]1.0[/code] will be result in the viewport being upscaled using MetalFX. Values greater than [code]1.0[/code] are not supported and bilinear downsampling will be used instead. A value of [code]1.0[/code] will use MetalFX at native resolution as a TAA solution.
More information: [url=https://developer.apple.com/documentation/metalfx]MetalFX[/url].
[b]Note:[/b] Only supported when the Metal rendering driver is in use, which limits this scaling mode to macOS and iOS.
</constant>
<constant name="SCALING_3D_MODE_MAX" value="5" enum="Scaling3DMode">
Represents the size of the [enum Scaling3DMode] enum.
</constant>
<constant name="MSAA_DISABLED" value="0" enum="MSAA">
Expand Down
5 changes: 5 additions & 0 deletions drivers/metal/metal_device_properties.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,8 @@ struct API_AVAILABLE(macos(11.0), ios(14.0)) MetalFeatures {
bool tessellationShader = false; /**< If true, tessellation shaders are supported. */
bool imageCubeArray = false; /**< If true, image cube arrays are supported. */
MTLArgumentBuffersTier argument_buffers_tier = MTLArgumentBuffersTier1;
bool metal_fx_spatial = false; /**< If true, Metal FX spatial functions are supported. */
bool metal_fx_temporal = false; /**< If true, Metal FX temporal functions are supported. */
};

struct MetalLimits {
Expand Down Expand Up @@ -115,6 +117,9 @@ struct MetalLimits {
uint32_t maxVertexInputBindingStride;
uint32_t maxDrawIndexedIndexValue;

double temporalScalerInputContentMinScale;
double temporalScalerInputContentMaxScale;

uint32_t minSubgroupSize; /**< The minimum number of threads in a SIMD-group. */
uint32_t maxSubgroupSize; /**< The maximum number of threads in a SIMD-group. */
BitField<RDD::ShaderStage> subgroupSupportedShaderStages;
Expand Down
15 changes: 15 additions & 0 deletions drivers/metal/metal_device_properties.mm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@
#import "metal_device_properties.h"

#import <Metal/Metal.h>
#import <MetalFX/MetalFX.h>
#import <spirv_cross.hpp>
#import <spirv_msl.hpp>

Expand Down Expand Up @@ -100,6 +101,11 @@
features.simdReduction = [p_device supportsFamily:MTLGPUFamilyApple7];
features.argument_buffers_tier = p_device.argumentBuffersSupport;

if (@available(macOS 13.0, iOS 16.0, *)) {
features.metal_fx_spatial = [MTLFXSpatialScalerDescriptor supportsDevice:p_device];
features.metal_fx_temporal = [MTLFXTemporalScalerDescriptor supportsDevice:p_device];
}

MTLCompileOptions *opts = [MTLCompileOptions new];
features.mslVersionEnum = opts.languageVersion; // By default, Metal uses the most recent language version.

Expand Down Expand Up @@ -285,6 +291,15 @@
#endif

limits.maxDrawIndexedIndexValue = std::numeric_limits<uint32_t>::max() - 1;

if (@available(macOS 14.0, iOS 17.0, *)) {
limits.temporalScalerInputContentMinScale = (double)[MTLFXTemporalScalerDescriptor supportedInputContentMinScaleForDevice:p_device];
limits.temporalScalerInputContentMaxScale = (double)[MTLFXTemporalScalerDescriptor supportedInputContentMaxScaleForDevice:p_device];
} else {
// Defaults taken from macOS 14+
limits.temporalScalerInputContentMinScale = 1.0;
limits.temporalScalerInputContentMaxScale = 3.0;
}
}

MetalDeviceProperties::MetalDeviceProperties(id<MTLDevice> p_device) {
Expand Down
8 changes: 8 additions & 0 deletions drivers/metal/rendering_device_driver_metal.mm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3982,6 +3982,10 @@ bool isArrayTexture(MTLTextureType p_type) {
return (uint64_t)limits.subgroupSupportedShaderStages;
case LIMIT_SUBGROUP_OPERATIONS:
return (uint64_t)limits.subgroupSupportedOperations;
case LIMIT_METALFX_TEMPORAL_SCALER_MIN_SCALE:
return (uint64_t)((1.0 / limits.temporalScalerInputContentMaxScale) * 1000'000);
case LIMIT_METALFX_TEMPORAL_SCALER_MAX_SCALE:
return (uint64_t)((1.0 / limits.temporalScalerInputContentMinScale) * 1000'000);
UNKNOWN(LIMIT_VRS_TEXEL_WIDTH);
UNKNOWN(LIMIT_VRS_TEXEL_HEIGHT);
UNKNOWN(LIMIT_VRS_MAX_FRAGMENT_WIDTH);
Expand Down Expand Up @@ -4017,6 +4021,10 @@ bool isArrayTexture(MTLTextureType p_type) {
return false;
case SUPPORTS_FRAGMENT_SHADER_WITH_ONLY_SIDE_EFFECTS:
return true;
case SUPPORTS_METALFX_SPATIAL:
return device_properties->features.metal_fx_spatial;
case SUPPORTS_METALFX_TEMPORAL:
return device_properties->features.metal_fx_temporal;
default:
return false;
}
Expand Down
1 change: 1 addition & 0 deletions platform/ios/detect.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -160,6 +160,7 @@ def configure(env: "SConsEnvironment"):
env.Prepend(
CPPPATH=[
"$IOS_SDK_PATH/System/Library/Frameworks/Metal.framework/Headers",
"$IOS_SDK_PATH/System/Library/Frameworks/MetalFX.framework/Headers",
"$IOS_SDK_PATH/System/Library/Frameworks/QuartzCore.framework/Headers",
]
)
Expand Down
1 change: 1 addition & 0 deletions platform/macos/detect.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -243,6 +243,7 @@ def configure(env: "SConsEnvironment"):
env.AppendUnique(CPPDEFINES=["METAL_ENABLED", "RD_ENABLED"])
extra_frameworks.add("Metal")
extra_frameworks.add("MetalKit")
extra_frameworks.add("MetalFX")
env.Prepend(CPPPATH=["#thirdparty/spirv-cross"])

if env["vulkan"]:
Expand Down
4 changes: 3 additions & 1 deletion scene/main/viewport.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4899,7 +4899,7 @@ void Viewport::_bind_methods() {

#ifndef _3D_DISABLED
ADD_GROUP("Scaling 3D", "");
ADD_PROPERTY(PropertyInfo(Variant::INT, "scaling_3d_mode", PROPERTY_HINT_ENUM, "Bilinear (Fastest),FSR 1.0 (Fast),FSR 2.2 (Slow)"), "set_scaling_3d_mode", "get_scaling_3d_mode");
ADD_PROPERTY(PropertyInfo(Variant::INT, "scaling_3d_mode", PROPERTY_HINT_ENUM, "Bilinear (Fastest),FSR 1.0 (Fast),FSR 2.2 (Slow),MetalFX (Spatial),MetalFX (Temporal)"), "set_scaling_3d_mode", "get_scaling_3d_mode");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it worth putting this behind a support check so platforms without metalfx won't show those options?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Per @clayjohn, these need to be included in the public APIs, so that they can be selected on a per-platform basis. Windows will have the option for a DirectX-specific scaler too. There may still be some details to work out here.

Copy link
Member

@Calinou Calinou Nov 25, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remember that different users working on the same project might be able to see those options, so if you couldn't, the scene would be saved with different values once you save it in the editor (since your inspector's enum would have fewer options available).

Therefore, the set of available properties in the inspector should remain the same across platforms.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I was thinking about removing this comment because definitely the simplest and least surprising thing would be to just list all options although FWIW setting this to an unsupported value causes it to fallback to not using scaling at runtime but it doesn't overwrite the project settings file even when other options are changed.

Maybe an option later on would be higher level cross platform options like Off, Bilinear, Spatial, Temporal with flags to allow those to defer to platform specific apis when available/appropriate, so if a project was using metal by default it would use fsr automatically rather than disabling scaling entirely on a non-mac. That might be too rigid for other scaling options to fit in, for example it looks like DirectSR does not make the distinction in the same way, but I imagine this will come up for other api specific additions in the future like frame gen.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is now done – we have specialisation options for ios and macos.

ADD_PROPERTY(PropertyInfo(Variant::FLOAT, "scaling_3d_scale", PROPERTY_HINT_RANGE, "0.25,2.0,0.01"), "set_scaling_3d_scale", "get_scaling_3d_scale");
ADD_PROPERTY(PropertyInfo(Variant::FLOAT, "texture_mipmap_bias", PROPERTY_HINT_RANGE, "-2,2,0.001"), "set_texture_mipmap_bias", "get_texture_mipmap_bias");
ADD_PROPERTY(PropertyInfo(Variant::INT, "anisotropic_filtering_level", PROPERTY_HINT_ENUM, String::utf8("Disabled (Fastest),2× (Faster),4× (Fast),8× (Average),16x (Slow)")), "set_anisotropic_filtering_level", "get_anisotropic_filtering_level");
Expand Down Expand Up @@ -4954,6 +4954,8 @@ void Viewport::_bind_methods() {
BIND_ENUM_CONSTANT(SCALING_3D_MODE_BILINEAR);
BIND_ENUM_CONSTANT(SCALING_3D_MODE_FSR);
BIND_ENUM_CONSTANT(SCALING_3D_MODE_FSR2);
BIND_ENUM_CONSTANT(SCALING_3D_MODE_METALFX_SPATIAL);
BIND_ENUM_CONSTANT(SCALING_3D_MODE_METALFX_TEMPORAL);
BIND_ENUM_CONSTANT(SCALING_3D_MODE_MAX);

BIND_ENUM_CONSTANT(MSAA_DISABLED);
Expand Down
2 changes: 2 additions & 0 deletions scene/main/viewport.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,8 @@ class Viewport : public Node {
SCALING_3D_MODE_BILINEAR,
SCALING_3D_MODE_FSR,
SCALING_3D_MODE_FSR2,
SCALING_3D_MODE_METALFX_SPATIAL,
SCALING_3D_MODE_METALFX_TEMPORAL,
SCALING_3D_MODE_MAX
};

Expand Down
2 changes: 2 additions & 0 deletions servers/rendering/renderer_rd/effects/SCsub
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,8 @@ env.servers_sources += thirdparty_obj
module_obj = []

env_effects.add_source_files(module_obj, "*.cpp")
if env["metal"]:
env_effects.add_source_files(module_obj, "metal_fx.mm")
env.servers_sources += module_obj

# Needed to force rebuilding the module files when the thirdparty library is updated.
Expand Down
2 changes: 1 addition & 1 deletion servers/rendering/renderer_rd/effects/fsr.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ FSR::~FSR() {
fsr_shader.version_free(shader_version);
}

void FSR::fsr_upscale(Ref<RenderSceneBuffersRD> p_render_buffers, RID p_source_rd_texture, RID p_destination_texture) {
void FSR::process(Ref<RenderSceneBuffersRD> p_render_buffers, RID p_source_rd_texture, RID p_destination_texture) {
UniformSetCacheRD *uniform_set_cache = UniformSetCacheRD::get_singleton();
ERR_FAIL_NULL(uniform_set_cache);
MaterialStorage *material_storage = MaterialStorage::get_singleton();
Expand Down
10 changes: 8 additions & 2 deletions servers/rendering/renderer_rd/effects/fsr.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,17 +31,23 @@
#ifndef FSR_RD_H
#define FSR_RD_H

#include "spatial_upscaler.h"

#include "../storage_rd/render_scene_buffers_rd.h"
#include "servers/rendering/renderer_rd/shaders/effects/fsr_upscale.glsl.gen.h"

namespace RendererRD {

class FSR {
class FSR : public SpatialUpscaler {
String name = "FSR 1.0 Upscale";

public:
FSR();
~FSR();

void fsr_upscale(Ref<RenderSceneBuffersRD> p_render_buffers, RID p_source_rd_texture, RID p_destination_texture);
virtual String get_label() const final { return name; }
virtual void ensure_context(Ref<RenderSceneBuffersRD> p_render_buffers) final {}
virtual void process(Ref<RenderSceneBuffersRD> p_render_buffers, RID p_source_rd_texture, RID p_destination_texture) final;

private:
enum FSRUpscalePass {
Expand Down
Loading