Draft of HTC passthrough (XR_HTC_passthrough) support with OpenXR

[seichter]

initial stab on supporting devices like HTC XR Elite with pass-through.

However, this raises design questions in openxr_interface.cpp in regards to how to share or differentiate Meta passthrough from HTC (and later potential Apple). At the moment is_passthrough_supported, is_passthrough_enabled, start_passthrough, stop_passthrough are using an internal instance to OpenXRFbPassthroughExtensionWrapper

HTC passthrough is rather simple but supports controlled alpha blending.

Reference: https://registry.khronos.org/OpenXR/specs/1.0/html/xrspec.html#XR_HTC_passthrough

Open to comments @BastiaanOlij

[dsnopek]

If this is specific to HTC, it may be more appropriate in https://github.com/GodotVR/godot_openxr_vendors

[BastiaanOlij]

OMG, with clearly defined pass through approaches, why do vendors keep doing their own thing... sigh
HTC has been off my radar because I lack hardware to test and develop with sadly.

Most AR devices (e.g. Hololens, ML2, Lynx R1) just assume the images we provide has alpha applied and the primary layer has its blend mode set to Alpha. The Lynx is the exceptional case here because it actually uses cameras, but it will just render the camera underneath to simulate AR. For Hololens and ML2 it doesn't really matter due to the way the image is projected.
I suspect Apple, if it ever gets OpenXR support (highly doubt it) will go this way too.
As far as I'm aware this is the "official" way to do passthrough in OpenXR.

Meta went a lot further creating a camera layer so there is more control over where the camera image is shown, without exposing that camera image to the user for privacy reasons. Pico basically adopted this extension for their pass through too.

It seems HTC decided to create a competing extension. Fun.

So extension wise your PR looks fine, it's now indeed a matter of tying it into the system. Godot supports the "ML2/Lynx R1/Standard OpenXR" approach out of the box, just seeing it as AR. It off course isn't passthrough for real AR devices and the Lynx R1 just emulates it as an AR first device.

So when we talk about passthrough, we're talking about the exceptional case where we need to enable a mode where the camera image is used. The design here was to try and keep it agnostic to the actual platform. So XRInterface just has an is_passthrough_supported, is_passthrough_enabled, start_passthrough and stop_passthrough method that needs to be implemented on the actual interface for the given platform.

For OpenXR that means we check the various implementations, it is unlikely that both FB and HTC wrappers are available and enabled.
So in OpenXRInterface the current passthrough_wrapper needs to be renamed to fb_passthrough_wrapper and we add a htc_passthrough_wrapper.

Then is_passthrough_supported changes to something like:

bool OpenXRInterface::is_passthrough_supported() {
	if ((fb_passthrough_wrapper != nullptr) && (fb_passthrough_wrapper->is_passthrough_supported())) {
		return true;
	} else if ((htc_passthrough_wrapper != nullptr) && (htc_passthrough_wrapper->is_passthrough_supported())) {
		return true;
	} else {
		return false;
	}
}

And similar changes need to happen to is_passthrough_enabled, start_passthrough and stop_passthrough, we first check which extension is enabled, and then use that extension to setup passthrough.

[BastiaanOlij]

If this is specific to HTC, it may be more appropriate in https://github.com/GodotVR/godot_openxr_vendors

In this case we decided long ago to handle the vendor specific changes inside of Godot, though I really hope one day soon it can all be replaced by a Khronos extension.

[seichter]

@BastiaanOlij the situation with is_passthrough_enabled, start_passthrough and stop_passthrough is indeed complicated design-wise. Because once the passthrough is requested with xrCreatePassthroughHTC (btw. there is also another flag triggering a mode similar to Meta devices with a custom mesh, which the current OpenXR backend doesn't allow, hence I ignored this; would allow for something like Augmented Virtuality) ... anyways. Once passthrough is requested, it is on and the alpha member in the color part of the struct controls how much comes through. Hence, the question how to fiddle this into the current API design? Make start_passthrough set it to 1.0f and stop_passthrough to 0.0f? It would be hacky but at least we would hide the underlying differences. Or we make the Meta passthrough react on a threshold of 0.5f of a alpha member and support the blending mode in the HTC devices.
I will try to extend my PR with something to tying it in as it was currently (binary on/off).

[m4gr3d]

If this is specific to HTC, it may be more appropriate in https://github.com/GodotVR/godot_openxr_vendors

In this case we decided long ago to handle the vendor specific changes inside of Godot, though I really hope one day soon it can all be replaced by a Khronos extension.

@BastiaanOlij We'll need to review this approach.. By that same logic, all the current Meta vendor extensions that are being added to the vendors plugin should be added directly inside of Godot as well.

[BastiaanOlij]

@BastiaanOlij We'll need to review this approach.. By that same logic, all the current Meta vendor extensions that are being added to the vendors plugin should be added directly inside of Godot as well.

You misunderstood me here, the passthrough logic is the exception to the rule because we decided to do it this way, a long time ago, mostly because when we did this we started with the VrAPI implementation, and then when we switched to OpenXR we only supported Meta on Android.

It's actually the use case that shows the problem with this approach, where we've exposed a passthrough mode to the end user, that only works for one vendor while for other vendors different approaches have to be taken. This means that any vendor that does not adopt the official way, nor Metas way, suddenly breaks even though we give the illusion that we support passthrough in the core. This is the danger of accepting Vendor specific logic in the core.

The "official" OpenXR way, as I understand it, is using the blend mode on the image being submitted, which we have exposed as the environment blend mode.
So:

• XR_ENVIRONMENT_BLEND_MODE_OPAQUE/XR_ENV_BLEND_MODE_OPAQUE means we don't show the background, we are in full VR mode
• XR_ENVIRONMENT_BLEND_MODE_ADDITIVE/XR_ENV_BLEND_MODE_ADDITIVE means the image is overlayed on top of the background, many true AR devices use this (and only support this mode because the way the optics work leave no choice).
• XR_ENVIRONMENT_BLEND_MODE_ALPHA_BLEND/XR_ENV_BLEND_MODE_ALPHA_BLEND means the image is overlayed on top of the background but using the alpha channel to mix, this mode is intended to trigger passthrough on camera based devices, but is also used by true AR devices (such as Magic Leap) that are able to black out the background.

So the "proper" way forward is to put pressure on Meta, Pico and HTC to support the above, and only use their extensions to unlock device specific passthrough features, and then remove the passthrough logic from the core and purely offer the blend mode solution in core.

But I think that ship sailed a long time ago and we're stuck with the choice we made here, might as well keep the exception and follow through with it.

[dsnopek]

So the "proper" way forward is to put pressure on Meta, Pico and HTC to support the above, and only use their extensions to unlock device specific passthrough features, and then remove the passthrough logic from the core and purely offer the blend mode solution in core.

But I think that ship sailed a long time ago and we're stuck with the choice we made here, might as well keep the exception and follow through with it.

So, it sounds like the plan is (1) keep vendor extensions for passthrough in Godot itself and (2) wait until all vendors to support the official OpenXR way so we can remove the vendor extensions from Godot.

I think this plan is OK, but I worry that if we keep adding new vendor extensions for passthrough, it could make nr 2 take much longer to happen (or maybe never happen if one of the vendors never adopts the official OpenXR way).

It may be worth considering coming up with a way to add new vendor passthrough extensions in the 'godot_openxr_vendors' repo, so that nr 2 only involves waiting for the vendors that we've already made exceptions for, rather than making the list of vendors longer and longer.

[BastiaanOlij]

@dsnopek I'd be for moving all the logic into vendors but the problem we have is that we'll likely end up breaking backwards compatibility. This is why I have an position that we should try and do the right thing for new APIs, but where we're dealing with older choices we accept the status quo.

It's really hard to judge whether other vendors will go down this path. The introduction of the passthrough APIs have muddied the water a lot because it's putting vendors now tackling this on the wrong foot.
I also have some personal opinions here I won't divulge in public but you're free to ask me on rocket chat :P

[dsnopek]

I'd be for moving all the logic into vendors but the problem we have is that we'll likely end up breaking backwards compatibility.

To be clear, I wasn't suggesting moving everything into 'godot_openxr_vendors' right away. The idea was to make it possible to implement passthrough extension in 'godot_openxr_vendors', but keep the current Meta extension for this in Godot. Then we can add HTC or any other future passthrough-related vendor extensions into 'godot_openxr_vendors', and we only have to wait for Meta (just one vendor) to switch to the official OpenXR way in order to remove its extension from Godot (rather than having to wait for Meta, HTC and any others that may get added later).

This would hopefully allow us to transition without breaking backwards compatibility.

Anyway, probably best to discuss in a higher bandwidth format, maybe at the next XR meeting?

[m4gr3d]

I'd be for moving all the logic into vendors but the problem we have is that we'll likely end up breaking backwards compatibility.

To be clear, I wasn't suggesting moving everything into 'godot_openxr_vendors' right away. The idea was to make it possible to implement passthrough extension in 'godot_openxr_vendors', but keep the current Meta extension for this in Godot. Then we can add HTC or any other future passthrough-related vendor extensions into 'godot_openxr_vendors', and we only have to wait for Meta (just one vendor) to switch to the official OpenXR way in order to remove its extension from Godot (rather than having to wait for Meta, HTC and any others that may get added later).

This would hopefully allow us to transition without breaking backwards compatibility.

Anyway, probably best to discuss in a higher bandwidth format, maybe at the next XR meeting?

I'm also inclined toward that approach. I think even with regard to the existing Meta passthrough functionality, we can leave it as is to avoid breaking backward compatibility, but add a logging message (and update the documentation accordingly) to recommend users to instead use the vendors plugin to access the full functionality.

As mentioned, let's follow up during the next XR meeting; I've added the PR to the list of PRs to review

[BastiaanOlij]

One of the options I was thinking off is to make this purely work the way OpenXR intended.
Eg:

• we deprecate the is_passthrough_enabled, is_passthrough_supported, start_passthrough and stop_passthrough methods in XRInterface
• we make it clear in the documentation that get_supported_environment_blend_modes returns the available modes and what they mean
• we make it clear in the documentation that set_environment_blend_mode should be used for passthrough/AR

Now in the vendor plugin, for any vendor that requires additional logic, we check what environment blend mode is set to and perform the additional logic.

We could add the ability to register a callback or emit a signal when set_environment_blend_mode changes the blend mode to make starting and stopping passthrough easier for those runtimes that require that.

To keep backwards compatibility we can:

• make sure that is_passthrough_enabled and is_passthrough_supported return true if XR_ENV_BLEND_MODE_ALPHA_BLEND is returned by get_supported_environment_blend_modes
• have start_passthrough set our environment blend mode to XR_ENV_BLEND_MODE_ALPHA_BLEND
• have stop_passthrough set our environment blend mode to XR_ENV_BLEND_MODE_OPAQUE

Also, kind of on a tangent but this has been a long outstanding 'wish' on my part. The project setting for environment blend modes should be an order. On AR only devices the only blend mode supported is XR_ENV_BLEND_MODE_ADDITIVE, which gives an error with our default XR_ENV_BLEND_MODE_OPAQUE value.

This way you can correctly set whether you want a game to start in AR/Passthrough mode and only fallback on opaque if you're on hardware not capable to do AR, or you can default to opaque but fall back to additive for VR games.

[dsnopek]

@BastiaanOlij That sounds like a decent way to do it!

[dsnopek]

Now that Meta's passthrough extension has moved to the godot_openxr_vendors repo via PR #87630 and GodotVR/godot_openxr_vendors#79, this PR should probably be redone for godot_openxr_vendors as well. I imagine that most of the code will be able to be reused.

[seichter]

Now that Meta's passthrough extension has moved to the godot_openxr_vendors repo via PR #87630 and GodotVR/godot_openxr_vendors#79, this PR should probably be redone for godot_openxr_vendors as well. I imagine that most of the code will be able to be reused.

Unfortunately, neither me or any of my students will have time to work on it. If somebody wants to reuse this PR please have a look at my repo.

[BastiaanOlij]

@seichter I finally found time to convert this over, see GodotVR/godot_openxr_vendors#151

Thanks for the work you did here, made it much easier to get this up and running!