Fix occlusion culling by using depth instead of Euclidean distance when selecting the closest point

[Rudolph-B]

Resolves #94210

The occlusion culler uses a point to compare with the depth mipmap to determine if an object is occluded or not. The occlusion culler currently chooses the point with the shortest Euclidean distance to the camera. While not entirely inaccurate, it can drift when compared to the actual closest point to the camera in terms of depth.

This inaccuracy increases when an object's AABB is skewed (relative to the camera perspective) and partially occluded by other objects, as seen in the MRP in #94210 and the 2D example below.

There were two possible solutions to this problem, and this PR implements the second solution.

Possible solutions:

1. Convert the entire occlusion culling algorithm to use Euclidean distance.
2. Select the closest point based on depth.

Although the first solution seems drastic, it mostly involved removing and simplifying code. It also had the benefit of being a bit more accurate (more likely to cull objects that can be culled) compared to solution 2. However, I eventually dropped this approach because it did not work well with an orthogonal camera, and fixing this began to drift out of the original issue's scope. A discussion on this might still be valuable, as it could lead to improved performance in most use cases (non-orthogonal cameras) compared to the second solution.

Implementing the second solution meant cycling through the corners of the object's AABB and determining the closest point. This approach is guaranteed to always provide an accurate depth when the object is fully within the view frustum. In cases where part of the object is outside the view frustum, it could provide closer values. Therefore, when it is inaccurate, it would still tend to favor drawing the object.

With the added accuracy of this fix, I think it would be worthwhile to retune the occlusion culling algorithm.

MRP used during testing provided by @unfa:
MRP.zip

[Rudolph-B]

I added a draft pull request for the first solution (#98257)

[lawnjelly]

I haven't debugged but I suspect this is on the right lines.

As I understand the previous approach was just using a single rather arbitrary sample which depended on the axis alignment of the bounding box to the camera position. With this approach each corner is taken into account.

There's also the possibility to do a quick reject of the minimum distance against z near before it checks the occlusion buffer.

The one thing I'm slightly concerned is this may end up checking bounding box corners that are outside the view frustum, which may be closer to the camera plane than necessary and result in too small a min_depth, and thus unnecessarily rendering objects that could be rejected. I'm not sure on this, would be nice to have confirmed. I.e. ideally you maybe would intersect the AABB with the view frustum to find the closest point of the AABB within view (or some equivalent technique).

I also suspect this whole approach will have tendency to occlusion cull correctly when a wall is head on to the camera, but at an angle, the min distance on e.g. the right, might be less that the occlusion buffer on the left (but the actual AABB distance there might still be behind the occlusion buffer). Probably needs a diagram to explain but maybe this is enough.

That is to say this would overly render, rather than over occlude, so wouldn't lead to graphical anomalies. But is likely to occlude less than it did before this PR (although before this PR it is broken, so saying it's faster doesn't help).

[Rudolph-B]

In my testing, this approach did not cull large objects that could have been culled because the corners of the AABB were close to the camera but still outside the view frustum. This issue is somewhat mitigated by the fix I implemented in #98257, but it could theoretically still occur.

I have a few ideas that could enhance the effectiveness and performance of occlusion culling, which I'm currently working on. I'll add intersecting the AABB with the view frustum to the list of improvements to consider.

I'm not entirely sure I fully understand your diagram. Is the idea that even though the red box is entirely occluded, it wouldn’t be culled because its min_depth is less than some parts of the blue wall's depth? If so, I agree, but addressing this would require a more significant change in approach.

[lawnjelly]

I'm not entirely sure I fully understand your diagram. Is the idea that even though the red box is entirely occluded, it wouldn’t be culled because its min_depth is less than some parts of the blue wall's depth?

Yup, that's it. However, the key point is that after the PR, now the min_depth is more accurate (in order to reduce false positives), this situation is more likely to occur. Agreed though, it is not necessarily simple to solve, it is just an observation, and false positives are more important to avoid than false negatives.

Notably, the approach in your other PR of using Euclidean distance in the occlusion buffer does not require a smaller min_dist, so is likely to result in less false negatives (probably similarish to the rate before).

[Rudolph-B]

Closing this since we are going with the other PR

[Flarkk]

I suggest reopening this PR as :

1. the edge case discussed above is in my opinion not worth the overhead introduced by moving the whole buffer to euclidian distance (see Store view depth instead of ray length in occlusion buffer #100907)
2. it ensures occlusion culling still works with very far away objects (see Enable rendering with unbounded far distance #99986 (comment))

[Rudolph-B]

@Flarkk I doubt it will still work in its current state. I'm away for the weekend, but I'll make the required adjustments and reopen soon.