Add a tutorial teaching about lighting in Drake

[SeanCurtis-TRI]

This includes a number of resource files (illustrations, models, environment
maps, etc.) In addition, it includes the source files from which these
where generated (where reasonable).

---

This change is 

[SeanCurtis-TRI]

+@rcory +@jwnimmer-tri for review

General notes:

• Revision 1 and 2 are separate PRs (changes to VTK) that the tutorial depends on. (They are Correct VTK's image-based lighting diffuse calculation #22629 and Preserve direct light specular reflections for smooth materials #22632, respectively). Feel free to ignore them.
• Feel free to tag anything that seems unclear, unhelpful, or any questions that you still have after attempting to read through the tutorial.

@jwnimmer-tri In addition to content, this is also a good time to talk about side files. Now that we have something concrete.

@rcory

If you haven't reviewed tutorials before, it's very easy. Simply check out the PR and then run:

bazel run //tutorials:configuring_lighting

Reviewable status: LGTM missing from assignees rcory,jwnimmer-tri(platform), commits need curation (https://drake.mit.edu/reviewable.html#curated-commits), missing label for release notes (waiting on @SeanCurtis-TRI)

[jwnimmer-tri]

FYI @rcory I'm waiting to let you take a first pass over this, before I do my review.

[jwnimmer-tri]

I guess Rick is pretty busy. I'll take a look at this tomorrow.

[rcory]

Sorry, we were pushing on a plan for data-collect (completed yesterday). I am planning on spending time on this today.

Reviewable status: 4 unresolved discussions, LGTM missing from assignees jwnimmer-tri(platform),rcory, missing label for release notes (waiting on @SeanCurtis-TRI)

---

tutorials/configuring_lighting.ipynb line 701 at r3 (raw file):

    "    directed cone limits the directions that rays are emitted. The cone has its\n",
    "    apex at the light `position` and the cone is rotated so its central axis\n",
    "    is aligned with the light's `direction` vector (such the cone's radius\n",

I know what you mean, but this wording threw me off at first. The illustration shows multiple vectors originating from the spot light's center, so when you say "its central axis is aligned with the light's direction", it might be clearer to to say "its central axis is aligned with the light's central direction vector".

---

tutorials/configuring_lighting.ipynb line 701 at r3 (raw file):

    "    directed cone limits the directions that rays are emitted. The cone has its\n",
    "    apex at the light `position` and the cone is rotated so its central axis\n",
    "    is aligned with the light's `direction` vector (such the cone's radius\n",

Suggestion:

such that the cone's radius

---

tutorials/configuring_lighting.ipynb line 706 at r3 (raw file):

    "    the cone's span, or the angle between the cone's central axis and the\n",
    "    boundary of the cone.\n",
    "  - __Directional light__: This can be thought of a point light that is infinitely\n",

Suggestion:

be thought of as a point

---

tutorials/configuring_lighting.ipynb line 717 at r3 (raw file):

    "\n",
    "Light interacts with matter in various ways. While the physics is fascinating\n",
    "and intricate, and rendering technology typically focuses on the _effects_ and\n",

Suggestion:

, rendering technology

[rcory]

Reviewable status: 30 unresolved discussions, LGTM missing from assignees jwnimmer-tri(platform),rcory, missing label for release notes (waiting on @SeanCurtis-TRI)

---

tutorials/configuring_lighting.ipynb line 48 at r4 (raw file):

    "physics (in comparison with older, classical illumination models).\n",
    "While it is truer to reality, it is still an approximation. There\n",
    "will inevitably be differences between renderings and reality.\n",

BTW I believe there might also be performance trade-offs (i.e., PBR would be slower)?

---

tutorials/configuring_lighting.ipynb line 52 at r4 (raw file):

    "### What makes it physical?\n",
    "\n",
    "PBR has three driving principles:\n",

nit while I'm sure these principles are important, I'm not sure they add much value to this tutorial? Should I care about these details, or would pointing to a Wikipedia article on PBR (for those interesting in learning more) be sufficient?

Edit: I see that you've linked to the Filament PBR documentation below - which is probably sufficient.

---

tutorials/configuring_lighting.ipynb line 61 at r4 (raw file):

    "      surfaces.\n",
    "\n",
    "PBR provides concrete benefits:\n",

FYI These points are easy to digest and tell me why I should care about PBR.

---

tutorials/configuring_lighting.ipynb line 93 at r4 (raw file):

    "    - Set the `force_to_pbr` flag to `true`.\n",
    "  - Add an [environment map](#environment-maps).\n",
    "  - Add a glTF file (Drake's preferred visual geometry representation)."

BTW to clarify, do you mean so long as a single glTF file is referenced in a model, that is sufficient to trigger PBR?

Also, what if someone uses a single glTF as a (Convex) collision geometry, but not otherwise referenced in any illustration roles? I presume PBR would only trigger if illustration (or perception roles?) use glTF?

---

tutorials/configuring_lighting.ipynb line 584 at r4 (raw file):

    "\n",
    "For subsequent renderings, we may not do all of these (for example, several\n",
    "renderings in a can use the same geometry)."

Suggestion:

can use the same geometry

---

tutorials/configuring_lighting.ipynb line 637 at r4 (raw file):

    "**ATTENTION**: Expect the rendered image to deviate from what is shown in\n",
    "the meshcat browser. Meshcat introduces an independent light set from those\n",
    "defined in the `RenderEngine`. Meshcat also uses a PBR illumination model\n",

Suggestion:

Meshcat also uses a PBR illumination model,

---

tutorials/configuring_lighting.ipynb line 848 at r4 (raw file):

    "the full ground box. We see a circular pool of light. The light is circular\n",
    "because the spot light's cone of light intersects the ground perpendicularly.\n",
    "If the spot light were tilted, it the pool of light would be elliptical.\n",

Suggestion:

the pool of light

---

tutorials/configuring_lighting.ipynb line 853 at r4 (raw file):

    "A more subtle difference is between the point light and directional light\n",
    "images. In both images, the ball's appearance is essentially the same. However,\n",
    "the ground box is different. With the point light, the ground direclty under\n",

Suggestion:

directly

---

tutorials/configuring_lighting.ipynb line 853 at r4 (raw file):

    "A more subtle difference is between the point light and directional light\n",
    "images. In both images, the ball's appearance is essentially the same. However,\n",
    "the ground box is different. With the point light, the ground direclty under\n",

nit I suppose it's really the ground proximal to the ball? (Directly under would be a dark shadow?)

Code quote:

the ground direclty under

---

tutorials/configuring_lighting.ipynb line 858 at r4 (raw file):

    "with the directional light. It's also worth pointing out that _within_ the\n",
    "pool of light, the spot light's pattern of illumination matches the point\n",
    "light. Where does this difference come from?\n",

nit This wording is confusing. It seems to point out that the spot and point lights match (within the spot light's pool of light), but then asks "where does this difference come from?". What difference are you referring to? Maybe you are referring to the difference between the point light and directional light, which you forgot to mention here?

Code quote:

    "pool of light, the spot light's pattern of illumination matches the point\n",
    "light. Where does this difference come from?\n",

---

tutorials/configuring_lighting.ipynb line 879 at r4 (raw file):

    "So, how does that apply to the images above?\n",
    "\n",
    "[TODO: illustration]\n",

FYI Not sure if this TODO was left intentionally to be visible in the tutorial? Same for others below.

---

tutorials/configuring_lighting.ipynb line 976 at r4 (raw file):

    "    \n",
    "The inverse-squared law is satisified by c = 0, l = 0, q = 1. In practice,\n",
    "it sometimes becomes convenient to introduce have the intensity decay\n",

Suggestion:

convenient to have the intensity decay

---

tutorials/configuring_lighting.ipynb line 977 at r4 (raw file):

    "The inverse-squared law is satisified by c = 0, l = 0, q = 1. In practice,\n",
    "it sometimes becomes convenient to introduce have the intensity decay\n",
    "more slowly by reintroducing constant or linear factors. The three\n",

Suggestion:

The sum of the three

---

tutorials/configuring_lighting.ipynb line 980 at r4 (raw file):

    "coefficients should total one.\n",
    "\n",
    "**Note**: attenuation does not apply to directional lights as there is\n",

Suggestion:

**Note**: Attenuation

---

tutorials/configuring_lighting.ipynb line 1000 at r4 (raw file):

    "\n",
    "comparator.set_render_engines([\n",
    "    RenderEngineSpec(\"Constant\", make_config(attenuation=[1, 0, 0])),\n",

nit Might be worth mentioning that zero attenuation causes full saturation (which is perhaps the same as having infinite intensity?).

---

tutorials/configuring_lighting.ipynb line 1051 at r4 (raw file):

   "outputs": [],
   "source": [
    "# Add several arbitrary light; each with an arbitrarily different color do better\n",

Suggestion:

lights

---

tutorials/configuring_lighting.ipynb line 1051 at r4 (raw file):

   "outputs": [],
   "source": [
    "# Add several arbitrary light; each with an arbitrarily different color do better\n",

Suggestion:

different color to better

---

tutorials/configuring_lighting.ipynb line 1124 at r4 (raw file):

    "surfaces that would otherwise have been lit. In rendering, we don't get\n",
    "occlusion for free. We create shadows through a variety of approximations. The\n",
    "spot light's light cone is our first example. In the real world, a spot light's\n",

Suggestion:

spot light's cone

---

tutorials/configuring_lighting.ipynb line 1212 at r4 (raw file):

    "### What lights can cast shadows?\n",
    "\n",
    "Spot lights and directional lights can cast shadows. Point lights cannot.\n",

But why not?? I found myself wondering why the shadow itself for point and spot lights (in this example) wouldn't end up being the same?

Edit: I see that this finally gets answered in the "Spot light limitations" section. Might be worth referring the reader to that section here.

Code quote:

Spot lights and directional lights can cast shadows. Point lights cannot

---

tutorials/configuring_lighting.ipynb line 1285 at r4 (raw file):

    "\n",
    "- The shadow cast from the directional light is smaller than that of the spot\n",
    "  light Not only that, it even appears smaller than the ball.\n",

Suggestion:

light. Not only that

---

tutorials/configuring_lighting.ipynb line 1286 at r4 (raw file):

    "- The shadow cast from the directional light is smaller than that of the spot\n",
    "  light Not only that, it even appears smaller than the ball.\n",
    "   - The directional shadow *is* definitely smaller than the point light shadow.\n",

Suggestion:

smaller than the spot light shadow

---

tutorials/configuring_lighting.ipynb line 1638 at r4 (raw file):

    "things closer. When the ball is closer to the camera than its shadow, it will\n",
    "appear larger. When the shadow is closer to the camera than the ball, the\n",
    "shadow will appear bigger. THe important thing to note is that these shadows are\n",

Suggestion:

The important thing

---

tutorials/configuring_lighting.ipynb line 1657 at r4 (raw file):

    "light properties doesn't have the effect you expect.\n",
    "\n",
    "Again, this discussion is going bias heavily towards intuition and\n",

Suggestion:

this discussion is going to bias heavily

---

tutorials/configuring_lighting.ipynb line 1683 at r4 (raw file):

    "direction _l_. The three views _v₀_, _v₁_, and _v₂_ form half-angle\n",
    "vectors _h₀_, _h₁_, and _h₂_, respectively. Only _h₁_ aligns with\n",
    "_n_, so only view _v₀_ receives the specular reflection from the\n",

Suggestion:

so only view _v₁_ receives the specular reflection

---

tutorials/configuring_lighting.ipynb line 1686 at r4 (raw file):

    "light source.\n",
    "\n",
    "In the real world, surface reflectivity can vary widely some crisp\n",

Suggestion:

can vary widely; some crisp

---

tutorials/configuring_lighting.ipynb line 1813 at r4 (raw file):

    "surface (its normal direction), _and_ the direction from the surface to\n",
    "the camera.\n",
    "4. When like strikes a surface at a \"grazing\" angle, the surface exhbits\n",

Suggestion:

When light strikes a surface

---

tutorials/configuring_lighting.ipynb line 1863 at r4 (raw file):

    "reflections, the effect is typically more noticable on dielectric\n",
    "materials, because their _base_ reflectivity is generally much lower\n",
    "than of metals.\n",

Suggestion:

than that of metals

---

tutorials/configuring_lighting.ipynb line 1951 at r4 (raw file):

    "across the two spheres. The metallic sphere's \"hot spot\" is\n",
    "larger and somewhat oversaturated in the center (we'll deal\n",
    "with the oversaturatin when we discuss [exposure](#exposure)\n",

Suggestion:

oversaturation

[rcory]

Reviewable status: 38 unresolved discussions, LGTM missing from assignees jwnimmer-tri(platform),rcory, missing label for release notes (waiting on @SeanCurtis-TRI)

---

tutorials/configuring_lighting.ipynb line 1987 at r4 (raw file):

    "We'll start with a simple environment map and a mirror ball.\n",
    "\n",
    "The image below shows a ball in a simple environment. The mirror ball sites on\n",

Suggestion:

The mirror ball sits on

---

tutorials/configuring_lighting.ipynb line 1992 at r4 (raw file):

    "\n",
    "![environment_map_scene](images/env_color_room_in_situ.png)\n",
    "**Figure:** the physical environment.\n",

Suggestion:

The physical environment

---

tutorials/configuring_lighting.ipynb line 2007 at r4 (raw file):

    "\n",
    "![](resources/env_color_room.png)\n",
    "**Figure:** the environment map drawn from the physical scene shown above."

Suggestion:

The environment map drawn

---

tutorials/configuring_lighting.ipynb line 2224 at r4 (raw file):

    "such specular reflection.\n",
    "\n",
    "But this image illustrates to additional phenomena: Fresnel\n",

Suggestion:

 this image illustrates two additional phenomen

---

tutorials/configuring_lighting.ipynb line 2235 at r4 (raw file):

    "so we observe a more intense reflection.\n",
    "\n",
    "Both spheres have a reddish tint on that covers a large portion\n",

Suggestion:

 have a reddish tint that covers

---

tutorials/configuring_lighting.ipynb line 2241 at r4 (raw file):

    "map's contribution to the diffuse lighting effects.\n",
    "\n",
    "Remember, diffuse illumination when light strikes a surface,\n",

nit The wording in this sentence seems odd.

---

tutorials/configuring_lighting.ipynb line 2248 at r4 (raw file):

    "it must be included in the diffuse calculation.\n",
    "\n",
    "In this case, for any givn point on the sphere, fully half of\n",

Suggestion:

for any given point

---

tutorials/configuring_lighting.ipynb line 2248 at r4 (raw file):

    "it must be included in the diffuse calculation.\n",
    "\n",
    "In this case, for any givn point on the sphere, fully half of\n",

nit not quite sure what "fully half" means.

[rcory]

Reviewable status: 46 unresolved discussions, LGTM missing from assignees jwnimmer-tri(platform),rcory, missing label for release notes (waiting on @SeanCurtis-TRI)

---

a discussion (no related file):
FYI when I start the tutorial, all cells in the notebook show up as already evaluated (and all renderings already shown without running any code). I'm assuming this isn't intentional (since you want the reader to start fresh).

I can easily clear things by going to the top menu and selecting Cell>All Output>Clear and saving the file.

---

tutorials/configuring_lighting.ipynb line 1117 at r4 (raw file):

   "source": [
    "<a id=\"shadows\"></a>\n",
    "## Shadows\n",

BTW in the discussion of shadows below, the topic of shadow edge sharpness was never discussed. I don't believe this is yet supported (or exposed?) in Drake, but probably worth mentioning at some point.

---

tutorials/configuring_lighting.ipynb line 2244 at r4 (raw file):

    "some portion gets refracted _into_ the material, bounces around\n",
    "and, finally, some remaining portion is emitted again. The\n",
    "_total_ diffuse affect is the sum of all such light. If\n",

Suggestion:

effect

---

tutorials/configuring_lighting.ipynb line 2314 at r4 (raw file):

    "    intensity light sources manifesting as dark colors to film and sensors. As\n",
    "    counter-intuitive as it might seem, a pixel with normalized color\n",
    "    (0.01, 0.01, 0.01) image is putting out 1 W of light, just like a white\n",

Suggestion:

(0.01, 0.01, 0.01) is putting

---

tutorials/configuring_lighting.ipynb line 2335 at r4 (raw file):

    "\n",
    "  - PNG: a normalized image with brightness of 1 W at each pixel.\n",
    "  - 7.5 HDR: an HDR image with maximum brightness of 7.5 W.\n",

Suggestion:

brightness of 7.5 W at each pixel

---

tutorials/configuring_lighting.ipynb line 2336 at r4 (raw file):

    "  - PNG: a normalized image with brightness of 1 W at each pixel.\n",
    "  - 7.5 HDR: an HDR image with maximum brightness of 7.5 W.\n",
    "  - 120 HDR: an HDR image with maximum brightness of 120 W."

Suggestion:

brightness of 120 W at each pixel

---

tutorials/configuring_lighting.ipynb line 2423 at r4 (raw file):

   "metadata": {},
   "source": [
    "Although the three images use environment maps with the exact same resolution\n",

FYI The 1-watt image has a light grey background, and the 7.5-w and 120-w images have a black background. Is this expected?

Edit: I think this is addressed in the exposure section below, but probably worth acknowledging this difference here, and defer the explanation to the following section.

---

tutorials/configuring_lighting.ipynb line 2455 at r4 (raw file):

    "energy as the image rendered with the 120-watt environment map. And yet, the\n",
    "\"dim\" image is not simply a copy of the bright image with each pixel having\n",
    "1/120th the value of the original. The dimmer image features smooth contours\n",

BTW Maybe it's the size of the image (unfortunate that I can zoom in), but it's difficult to see the contours that are referring to in the dim image.

Code quote:

The dimmer image features smooth contours

---

tutorials/configuring_lighting.ipynb line 2575 at r4 (raw file):

    "background intensity is functionally black and renders as such.\n",
    "\n",
    "Judicious use of exposure is critical in producing images that are both\n",

Is it worth mentioning that tuning exposure might still be insufficient to capture the desired quality (e.g., this issue).

---

tutorials/configuring_lighting.ipynb line 2590 at r4 (raw file):

    "\n",
    "Open source environment maps can be downloaded from repositories like\n",
    "[Polyhaven](https://polyhaven.com/).\n",

BTW As far as I could tell, Polyhaven .hdr environment maps don't seem to come in the various energy flavors (i.e., 1-watt, 7.5-watt, etc.). Is there a reason for that?

[rcory]

First pass done.

Reviewable status: 46 unresolved discussions, LGTM missing from assignees jwnimmer-tri(platform),rcory, missing label for release notes (waiting on @SeanCurtis-TRI)

[SeanCurtis-TRI]

Big question for you before I get to the small details:

Were there any big picture comments?

Missing topics?
Questions that you still don't have answered? Questions you didn't have before the tutorial that you have now? ETc.

Reviewable status: 46 unresolved discussions, LGTM missing from assignees jwnimmer-tri(platform),rcory, missing label for release notes

---

a discussion (no related file):

Previously, rcory (Rick Cory) wrote…

FYI when I start the tutorial, all cells in the notebook show up as already evaluated (and all renderings already shown without running any code). I'm assuming this isn't intentional (since you want the reader to start fresh).

I can easily clear things by going to the top menu and selecting Cell>All Output>Clear and saving the file.

Actually, that's exactly what I wanted.