feat(ngff): Support displaying "labels" for "multiscales" nodes

[manzt]

Towards #45

demo: https://deploy-preview-242--vizarr.netlify.app/?source=https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.4/idr0052A/5514375.zarr

Making progress—still a work in progress, but the initial plumbing is in place. That will need a refactor later alongside broader state management (probably at some latter point). However, things are showing up.

Key files:

• src/layers/LabelLayer.ts – Custom deck.gl layer for rendering labels.
• src/components/LayerController/Labels.tsx – UI controls for ImageLabelLayer, including the opacity slider.

With this setup, we can iterate on rendering and UI controls. This PR is in a good spot to move forward.

Screen.Recording.2025-02-25.at.23.31.28.mov

@will-moore @jluethi @joshmoore

[will-moore]

@manzt That's looking great, thanks.

I also tried testing with a different IDR sample at https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.4/idr0052A/5514375.zarr.
This one actually has 2 sets of labels at https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.4/idr0052A/5514375.zarr/labels/Cell/ and https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.4/idr0052A/5514375.zarr/labels/Chromosomes/ so this would need to handle a list of labels layers rather than just a single layer.
It would also be cool if you could start at the image itself and find all the child labels, rather than needing to start at the labels themselves.
But I think I should be able to work on those changes (at some point) since they hopefully shouldn't be too tricky... Unless someone else would like to have a go?

[manzt]

It would also be cool if you could start at the image itself and find all the child labels, rather than needing to start at the labels themselves.

Agree this would be ideal. From an image root, where can one find the paths to the associated labels? I was only about to find paths in the one metadata pointing from labels up.

[will-moore]

You just have to check for the existence of /labels/.zattrs. That's what's happening at https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.4/idr0052A/5514375.zarr to show the Cell & Chromosomes links.
So, we do that check for every image and just ignore if /labels/.zattrs is not found.

[manzt]

Ok, I updated the PR to handle this case. Now if you provide a labels node, it will just open as a regular multiscale array. But we will look for /labels/.zattrs.

[will-moore]

@manzt Nice - that's loading both labels now 👍 .

I think there's an issue with the T-slider not updating labels? When I scroll through T I don't see any change in labels (but they do update when I scroll through Z).

I tried to check that the labels really do update through T by opening the label image itself...
But this gives Cannot read properties of null (reading '0') at

https://deploy-preview-242--vizarr.netlify.app/?source=https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.4/idr0052A/5514375.zarr/labels/Chromosomes

This works in vanilla vizarr: https://hms-dbmi.github.io/vizarr/?source=https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.4/idr0052A/5514375.zarr/labels/Chromosomes

[manzt]

I think there's an issue with the T-slider not updating labels? When I scroll through T I don't see any change in labels (but they do update when I scroll through Z).

Not totally sure. I can have a look.

[jluethi]

This is awesome to see @manzt ! ❤️

cc @tcompa @zonia3000 Do you have capacity to help here @zonia3000 ? Would be great to try this on our typical test data (e.g. this one with labels in both 2d & 3d images: https://zenodo.org/records/13305316). And then to see if we can help with some of the things mentioned as TODOs in this PR :)

[manzt]

Sounds good! First thing would be to see if you can get your own images (with labels) displayed with the current state of the PR. Then, with something visible we could try to iron out exactly what we need to prioritize to make it usable/useful, working backwards what code to write. For example, I know that we will at least need to fine a way to make ImageLabelLayer have a transparent background.

Some other ideas:

• label opacity
• label color(s)

We can then focus on making the rendering (ImageLabelLayer) parameterized by these properties, as well as make UI for adjusting them.

[tcompa]

Hi there, thanks a lot. As a first comment, I tried looking at one of the Fractal images - which is available at https://raw.githubusercontent.com/tcompa/hosting-ome-zarr-on-github/refs/heads/main/20200812-CardiomyocyteDifferentiation14-Cycle1_mip.zarr/B/03/0/

Sanity check 1: this image is viewed correctly in the standard vizarr viewer:

https://hms-dbmi.github.io/vizarr/?source=https://raw.githubusercontent.com/tcompa/hosting-ome-zarr-on-github/refs/heads/main/20200812-CardiomyocyteDifferentiation14-Cycle1_mip.zarr/B/03/0/

---

Sanity check 2: the image has a label array (under /labels/nuclei), which can be rendered by itself

https://deploy-preview-242--vizarr.netlify.app/?source=https://raw.githubusercontent.com/tcompa/hosting-ome-zarr-on-github/refs/heads/main/20200812-CardiomyocyteDifferentiation14-Cycle1_mip.zarr/B/03/0/labels/nuclei

Minor glitch in this page: I can add the same channel N times:

---

However something seems a bit off when I load the image+labels:

https://deploy-preview-242--vizarr.netlify.app/?source=https://raw.githubusercontent.com/tcompa/hosting-ome-zarr-on-github/refs/heads/main/20200812-CardiomyocyteDifferentiation14-Cycle1_mip.zarr/B/03/0/

The nuclei label is identified and showed in the top-left panel, but I cannot manage to get to see.
I also tried to disable the actual image channels:

---

My first guess for the explanation is that our label is a multiscale array.

The logs include a 404 response for the GET of https://raw.githubusercontent.com/tcompa/hosting-ome-zarr-on-github/refs/heads/main/20200812-CardiomyocyteDifferentiation14-Cycle1_mip.zarr/B/03/0/labels/nuclei/.zarray, but in fact there is no array at that path. There would be one at https://raw.githubusercontent.com/tcompa/hosting-ome-zarr-on-github/refs/heads/main/20200812-CardiomyocyteDifferentiation14-Cycle1_mip.zarr/B/03/0/labels/nuclei/0/.zarray instead (note the additional /0).

[will-moore]

In the last image + labels case, I'm seeing console errors IndexError: too many indicies for array; expected 3, got 4 which is likely due to the fact that the image has shape 3,1,2160,5120 whereas the labels have shape 1,540,1280.

[jluethi]

In the last image + labels case, I'm seeing console errors IndexError: too many indicies for array; expected 3, got 4 which is likely due to the fact that the image has shape 3,1,2160,5120 whereas the labels have shape 1,540,1280

Yes, these are the typical dimensions we'd have. Context:

1. The x & y of the label are sometimes (like in this example) lower resolution than the image it is based on
2. We don't include a channel axis c in the label image, as our labels don't (/can't) have channels. This is defined in the multiscales metadata of the label image

My questions from this:
a) are we reyling on the multiscales of the image to load the labels?
b) Do you typically save labels with the same axis, including a c axis @will-moore ?

[manzt]

we can support labels of different dimensions with the caveat that the axis labels MUST be a subset of the source image. Otherwise there is no way to align the selections across layers.

[will-moore]

@jluethi No, labels shouldn't need to have a C axis, although I see in the examples above we do have c:1 dimension.

[jluethi]

we can support labels of different dimensions with the caveat that the axis labels MUST be a subset of the source image. Otherwise there is no way to align the selections across layers.

That makes sense! And yes, I would agree that the axis of labels MUST be a subset. In our case, it's often czyx images & zyx labels.

There is the weird edge-case of arrays that have singleton dimensions. I think the limitation is fair that labels cannot introduce new singletons (e.g. a label shouldn't be saved as tzyx with singleton t dimension if the input image was czyx).

[manzt]

Great, thanks everyone! I forgot about the extensions feature and will give it a try. I think I'm also getting an error now due to a bug in Viv and latest deck.gl (#243). I'm going to revert those changes on this branch so we can keep iterating and not have it as a blocker.

[will-moore]

Wow, really nice work. Thanks @manzt! They're looking great!

[manzt]

I'm not sure when I'll be able to push the additional UI changes through. However, I think in this state I'd vote to try to get this merged (maybe behind an ?experimental flag in the URL for now, or ?labels=true).

Then, we have separate, smaller PRs to clean up the UI for layer properties (opacity, on/off), change color palette if necessary, etc. Those will be easier to review and hopefully contribute.

The main outstanding issue IMO is that merging "as is" changes the default behavior of vizarr for older URLs, so I'm trying to think of how we could keep the previous behavior and maybe op-in to showing the labels? Does that make sense?

We could maybe show that the image has labels in the UI, but you need to toggle them on first?

[jluethi]

I just grabbed a random palette from Color Brewer, definitely not set on it. Happy to discuss other options, like the palette used in napari (I think it comes from scikit-image). However, I'd mostly like to choose a default and avoid exposing some configuration (outside of the NGFF spec).

Fully agreed on having one default colormap and not exposing this further for this effort here! I'd have a strong preference for the "large palette" one, because the smaller one repeats so often that neighboring labels with unique values often get the same color assigned in the example here.
Both the small & large shown above look much nicer in the saturation! :)

I'm not sure when I'll be able to push the additional UI changes through. However, I think in this state I'd vote to try to get this merged (maybe behind an ?experimental flag in the URL for now, or ?labels=true).

That would work for us!

We could maybe show that the image has labels in the UI, but you need to toggle them on first?

That would be even better! :) We can either directly start with this or start with the experimental flag and then add this. I think having them toggled off by default is fine and one can then activate them as wanted.

Then, we have separate, smaller PRs to clean up the UI for layer properties (opacity, on/off), change color palette if necessary, etc. Those will be easier to review and hopefully contribute.

We'd be happy to make contributions for this! :)

[manzt]

I'd have a strong preference for the "large palette" one

Alright, let's just go with that one for now and we can reassess in the future if/when something comes up.

That would be even better! :)

Ok, I'll push this PR to get some indicator of labels ("off" by default), which can be enabled (as a group) with opacity sliders. We could then add more granular controls to toggle on/off individual labels in a separate PR.

Will try to land by the end of the week, but hopefully the deploy preview URLs are usable for the time being to test out the feature futher.

[jluethi]

That's awesome! We've deployed the branch locally as well to use it with some non-public data and our authentication, I'll have a look if I spot any general issues coming up.

Very much looking forward to using this more, thanks a lot for your efforts here Trevor!

[jluethi]

I tested this current version a bit more, it's really amazing @manzt ! 🎉

I visualized some large, remote (streamed not from my local machine, but still from a server in my institution) images we have with 39632 labels and it holds up! One can still dynamically browse it, it loads through the resolutions etc.

Screen.Recording.2025-03-06.at.17.52.21.mp4

One can notice some slight lags, e.g. when changing opacity. But it still works nicely. Great to see that it scales to that level!

Re lookup table: It's looking really good with the current preset! The only thing that would be worth changing is the default opacity. In case we don't default to 0 opacity, 100% opacity for labels is quite strong. 50-60% opacity looks pretty good in my tests though and would be worth considering.

---

I also tested it in the multiple labels in the same image, with the labels having different resolutions. That fails with the following error on the console: "All labels must share the same shape. Mismatched labels found: [1,540,1280]". I'd say this is a fair initial constraint and we may look into making a proposal to tackle that later.

For multiple labels of the same size, it works nicely!

---

Big 👏🏻👏🏻👏🏻 on this effort, it's super cool! And besides maybe changing the default label opacity, I'd say this is a nice first version that can be built on!

[manzt]

One can notice some slight lags, e.g. when changing opacity. But it still works nicely. Great to see that it scales to that level!

The fact it happens when moving the opacity slider makes me think there is something in the layer/shader that isn't optimized. Those kind of changes should be instant (conditioned on having the data in memory). Maybe something for a follow up PR with some profiling.

50-60% opacity looks pretty good in my tests though and would be worth considering.

Nice suggestion. Just set to 0.5 be default.

I also tested it in the multiple labels in the same image, with the labels having different resolutions.

I think it should be totally doable to support labels with separate resolutions. It was easiest to support "aligning" the source selection (for t,z,c) on a shared set of axes. The code is here:

vizarr/src/io.ts

Lines 251 to 291 in 16d79cc

	function getTransformSourceSelectionFromLabels(
	labelsResolutions: Array<{ shape: Array<number>; labels: Array<string> }>,
	source: { shape: Array<number>; labels: Array<string> },
	) {
	// representative source for labels
	const labelsSource = labelsResolutions[0];
	utils.assert(
	source.shape.length === source.labels.length,
	`Image source axes and shape are not same rank. Got ${JSON.stringify(source)}`,
	);
	utils.assert(
	labelsSource.shape.length === labelsSource.labels.length,
	`Label axes and shape are not same rank. Got ${JSON.stringify(labelsSource)}`,
	);
	utils.assert(
	labelsSource.labels.every((label) => source.labels.includes(label)),
	`Label axes MUST be a subset of source. Source: ${JSON.stringify(source.labels)} Labels: ${JSON.stringify(labelsSource.labels)}`,
	);
	for (const { labels, shape } of labelsResolutions.slice(1)) {
	utils.assert(
	utils.zip(labels, labelsSource.labels).every(([a, b]) => a === b),
	`Error: All labels must share the same axes. Mismatched labels found: ${JSON.stringify(labels)}`,
	);
	utils.assert(
	utils.zip(shape, labelsSource.shape).every(([a, b]) => a === b),
	`Error: All labels must share the same shape. Mismatched labels found: ${JSON.stringify(shape)}`,
	);
	}
	// Identify labels that should always map to 0, regardless of the source selection.
	const excludeFromTransformedSelection = new Set(
	utils
	.zip(labelsSource.labels, labelsSource.shape)
	.filter(([_, size]) => size === 1)
	.map(([name, _]) => name),
	);
	return (sourceSelection: Array<number>): Array<number> => {
	return labelsSource.labels.map((name) =>
	excludeFromTransformedSelection.has(name) ? 0 : sourceSelection[source.labels.indexOf(name)],
	);
	};
	}

. We could make a transform source selection for each label, rather than having the labels share the same transform.

[manzt]

I also tested it in the multiple labels in the same image, with the labels having different resolutions.

Would you mind trying with the latest changes? I think I found a solution to support separate labels with different resolutions.

[jluethi]

Would you mind trying with the latest changes? I think I found a solution to support separate labels with different resolutions.

Super cool! I can confirm that this now works with labels at different resolutions as well! ❤️ Thanks @manzt !

And the default 50% opacity is nice!

From the "testing it as a user" side, I think this is a really good initial version of it, ready to be shipped! 👏🏻

[manzt]

Alright, added a basic UI (default off for the labels). Each label needs to be toggled on to be visible. Let's ship it!

[will-moore]

👍 Great!! 💯

[joshmoore]

👏