-
Notifications
You must be signed in to change notification settings - Fork 45
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Extend the axes fields in multiscales metadata #57
Conversation
First draft for separate axes metadataAdapted from https://github.com/saalfeldlab/n5-ij/wiki/Transformation-spec-proposal-for-NGFF#proposal-a. Design choiceAxes metadata describes the physical coordinate space (= output coordinate space after transformation). It is defined separate from Open QuestionsUnits: Do we restrict the units or is this free text? If we choose to restrict, what set of units is allowed? Some options that were brought up in #35: SI-Units, https://people.csail.mit.edu/jaffer/MIXF/CMIXF-12. Level of requirement: E.g., must there be units or is it should? Also, do we allow for types that are not in "space", "time", "channel" (right now it's SHOULD, so we do allow other values). Axes order: Axes names are not fixed any more, so we can't enforce the order Misc
|
Are we still restricting the number of dimensions to 5? NB: there's a few other places in the example JSON sections that need to be updated. |
Yes, I also made this more explicit in b3523c9.
Indeed, we should clarify this more. We can't rely on fixed axes names "xyz" any more (and the demand for this was very clear in the last meeting). But we can add a MUST or SHOULD that the spatial axes should be last.
Yes, I will wait with updating until there is a consensus to adopt this proposal and we settled some of the open questions. |
i think units are a must especially once transforms and coordinate systems and other image spaces come into the picture. btw, there is a bit of discussion around metadata of transforms in this document (https://docs.google.com/document/d/11gCzXOPUbYyuQx8fErtMO9tnOKC3kTWiL9axWkkILNE/edit) which is part extending the bids-specification. regarding the actual transformations themselves, itk has a set of good transform representations (i'm sure @thewtex can say a whole lot more :) ) |
Thanks @constantinpape for following-up on this. Would it be useful to extract a list of questions from the neuroglancer discussion that may be relevant to discuss here? Regarding the question about restricting the number of dimension to be maximally 5, and your reply making this more explicit in the docs: Is there really a consensus to do this? Also regarding enforcing explicitly for the last two dimensions to be spatial? I don't think that is generally useful, and would possible limit what can be done in the future with ome-zarr files. Tools that do have specific requirements on the contained data (e.g. requiring at least two or three spatial dimensions, or with more than 5 dimensions etc.) could just fail with an error message in those cases. |
Yes, that would be helpful.
I think there is still quite some discussions going on about this restrictions, with probably more support to lift this requirement. However, I would like to avoid opening up this topic for the next release to focus on the axis metadata format. I think lifting the requirement would be very easy in the current proposal (but let me know if you see any potential issues). Same for the axis order discussion. |
Okey. Is there a timeline for when the next release is planned? If the axes metadata is expressed as lists is a good start to express more or less dimensions. As already discussed a bit, one has to be careful by implying to much from a label name, i.e. assuming too much implicitly, e.g. when writing |
While in common cases it seems to make sense to label full coordinate spaces, e.g. where you have a common xyz coordinate space to which multiple volumes are aligned, in general I think it actually makes more sense to label individual dimensions rather than the full coordinate space, and that could be accomplished just by using longer dimension names, and possibly having some way to abbreviate them in viewers. For example: fafb.v14.x For example, you might have a combined EM/functional dataset consisting: EM alignment v2 with dimensions: Multi-channel functional recording experiment 1: dataset.v2.x Multi-channel functional recording experiment 2: dataset.v2.x By labeling dimensions we can fully represent the relationship between the three volumes. If we labeled entire coordinate spaces, we would have to assign a separate coordinate space label to each volume. |
@unidesigner ( Hi Stephan!)
yes, agreed. Label names are possibly arbitrary, but possibly not. They should be allowed to carry information (I quite like @jbms 's example), but not required to.
This is essentially what we describe in our transformation "proposal - D": giving names to input and output axes, and specifying very explicitly which input axes a given transform applies to and what output axes it generates. It's quite cumbersome though, and likely "going overboard". |
There are two steps for this: getting this PR and #63 merged to incorporate transformations (of course including discussions on what still needs to change with the proposals). Then producing example data and making sure it runs in the reference implementations (currently Fiji via MoBIE, vizarr and napari via the napari-ome-zarr plugin). I can't give you a clear timetable on the first point, because this very much depends on how much discussion and changes are needed. Updating the reference implementations should be relatively fast and be feasible in ~ 1 week (depending of course on the availability of developers / contributors of the individual tools).
In the current proposal I introduced 3 lists in the axes objects:
For the transformations please refer to #63 and have a look at the current proposal and how it relates to what you have written below. (Note that this is based on https://github.com/saalfeldlab/n5-ij/wiki/Transformation-spec-proposal-for-NGFF and inspired by Proposal C but I tried to simplify the proposal a bit; the major differences are listed on top of the PR). @jbms the current draft allows arbitrary labels and has an additional field |
My opinion on open questions
units SHOULD be
but any string is technically allowed.
Agree with SHOULD, I'll probably only ever use these tree, but others should be allowed ( e.g. "polarization angle" )
SHOULD. if units are empty "arbitrary" / "pixel" units are implied |
I noticed the units specified in the proposal all have a multiplier of 1, i.e. "nm" or "ms". But what about other multipliers, like "4nm"? Is the intention that you always have to specify units with a multiplier of 1, and rely on an affine transformation to transform your data to those units? In many cases I expect there will be no affine transform, and you just want to indicate the actual voxel size. Having to extract that from an affine transform seems undesirable. |
@jbms this is equivalent to giving the diagonal elements of the affine only. does that work for you? |
I think it may still be desirable to be able to specify the "native" resolution of the data in many cases. For example, Neuroglancer chooses the units (including multiplier) of the global coordinate space by default to match the native resolution of the data. The units/multiplier of the global coordinate space affect how coordinates are displayed, the step size for discrete movement commands, and also have implications for numerical precision. If there is just an affine transform, we could attempt to heuristically extract the units and multiplier from that, but that seems to leave more ambiguity than if the units are specified explicitly. In general, it seems that if you allow SI prefixes, which are just another form of multiplier, albeit one that is nicer to display in a scale bar, then it would be logical to also allow arbitrary multipliers. |
The way I think of it (and this is highly influenced by my exposure to xarray / netcdf), Since we don't have axes explicitly populated with values in metadata, the |
You have a good point that by using a coordinate array you gain an additional level of indirection and can use it to specify multipliers. But I would imagine that for regularly spaced data a coordinate array would not normally be specified. According to the udunits2 library, "4nm" is a unit: In general while aesthetically "nanometer" is a nicer unit to work with than "4 nanometer", I don't see why fundamentally there should be support for "1e-9 meter" as a unit but not for "4e-9 meter" as a unit. If the affine transform is not a permutation of a diagonal matrix, then we can't infer scale factors from it, so we would be back to just using whatever units are specified explicitly. For example, maybe you have aligned functional data to an EM volume, and there is an affine transform involved, but you want to specify the EM native resolution as the explicit units. It sounds like the spec is merely suggesting a syntax for units, and so "4nm" could still be used, but I think it would help to specify how these units are expected to be used. |
There are two related but separate (in my mind) issues that I feel we're mixing up a bit and will try to dis-entangle, specifically how do we distinguish between:
relating to the conversation aboveHere's how I see these mapping in @jbms comment above - please correct / confirm Jeremy.
this is (1), and I agree.
this refers to (2)
and this is (1), and I agree. is (2) in the spec?As in Jeremy's example, say I have an EM dataset, and I specify the (1) transform "this image has resolution My viewer now needs to know what space I want my EM to be displayed in before it can do the right thing. Either, the spec covers (1) only, or needs a much more concrete sense of coordinate space / axes as above to be unambiguous - something more like our proposal D. a worryI worry a little about having different "ways" of going from pixel to physical space. Someone soon is going to label axes with I personally would prefer to almost always only use a scale transform (maybe with an offset) to describe the physical extents of a dataset independent of all others and store transforms of type (2) somewhere that also explicitly says that this transform takes image "moving" to image "target". |
In regards to your worry, I would say that if you specify that the units of the output space are "4nm", then your transform had better be designed to output coordinates in units of "4nm", not units of "nm", i.e. all of the coefficients in that row of the affine transform should be divided by 4 compared to if the output space is "nm". That is the same issue that comes up with having an output unit of "um", and a scale factor of 1e-3, vs having an output unit of "nm" and a scale factor of 1. |
|
Thanks for all the discussions and feedback.
I very much agree with your assessments, esp. SHOULD for the units (relevant for all the following discussions).
I personally would also prefer "clean" units, i.e. only having "nanometer" or "nm" in the recommended unit names and not "4 nm". Regarding the distinction 1) use transformations to specify discrete to physical space vs 2) to specify physical to physical space: I agree that |
@axtimwalde Under the current proposal, I guess it would also be necessary to copy the transformation metadata that was written to volume B to the segmentation volume as well. To me a key question is how often we want to say "when using volume B, always use this transformation" as opposed to having some way of storing "here is a transformation from volume A coordinate space to volume B coordinate space" and then creating some sort of "view" of volume B with the transformation applied. |
Great points that also apply to Story 1, 2, and 3, I do not see a clear boundary between these scenarios when it comes to the transformation spec and would therefore keep them as compatible as possible. In my understanding all transformed volumes are views. The storage location for that displacement field can be an absolute URL and does not have to exist in the same container as volume A and B, so it can be re-used. It is also likely that volume A is in a different container than volume B. We may want to add a more every-day story than the BigWarp scenario for non-rigid transformations like ultra-sound scanners or radars? |
Yeah, to be clear I wasn't really drawing a distinction between affine vs non-affine transformations here, but rather just getting at the idea that the transformation could be decoupled from the volume itself. In my view Story 2 is a little different because there the cutout location is known upon creation and the "transformation" isn't going to change. For an affine or non-rigid transform you may still wish to work with the untransformed volume for various reasons, and there may be more than one possible transform. Additionally, the original volume may already be referenced elsewhere before you have computed the affine or non-rigid transformation, and in that case you don't necessarily want to change the meaning of those existing references by modifying its metadata. |
Yes, transformations can be decoupled from the volume. But I also think that there is not clear difference between a cutout offset or pixel scaling factor and a registration transformation. E.g. imagine an SEM volume stored as a 3D dataset in ome-zarr format. The microscope will most likely export data such as the nominal pixel spacing in x and y as well as a nominal spacing in z. These values can be used to display a view of the array data, but we also know that this information is not physically correct and therefore have to perform alignment and z-thickness compensation, the combination of the nominal values and the alignment transformations can now be used to display another view of the array data. Mapping the result into a canonical atlas space is yet another view of the data. Where do those transformations go? Depends on the application. Since we can store arbitrary meta-data in zarr/N5/HDF5 and friends, it is perfectly fine to for that information to be associated with a volume, or with a multi-scale group, or somewhere else. Were you suggesting to wrap all transformation meta-data into named view-tags? I find that appealing. It brings us back to groups (sequence) of transformations (@bogovicj) had this in our original proposal. Such groups could be used by named references to create compositions and subsets of transformations. In the above FIB-SEM scenario, we would most likely have the groups "microscope", "alignment", "atlas" that can be composed at will. I find this very appealing. |
Story 5. slightly different registration problem.
|
I agree that the pixel size from the microscope may be unreliable and in that sense is similar to a registration transformation. But the cutout offsets would most naturally be expressed in units of voxels (zarr-developers/zarr-specs#122) and then would be exact and immutable and independent of any other transformations. As far as other transformations, though, I don't have a concrete proposal for exactly how it might work, but what I'm suggesting is to be able to specify named transformation objects independent of any particular volume, and optionally identify the source and target coordinate spaces with some string identifier. And then similarly define named "view" objects that reference a transformation, or sequence of transformations, and an array or multiscale array. Array or multiscale arrays could also optionally specify a named coordinate space, and then viewers could potentially automatically suggest compatible transformations that could be applied. I think it could be useful to distinguish between "arrays" that are indexed by discrete coordinates and "views" that are indexed by continuous coordinates. Otherwise if we reference a given "multiscale" or "array" that has transformations applied, we would have to additionally specify if we are referring to the original discrete coordinate space or the transformed continuous coordinate space. |
This sounds to me very similar to what we currently do in MoBIE. Here is an example of a So, in that sense the transformations are decoupled from the volume itself and one can have different views on the same volume with different transformations. However, we currently require that the transformations in the One other question: I am confused about the concept of a voxel size. Already when one rotates the data a little bit the voxel grid does not align anymore with the physical coordinate system and I am confused how to define voxel size in such cases?! |
The concept of views could also help for what we are doing: We have a workflow doing segmentations and registrations, and during this process we incrementally update the Zarr metadata to store the transformations. We also may rerun the workflow with different sets of parameters, leading to versioned results that we want to be able to view and compare. If we can only have a single transformation strongly coupled with an image, we need to create a copy of the Zarr for not overwriting the original. With views we would be able to define a view for each set of parameters and tell which transformations belong to that view and which segmentations. |
Happy to see this conversation and the interest in views.
Yes, I want this too. and so does @axtimwalde wro write:
I'll wrote something up awhile ago, and I'll flesh out for tomorrow and would love to discuss. |
My vote also goes towards a simplified version of the |
A quick wrap up before the meeting tomorrow (summary from a quick pre-meeting with @sbesson, @will-moore, @joshmoore):
|
A quick point to think about too (we'll discuss in the call today, but wanted to write it down here next to everything else). The spec allows for binary storage the transform data at some path in the container. Using the dict to specify axes doesn't work in that case. So we need either to
|
Good point @bogovicj. I forgot that we allow binary data here already. Let's discuss during the meeting. |
I will merge this PR now and finalize the axes and initial transformations in a separate PR:
|
Extend the axes fields in multiscales metadata SHA: 90de9f1 Reason: push, by @constantinpape Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Final house-keeping:
A big thank you to everyone who commented here and helped to bring this into shape! |
This pull request has been mentioned on Image.sc Forum. There might be relevant details there: https://forum.image.sc/t/next-call-on-next-gen-bioimaging-data-tools-2022-01-27/60885/11 |
Proposal summary
This proposal adds "axes" metadata to describe the axes / dimensions of a physical coordinate data space and "transformation" metadata.
Both axes and transformation metadata are then used in "multiscales" to describe the physical coordinate system of the data and the mapping from data coordinates to physical coordinates (for each scale).
Note that we only introduce simple "scale" and "translation" transformations here, but intend to expand the set of supported transformations in the future. See #67 for initial discussions.
The main motivation of the proposed changes is to enable expressing "simple" spatial coordinates for a multiscale image that can be interpreted by all viewers (i.e. also viewers without support for more complex transformations like standard Fiji).
As a second design goal, the transformation spec outlined here should be extensible to more complex transformations (see again discussions in #67) to be able to later express these in the same spec.
Some example data for ome.zarr files using this proposal are available here: https://github.com/ome/ome-ngff-prototypes.