managing ROIs that disappear/change over time

[amchagas]

Hi folks!

I am working on a project to monitor drosophila embryo development over long periods of time.

setup:

• We have a cheap ELP camera recording a 3D printed plate with 4 wells.
• Within each well there are 6 embryos in their eggs.
• Using Bonsai pipeline we:
  • take a 4fps video of the plate for ~16 hours
  • create ROIs of each embryo (based on HSV threshold) and track the width, height and average brightness of each embryo.

problem 1:

Sometimes the ROI indexes change overtime (I think this is due to the fact that they have similar sizes), even though I am not using any type of sort for them. When that happens, my datastreams get mixed and so I would like to avoid this automatic changes in the ROI indexes.

As one solution for this, we improved lighting conditions and everything looks quite stable now. However it would be good to know if there is a way to manually fix ROI indexes/locations in software. The reason being that if other people use this pipeline and are not careful with lighting they are going to get data that is mixed and looks good enough for analysis (the worst kind of data)

problem2:

The embryos hatch at different times, and when the first one out of the 24 hatches and moves out of its original place, that ROI is "lost" and an exception is thrown since that specific ROI index does not exist anymore.

Question:

Is there an easy way to handle this exception? maybe put all the values related to that ROI and index to zeros?

I have addded a sample video to a dropbox folder. This can be accessed here

I hope all of this is clear. Thanks for developing Bonsai and for the support here!!

---

Update 1:

Seems Problem 1 can be solved by using a conversion to grayscale and then simple thresholding, instead of using HSV thresholding. From what I was able to put together, as the embryos develop and get brighter in their center, the HSV threshold method was giving out dark spots in the middle which was significantly changing the contours detected by "find contours" and therefore giving out ROI changes...

Update 2:

Seems I was not correct with Update 1. The problem is not as pronounced when I use simple grayscale and thresholding, but it is still there... here is a small gif showing what is going on:

[bruno-f-cruz]

Could you perhaps link a shorter video please? Also, would sort the ROIs by X position work in your case?

[amchagas]

Thanks for looking into this @bruno-f-cruz ! Sorting the ROIs by spatial location would work great in this case (is this what you mean by X location?)

Here is a shorter video https://www.dropbox.com/s/8ije0pujqqdp9th/cropped.mp4?dl=0

[bruno-f-cruz]

Heya,

I think I would always run some sort of post-hoc analysis on these videos. But I will leave you with an example to at least solve problem 1.
TLDR, if you calculate the signed distance between a edge of the image to each blob, it turns out to be a pretty easy metric to sort the identity of the blobs. You could come up with something else and modify the script. This was the first metric that came to mind.
track_movement.zip

Regarding problem 2. The hard part is not so much preventing the crash as it is reordering the blobs. To prevent the crash you could apply a Merge on the list of blobs or Take(SizeOfList). This will always return all the blobs, ordered, inside the list.
The problem then becomes an identity assignment issue (i.e., if you droped the 4th blob, the 5th blob will now become your 4th). In your case you could code some heuristc that determines the difference between all centroids and previous frame centroids. The one with the largest different to every single one of them gets kicked out. This would probably be an heuristic to be written in a PythonTransform node perhaps.

I look forward to see what other clever solutions other people might come up with!
Bruno