Refactor top-level LSST metadetect

[arunkannawadi]

This is a preview of the refactoring I mentioned in #220 . It currently refactors only the top-most level making run_metadetect, which is the typical entry point, a wrapper around a Task. The extensive coverage with unit tests should make us all feel confident going about such refactoring. The fact that the the unit tests pass without any changes also shows that the public APIs remain unchanged.

[beckermr]

Why were the ngmix upstream tests removed?

Ditto for the extra test skeips etc. All of that needs to be backed out before we can consider merging this.

[arunkannawadi]

Thanks to @ismael-mendoza for doing the big chunk of this work. We've tested this by also running the shear recovery tests in descwl-shear-sims (shear_test_descwl/test_shear_meas) and that seems to pass.

[arunkannawadi]

@esheldon @beckermr - have you had a chance to go through this PR?

[esheldon]

This looks OK to me. I think we need to run it at scale to stress test it, and look at outputs for sanity checks.

[arunkannawadi]

I am happy to run extensive tests but would need some handholding to get started.

[beckermr]

I have some comments/questions. Sorry for the delay!

[arunkannawadi]

I made changes/addressed the questions from previous review and I think this is ready for another round of review, Matt.

[beckermr]

LGTM on the code! One more request.

Can we add some documentation somewhere or links to it to explain a bit about how the config works, how we can ensure the config is not changed, etc.?

We had a ton of discussion on this, but I forgot the results and it feels like we should write it down.

[arunkannawadi]

I've added a README to the lsst subpackage explaining some of this. Let me know what else you'd like to see.

Additionally, I'd like to include py3.12 to the matrix test (not necessarily in this PR though) but it looks like it couldn't find consistent set of dependencies. GitHub Actions log here.

[beckermr]

Looks like sep is not on numpy2 yet and the feedstock is falling behind. I am working on it now.

[arunkannawadi]

Thank you! However, the LSST-specific tests, which is where I tried adding 3.12 should not require sep at all in the first place. It looks like it's trying to install everything in requirements.txt when only a subset of them are required for the lsst subpackage. sxdes is another one that's not required.

[beckermr]

Ah we can fix that as well by separating out the files. Are you adding 3.12 here or in another PR?

[arunkannawadi]

I will add 3.12 in another PR.

[beckermr]

@arunkannawadi I can help there to separate out the requirements if needed!

[arunkannawadi]

Alright, thanks. That'll be in a week or so after this week's stackvana is out. I don't have the permissions to merge the PR, so I'll leave that to you and Erin.

[beckermr]

@esheldon you ok to merge here?