Optimize Tests and reduce CI time

[NJManganelli]

CI Tests take up a notable amount of time and there's optimization potential to streamline things.

As an example, one test added for PR #1272 was taking ~8minutes of total single-core test-time pre-optimization.

Overall, coffea tests are a significant portion of the planned scikit-hep integration tests, so this is important to address

Crossreference: #1254

[lgray]

One reason that python 3.13 takes so long on ubuntu is that it is building numpy for reasons that I have not understood. I'll dig here! This is obviously taking an enormous amount of unnecessary time.

[lgray]

This appears to be some negative interference between tritonclient and python 3.13, where numpy doesn't make wheels for numpy 1.x for python 3.13 and tritonclient doesn't yet allow numpy 2. So we have to build it from scratch. Yikes!

triton-inference-server/client#826