feat: add stream stats endpoint

[rickstaa]

This pull request adds a new stream stats endpoint which can be used to retrieve the fps metrics in a way that doesn't affect performance. The next metrics needed for the research pod are perceived delay and temporal consistency.

API

Current API endpoint response format is:

• /stats: provides a json dict with the fps per stream id:

{
  "d1cef5ee-af29-40e3-9319-94bd6198f7e4": {
    "fps": 26.9914709381012,
    "minute_avg_fps": 27.3940920272762,
    "minute_fps_array": [26.9721219115062, 26.9906902091821, 27.9852092292336, 26.9906612312192, 27.976133001411, 26.9760203680755, 27.9585092367844, 26.9800495214337, 27.9941284555052, 26.9949415258843, 26.9525814878732, 27.979124747047, 26.9477915641951, 27.973701028521, 26.9471870972504, 27.9740528910397, 26.990279990455, 27.9709352333859, 26.9890938151392, 27.9678948983217, 26.982630606103, 27.9854588675333, 26.9878387131121, 26.9863939729004, 26.9792972474059, 26.9742334030313, 27.972160874709, 26.9870556317982, 27.9858339067024, 26.9771424829943, 26.9765737861122, 27.9734389120503, 26.9795166922633, 27.9746999890651, 26.9503308098177, 27.9816202167904, 26.9772748023089, 27.9668710039469, 26.949980216866, 26.9940629258102, 27.9736812976316, 26.9859593133103, 27.9753313528226, 26.9702430026922, 27.9873811615488, 26.9769022953053, 26.9767711386479, 27.9656843434855, 26.9945186819871, 27.9507721776836, 26.9851025502976, 27.9732248963055, 26.992512951714, 26.9902535225705, 27.9717879343986, 26.977746919863, 27.9980087256232, 26.9758180943609, 27.9788278334353, 26.9914709381012]
  }
}

• /stats/{stream_id}: provides a json dict with the fps for a given stream.

{
    "fps": 26.9914709381012,
    "minute_avg_fps": 27.3940920272762,
    "minute_fps_array": [26.9721219115062, 26.9906902091821, 27.9852092292336, 26.9906612312192, 27.976133001411, 26.9760203680755, 27.9585092367844, 26.9800495214337, 27.9941284555052, 26.9949415258843, 26.9525814878732, 27.979124747047, 26.9477915641951, 27.973701028521, 26.9471870972504, 27.9740528910397, 26.990279990455, 27.9709352333859, 26.9890938151392, 27.9678948983217, 26.982630606103, 27.9854588675333, 26.9878387131121, 26.9863939729004, 26.9792972474059, 26.9742334030313, 27.972160874709, 26.9870556317982, 27.9858339067024, 26.9771424829943, 26.9765737861122, 27.9734389120503, 26.9795166922633, 27.9746999890651, 26.9503308098177, 27.9816202167904, 26.9772748023089, 27.9668710039469, 26.949980216866, 26.9940629258102, 27.9736812976316, 26.9859593133103, 27.9753313528226, 26.9702430026922, 27.9873811615488, 26.9769022953053, 26.9767711386479, 27.9656843434855, 26.9945186819871, 27.9507721776836, 26.9851025502976, 27.9732248963055, 26.992512951714, 26.9902535225705, 27.9717879343986, 26.977746919863, 27.9980087256232, 26.9758180943609, 27.9788278334353, 26.9914709381012]
 }

Possible improvements are:

• Using something more user friendly as the id. This one corresponds to the webrtc id which also has its benefits.

Performance Investigation

@eliteprox I created a small python script to see what impact this new endpoint has on the ComfyStream performance (See #80). Below you will see the results of running the Dreamshaper workflow inside a dev container. I used the following command:

sudo env "PATH=$PATH" python monitor_pid_resources.py --pid 18633 --spy --output resources_output.csv --duration 120

The spy profiler will take some resources itself so it will drop fps with 3-5.

Dreamshaper Without FPS endpoint

Without Spy Profiling

I get a stable 15-16 FPS.

resources_output.csv

AVERAGE - CPU: 412.28%, RAM: 5778.14MB, GPU: 54.00%, VRAM: 3876.00MB

With Spy Profilling

Here I now get 10-11 fps due to the resources to do such detailed profiling.
resources_output.csv

Show flamegraph

AVERAGE - CPU: 297.05%, RAM: 5791.69MB, GPU: 39.52%, VRAM: 3876.00MB

Dreamshaper With FPS Endpoint

Without Spy Profiling

Without calling fps every second:

I get 15-16 fps and the same usage as before.

resources_output.csv

AVERAGE - CPU: 419.03%, RAM: 5501.18MB, GPU: 55.67%, VRAM: 3876.00MB

Call fps every second:

I get the same fps and results

resources_output.csv

AVERAGE - CPU: 414.68%, RAM: 5626.44MB, GPU: 55.50%, VRAM: 3876.00MB

Even if I do 100 requests per seconsd nothing changes.

With Spy Profiling

Without calling fps every second:

I get 10-11 fps similar to above.

resources_output.csv

Show flamegraph

AVERAGE - CPU: 305.45%, RAM: 5668.24MB, GPU: 39.15%, VRAM: 3876.00MB

Call fps every second:

resources_output.csv

Show flamegraph

AVERAGE - CPU: 299.04%, RAM: 5740.08MB, GPU: 38.77%, VRAM: 3876.00MB

Conclusion

This pull request adds a new stats endpoint without affecting the performance.

[rickstaa]

@ecmulli The API format is ready for your review:

{
  "d1cef5ee-af29-40e3-9319-94bd6198f7e4": {
    "fps": 26.9914709381012,
    "minute_avg_fps": 27.3940920272762,
    "minute_fps_array": [26.9721219115062, 26.9906902091821, 27.9852092292336, 26.9906612312192, 27.976133001411, 26.9760203680755, 27.9585092367844, 26.9800495214337, 27.9941284555052, 26.9949415258843, 26.9525814878732, 27.979124747047, 26.9477915641951, 27.973701028521, 26.9471870972504, 27.9740528910397, 26.990279990455, 27.9709352333859, 26.9890938151392, 27.9678948983217, 26.982630606103, 27.9854588675333, 26.9878387131121, 26.9863939729004, 26.9792972474059, 26.9742334030313, 27.972160874709, 26.9870556317982, 27.9858339067024, 26.9771424829943, 26.9765737861122, 27.9734389120503, 26.9795166922633, 27.9746999890651, 26.9503308098177, 27.9816202167904, 26.9772748023089, 27.9668710039469, 26.949980216866, 26.9940629258102, 27.9736812976316, 26.9859593133103, 27.9753313528226, 26.9702430026922, 27.9873811615488, 26.9769022953053, 26.9767711386479, 27.9656843434855, 26.9945186819871, 27.9507721776836, 26.9851025502976, 27.9732248963055, 26.992512951714, 26.9902535225705, 27.9717879343986, 26.977746919863, 27.9980087256232, 26.9758180943609, 27.9788278334353, 26.9914709381012]
  }
}

Let me know if you want to change something, before sending it to John for review. I will create a CLI which users can use to get the stats over a longer period.

[rickstaa]

@eliteprox here are the new results after the fix in #80. I didn't add the flamegraph since that should not change.

Profiler results

With fps metrics

Without calling fps endpoint

AVERAGE - CPU: 318.29%, RAM: 19326.14MB, GPU: 46.72%, VRAM: 4318.00MB

Call FPS endpoint in loop

When calling in a loop 30 times a second (i.e. watch -n 0.033 curl -s http://localhost:8889/streams/stats . I get:

AVERAGE - CPU: 324.97%, RAM: 19241.18MB, GPU: 45.85%, VRAM: 4318.00MB

Without fps metrics

AVERAGE - CPU: 330.11%, RAM: 19157.20MB, GPU: 46.47%, VRAM: 4318.00MB

[eliteprox]

LGTM! Thanks for the PR @rickstaa!