Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ui: Add camera selector to input #3

Closed
wants to merge 12 commits into from

Conversation

victorges
Copy link
Contributor

@victorges victorges commented Dec 4, 2024

This is to allow changing the input camera when using the comfystream app. For example,
this will allow using a virtual camera from OBS as the input, which was the main goal and
test scenario here.

I have also updated the use-peer logic to support updating the RTC connection when the
stream changes.

Notice that I wasn't able to run the comfystream flow E2E so was only able to test the
frontend select component, but not if the RTC logic works when we switch it. It could be
helpful to have an example comfyui workflow in the repo for easy testing.

@yondonfu
Copy link
Owner

yondonfu commented Dec 4, 2024

Will review when I'm back in front of the computer by beginning of next week! In the meantime, a brief high level description of the changes + expected behavior would be helpful for the review.

@victorges
Copy link
Contributor Author

@yondonfu Sorry, I opened the PR so we could start testing it but forgot it would go to your repo. Added the description!

@victorges
Copy link
Contributor Author

@yondonfu FYI I'm going to update this tomorrow (Monday) so the camera picker goes in the setup popup instead of the camera view. This will make the code a lot simpler and will also avoid the problem of switching the camera after the stream has started, which isn't working well rn. I can ping you again once it's ready for review.

@victorges
Copy link
Contributor Author

@yondonfu ok Yondon it's good to go now :)

For some reason it started getting OverconstrainedError
when requesting exact resolution with a deviceId.
Sometimes the camera fails to return a 512x512 stream
when we ask for exact resolution+device. Fix it by creating
a canvas element instead and manually drawing the video
inside, capturing its stream instead.

Man the web is madness.
@victorges
Copy link
Contributor Author

victorges commented Dec 10, 2024

@yondonfu OK one last update... So what happened was that after adding the deviceId constraint, for some really weird reason I started getting a OverconstrainedError when getting the media stream (even if I'm picking from the same device as when not specifying a deviceId...).

So I first switched back to ideal request on the resolution here: d2591d8

The problem then was that I wasn't getting a 512x512 from the camera anymore. It was an inconsistent behavior and I normally got 512x512 only on the first load of the page then I got 512x480. To fix this and make sure we always got a 512x512 stream out of the webcam component, I implemented a canvas element where we draw the stream instead and then always extract a 512x512 stream from it. Here it is: 21c5654

The irony here is that, if we had started with this implementation from the beginning, we could even keep the dynamic camera switcher under the camera, since the output stream would always stay the same and it wouldn't require any changes on the use-peer RTC handler either. I have already spent way too much time on this though and will leave the camera switcher in the settings popup which I think is a good UX anyway. It is consistent+more privacy aware, allowing you to pick camera before streaming, like joining a call.

NOW I'm (hopefully) done. Eric already ran his tests and it seems to be all good on his setup. So let me know what you think on the review.

Copy link
Owner

@yondonfu yondonfu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for these changes! I tested them locally and confirmed that the camera selector is working with OBS Virtual Camera.

I recently introduced the ability to configure the webcam frame rate in 40db32d which resulted in a bunch of conflicts with this branch. The conflicts were a bit hairy so I ended up copying your branch and resolving the conflicts in my own PR here with a few additional tweaks to make your changes play well with my frame rate related changes.

In the interest of getting these changes in sooner rather than later, I'm going to merge that PR and close this one, but do lmk if you spot any issues with the resolutions + tweaks in that PR!

@yondonfu
Copy link
Owner

Superseded by #7

@yondonfu yondonfu closed this Dec 13, 2024
eliteprox added a commit that referenced this pull request Mar 10, 2025
* remove node package install from launch (#3)
* fix npm package install for devcontainer
ryanontheinside pushed a commit to ryanontheinside/comfystream_inside that referenced this pull request Mar 12, 2025
commit 37a24c1
Author: John | Elite Encoder <john@eliteencoder.net>
Date:   Tue Mar 11 17:29:14 2025 -0400

    Update file path in README.md (yondonfu#156)

commit 8f7ce22
Author: Rick Staa <rick.staa@outlook.com>
Date:   Tue Mar 11 22:28:57 2025 +0100

    fix(dev): handle NoneType error in monitor script (yondonfu#155)

    This commit fixes a NoneType error in the resource monitoring script that occurs
    when certain processes lack a name.

commit fc0f6ef
Author: John | Elite Encoder <john@eliteencoder.net>
Date:   Tue Mar 11 13:02:41 2025 -0400

    fix: prevent `AssertionError` by setting `max_workers` to 1 (yondonfu#154)

    * change max_workers to 1, add error handling to VideoStreamTrack and AudioStreamTrack

commit f0200a2
Author: John | Elite Encoder <john@eliteencoder.net>
Date:   Tue Mar 11 10:09:30 2025 -0400

    chore: update ComfyUI-TensorRT node to origin branch (yondonfu#153)

commit 7513393
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Mon Mar 10 17:04:22 2025 -0400

    chore(deps): bump docker/login-action from 2 to 3 (yondonfu#149)

    Bumps [docker/login-action](https://github.com/docker/login-action) from 2 to 3.
    - [Release notes](https://github.com/docker/login-action/releases)
    - [Commits](docker/login-action@v2...v3)

    ---
    updated-dependencies:
    - dependency-name: docker/login-action
      dependency-type: direct:production
      update-type: version-update:semver-major
    ...

    Signed-off-by: dependabot[bot] <support@github.com>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit 098fae6
Author: John | Elite Encoder <john@eliteencoder.net>
Date:   Mon Mar 10 15:52:10 2025 -0400

    fix: remove node package install from launch (yondonfu#151)

    * remove node package install from launch (yondonfu#3)
    * fix npm package install for devcontainer

commit 8d21037
Author: Rick Staa <rick.staa@outlook.com>
Date:   Mon Mar 10 20:30:02 2025 +0100

    fix: include Prometheus client dependency in project.toml (yondonfu#152)

    This commit ensures that the `prometheus-client` dependency is explicitly
    specified in both `requirements.txt` and `pyproject.toml` for consistency
    and proper dependency management.

commit 76d820c
Author: Rick Staa <rick.staa@outlook.com>
Date:   Mon Mar 10 17:56:32 2025 +0100

    feat(server): add Prometheus metrics endpoint (yondonfu#134)

    This commit introduces a `/metrics` endpoint for Prometheus to scrape stream
    metrics, including FPS and average FPS per stream. Additionally, it adds the
    `--stream-id-label` argument, allowing users to optionally include the `stream-id`
    label in Prometheus metrics.

    Co-authored-by: jpotter92 <git@hjpotter92.email>

commit 8aa9baa
Author: Rick Staa <rick.staa@outlook.com>
Date:   Mon Mar 10 15:02:29 2025 +0100

    feat: add resource profiler script (yondonfu#80)

    * feat(dev): add profiler script for resource usage tracking
    This commit adds a lightweight profiler script for developers working on
    ComfyStream to monitor resource usage and compare it against previous runs.

commit c343599
Author: Varshith Bathini <varshith15@gmail.com>
Date:   Sat Mar 8 03:00:37 2025 +0530

    feat: multi prompt dynamic node update (yondonfu#93)

    * feat: multi prompt dynamic node update

    * fix formatting

    ---------

    Co-authored-by: Elite <john@eliteencoder.net>

commit 98d4309
Author: John | Elite Encoder <john@eliteencoder.net>
Date:   Fri Mar 7 16:20:41 2025 -0500

    chore: reorganize workflows, add cliptext model to workflows (yondonfu#125)

    * reorganize workflows for easier indexing, add cliptext model to workflows for reliability, add 512x512 image for depthmap accelerated workflow, add florence-sam2 workflow + workflows using ConditioningConcat
    ---------
    Co-authored-by: ryanontheinstide <ryanfosdick87@gmail.com>

commit a2913c6
Author: John | Elite Encoder <john@eliteencoder.net>
Date:   Fri Mar 7 16:19:07 2025 -0500

    Revert "chore(deps): bump tailwind-merge from 2.6.0 to 3.0.2 in /ui (yondonfu#105)" (yondonfu#143)

    This reverts commit fac67e5.

commit 167efd3
Author: John | Elite Encoder <john@eliteencoder.net>
Date:   Fri Mar 7 16:18:39 2025 -0500

    Revert "chore(deps-dev): bump tailwindcss from 3.4.17 to 4.0.12 in /ui (yondonfu#142)" (yondonfu#144)

    This reverts commit a67a741.

commit a67a741
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Fri Mar 7 15:41:25 2025 -0500

    chore(deps-dev): bump tailwindcss from 3.4.17 to 4.0.12 in /ui (yondonfu#142)

    Bumps [tailwindcss](https://github.com/tailwindlabs/tailwindcss/tree/HEAD/packages/tailwindcss) from 3.4.17 to 4.0.12.
    - [Release notes](https://github.com/tailwindlabs/tailwindcss/releases)
    - [Changelog](https://github.com/tailwindlabs/tailwindcss/blob/main/CHANGELOG.md)
    - [Commits](https://github.com/tailwindlabs/tailwindcss/commits/v4.0.12/packages/tailwindcss)

    ---
    updated-dependencies:
    - dependency-name: tailwindcss
      dependency-type: direct:development
      update-type: version-update:semver-major
    ...

    Signed-off-by: dependabot[bot] <support@github.com>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit fac67e5
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Fri Mar 7 15:37:03 2025 -0500

    chore(deps): bump tailwind-merge from 2.6.0 to 3.0.2 in /ui (yondonfu#105)

    Bumps [tailwind-merge](https://github.com/dcastil/tailwind-merge) from 2.6.0 to 3.0.2.
    - [Release notes](https://github.com/dcastil/tailwind-merge/releases)
    - [Commits](dcastil/tailwind-merge@v2.6.0...v3.0.2)

    ---
    updated-dependencies:
    - dependency-name: tailwind-merge
      dependency-type: direct:production
      update-type: version-update:semver-major
    ...

    Signed-off-by: dependabot[bot] <support@github.com>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit f8e684c
Author: John | Elite Encoder <john@eliteencoder.net>
Date:   Fri Mar 7 15:25:33 2025 -0500

    fix default camera bug (yondonfu#139)

commit fadf199
Author: John | Elite Encoder <john@eliteencoder.net>
Date:   Fri Mar 7 15:25:12 2025 -0500

    fix: remove missing startup script (yondonfu#138)

    * fix missing startup script

commit a31204e
Author: Rick Staa <rick.staa@outlook.com>
Date:   Fri Mar 7 19:12:16 2025 +0100

    feat(ui): add settings search queries (yondonfu#91)

    * feat(ui): add settings search queries
    This commit gives users the ability to set the stream settings using query parameters.
    ---------
    Co-authored-by: ryanontheinstide <ryanfosdick87@gmail.com>
    Co-authored-by: Elite <john@eliteencoder.net>

commit 5725fb4
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Fri Mar 7 12:12:58 2025 -0500

    chore(deps-dev): bump husky from 8.0.3 to 9.1.7 in /ui (yondonfu#104)

    Bumps [husky](https://github.com/typicode/husky) from 8.0.3 to 9.1.7.
    - [Release notes](https://github.com/typicode/husky/releases)
    - [Commits](typicode/husky@v8.0.3...v9.1.7)

    ---
    updated-dependencies:
    - dependency-name: husky
      dependency-type: direct:development
      update-type: version-update:semver-major
    ...

    Signed-off-by: dependabot[bot] <support@github.com>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit a38cb3e
Author: Rick Staa <rick.staa@outlook.com>
Date:   Fri Mar 7 18:08:37 2025 +0100

    refactor(server): improve FPS Stats collection logic (yondonfu#141)

    This commit extracts the FPS statistics collection into its own class to keep
    the `VideoStreamTrack` implementation cleaner and more maintainable. This also
    makes the logic reusable across different components.

commit 8622a31
Author: hjpotter92 <hjpotter92@users.noreply.github.com>
Date:   Fri Mar 7 18:14:23 2025 +0530

    workflows: Disable github action unless running in livepeer fork (yondonfu#140)

    Rename ui-kit release workflow

commit a55e3f3
Merge: a65a0f3 c990723
Author: hjpotter92 <hjpotter92@users.noreply.github.com>
Date:   Fri Mar 7 13:07:10 2025 +0530

    Merge pull request yondonfu#132 from livepeer/main

    backporting from `livepeer/` fork

commit a65a0f3
Author: John | Elite Encoder <john@eliteencoder.net>
Date:   Thu Mar 6 22:14:39 2025 -0500

    remove multi-controlnet patch (yondonfu#136)

commit c990723
Author: John | Elite Encoder <john@eliteencoder.net>
Date:   Fri Mar 7 00:10:58 2025 +0000

    fix whitespace

commit 5bcc444
Merge: b298abe f1b0fb1
Author: John | Elite Encoder <john@eliteencoder.net>
Date:   Thu Mar 6 10:07:48 2025 -0500

    Merge branch 'main' into main

commit b298abe
Author: hjpotter92 <hjpotter92@users.noreply.github.com>
Date:   Thu Mar 6 11:53:52 2025 +0530

    Add workflow for building comfyui-base images (yondonfu#2)

    * docker: Add workflow for building comfyui-base images

    * Add attestations to built docker images

commit f1b0fb1
Author: Rick Staa <rick.staa@outlook.com>
Date:   Wed Mar 5 23:43:35 2025 +0100

    fix: ensure patched torch graph is always synced on inference errors (yondonfu#129)

    * fix: ensure patched torch graph is always synced on inference errors

    This commit ensures that the patched torch graph remains synchronized
    even when an error occurs during inference, preventing potential
    inconsistencies and adds logging for controlnet tensor cloning errors
    ---------

    Co-authored-by: Elite <john@eliteencoder.net>

commit 93c3d35
Author: John | Elite Encoder <john@eliteencoder.net>
Date:   Wed Mar 5 16:49:57 2025 -0500

    fix mediapipe 0.10.20 conflict by unpinning protbuf in ComfyUI TensorRT (yondonfu#126)

commit 5ed95c2
Author: Rick Staa <rick.staa@outlook.com>
Date:   Wed Mar 5 22:03:19 2025 +0100

    refactor(dev): improve Ansible ComfyUI password behavior (yondonfu#127)

    This commit ensures that a unique password is generated on each run unless
    the user specifies a password themselves, making the deployment more secure.
    It allows users to provide their own password via extra-vars or environment
    variables while automatically generating a random password if none is provided.
    It also improves the ComfyUI caddy file template name.

commit d98e1b5
Author: Rick Staa <rick.staa@outlook.com>
Date:   Wed Mar 5 19:35:09 2025 +0100

    feat: add stream stats endpoint (yondonfu#48)

    Adds a new stream stats endpoint which can be used to retrieve the fps metrics in a way that doesn't affect performance.
    ---------

    Co-authored-by: Evan Mullins <evancmullins@gmail.com>

commit c5d5e48
Author: John | Elite Encoder <john@eliteencoder.net>
Date:   Wed Mar 5 13:17:09 2025 -0500

    resolve missing folder error (yondonfu#128)

commit 0cee22c
Author: Rick Staa <rick.staa@outlook.com>
Date:   Tue Mar 4 22:27:54 2025 +0100

    feat(dev): add Ansible playbook for ComfyStream setup (yondonfu#114)

    This commit adds an Ansible playbook for automated ComfyStream setup, switches Caddy release to `stable`, and prevents cloud-init from duplicating import lines. Also introduces a `--bare-vm` option for deploying a clean TensorDock instance without ComfyStream and refines cloud-init template behavior for better reliability.

commit d6b84ed
Author: John | Elite Encoder <john@eliteencoder.net>
Date:   Tue Mar 4 12:38:53 2025 -0500

    download latest ui files when missing (yondonfu#116)

commit 258afb8
Merge: 342e302 4dbbd13
Author: hjpotter92 <hjpotter92@users.noreply.github.com>
Date:   Tue Mar 4 16:50:50 2025 +0530

    Merge pull request yondonfu#1 from livepeer/feature/docker-builds

    workflows: Automating docker image build pipeline

commit 4dbbd13
Author: hjpotter92 <git@hjpotter92.email>
Date:   Tue Mar 4 15:26:04 2025 +0530

    dockerfile: Reduce layers by compining `RUN` stages

commit e7900e5
Author: hjpotter92 <git@hjpotter92.email>
Date:   Tue Mar 4 11:22:36 2025 +0530

    docker: Use self-hosted runner for building docker image

commit 9d6c79f
Merge: 6fdcac7 342e302
Author: hjpotter92 <hjpotter92@users.noreply.github.com>
Date:   Tue Mar 4 11:03:22 2025 +0530

    Merge branch 'main' into feature/docker-builds

commit 342e302
Author: John | Elite Encoder <john@eliteencoder.net>
Date:   Mon Mar 3 09:45:18 2025 -0500

    add publisher-id for registry publication (yondonfu#107)

commit 52f1327
Author: John | Elite Encoder <john@eliteencoder.net>
Date:   Mon Mar 3 09:44:52 2025 -0500

    release: update version to 0.0.3 (yondonfu#109)

    * Update pyproject.toml and package.json version to 0.0.3

    * workflows: Update release workflow to be more precise for ui files

    ---------

    Co-authored-by: hjpotter92 <git@hjpotter92.email>

commit d7bf4b5
Author: John | Elite Encoder <john@eliteencoder.net>
Date:   Mon Mar 3 09:34:33 2025 -0500

    keep empty static folder (yondonfu#113)

commit 6fdcac7
Author: hjpotter92 <git@hjpotter92.email>
Date:   Mon Mar 3 12:26:20 2025 +0530

    workflow: Use external github action for cleaning up disk space

commit 3c83e6a
Author: hjpotter92 <git@hjpotter92.email>
Date:   Mon Mar 3 11:28:34 2025 +0530

    workflows: Testing out github builds for docker images
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants