Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

new whip server #3432

Open
wants to merge 10 commits into
base: master
Choose a base branch
from
Open

new whip server #3432

wants to merge 10 commits into from

Conversation

j0sh
Copy link
Collaborator

@j0sh j0sh commented Mar 5, 2025

Introduces a WHIP endpoint at /live/video-to-video/{streamid}/whip

This strictly affects ingest only; it does not touch selection, trickle or anything beyond those.

There should also be no changes to the existing MediaMTX whip / rtmp ingest. The old ingest is safe to run alongside this and there should be no behavioral changes to that. The one modification is to tighten up request methods in the HTTP handler (adding POST prefixes etc) to prevent route conflicts with the new WHIP endpoints.

New Configuration Options

Currently implemented directly as environment variables.

  • LIVE_AI_WHIP_ADDR: UDP binding address for WebRTC (default: 0.0.0.0:7290)
  • LIVE_AI_NAT_IP: Public IP for NAT traversal (optional)
  • LIVE_AI_VIDEO_GATHER_TIMEOUT: Timeout for gathering video tracks (default: 5 seconds)
  • LIVE_AI_PLAYBACK_HOST: RTMP URL for output (default: rtmp://localhost/)
  • LIVE_AI_ALLOW_CORS: Enable CORS headers for browser clients (optional)

Other Changes

Added a clog.Info function for a slog-style logging API

Not Yet Implemented

  • Proper handling + documentation for the new configuration vars
  • ICE restarts + trickle ICE
  • DELETE
  • TCP, TURN, etc

These todos are all in Linear for later follow-up.

@github-actions github-actions bot added dependencies Pull requests that update a dependency file go Pull requests that update Go code AI Issues and PR related to the AI-video branch. labels Mar 5, 2025
Copy link

codecov bot commented Mar 5, 2025

Codecov Report

Attention: Patch coverage is 22.55370% with 649 lines in your changes missing coverage. Please review.

Project coverage is 32.04433%. Comparing base (9a27435) to head (a5633d6).
Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
media/whip_server.go 0.00000% 430 Missing ⚠️
server/ai_mediaserver.go 3.01205% 160 Missing and 1 partial ⚠️
clog/clog.go 0.00000% 29 Missing ⚠️
media/rtp_segmenter.go 82.53012% 20 Missing and 9 partials ⚠️
Additional details and impacted files

Impacted file tree graph

@@                 Coverage Diff                 @@
##              master       #3432         +/-   ##
===================================================
- Coverage   32.16679%   32.04433%   -0.12246%     
===================================================
  Files            147         150          +3     
  Lines          41033       41867        +834     
===================================================
+ Hits           13199       13416        +217     
- Misses         27058       27664        +606     
- Partials         776         787         +11     
Files with missing lines Coverage Δ
media/whip_connection.go 100.00000% <100.00000%> (ø)
clog/clog.go 53.76344% <0.00000%> (-9.93083%) ⬇️
media/rtp_segmenter.go 82.53012% <82.53012%> (ø)
server/ai_mediaserver.go 5.70687% <3.01205%> (-1.35389%) ⬇️
media/whip_server.go 0.00000% <0.00000%> (ø)

... and 3 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9a27435...a5633d6. Read the comment docs.

Files with missing lines Coverage Δ
media/whip_connection.go 100.00000% <100.00000%> (ø)
clog/clog.go 53.76344% <0.00000%> (-9.93083%) ⬇️
media/rtp_segmenter.go 82.53012% <82.53012%> (ø)
server/ai_mediaserver.go 5.70687% <3.01205%> (-1.35389%) ⬇️
media/whip_server.go 0.00000% <0.00000%> (ø)

... and 3 files with indirect coverage changes

🚀 New features to boost your workflow:
  • Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@j0sh j0sh force-pushed the ja/whip-server branch from c88ed2d to 12e26d0 Compare March 7, 2025 00:19
@j0sh j0sh marked this pull request as ready for review March 7, 2025 00:45
@j0sh
Copy link
Collaborator Author

j0sh commented Mar 7, 2025

Added the ability to disable WHIP ingest entirely by omitting the LIVE_AI_WHIP_ADDR env var. This was mostly done to fix tests but may be useful if we need to disable whip for whatever reason, without a roll back. 522d9e6

Ideally we'd have a CLI argument like -whipIngest but that can come later.

Copy link
Contributor

@leszko leszko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work @j0sh Added some comments, mostly to the ai_mediaserver.go part. I was not able to review in depth the media part, in particular the whip_server.go.

My suggestion is to address the comments and get it merged, then we can test it in action:

  1. I'd not use env vars, but move it to flags (as we do with everything else)
  2. It'd be super beneficial to describe how you test this whip endpoint locally (can be a README or even some short Notion doc)

ls.HTTPMux.Handle("/live/video-to-video/smoketest", ls.SmokeTestLiveVideo())

// Configure WHIP ingest only if an addr is specified.
// TODO use a proper cli flag
if os.Getenv("LIVE_AI_WHIP_ADDR") != "" {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about moving this to flags? I think we don't use anywhere the env variables directly in go-livepeer, but just adding more flags.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes it has been mentioned in several places this can be done as a follow up to this PR

pipeline = authResp.Pipeline
}

if len(authResp.paramsMap) > 0 {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see there is a lot of code copied from StartLiveVideo(), I'd consider refactoring it and re-use. It may be okeyish to just copy-paste the code like you did, but I think there is some functionality missing (e.g. I don't see sending stream_trace) into Kafka, which may make it more difficult to analyze.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can add the stream_trace (I think that was added after we wrote this code) but wanted to avoid modifying existing ingest code paths - they are too different and the risk was too high of breaking something

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, then let's just add the stream_trace.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added stream_trace and some other missing metrics in 7236f83

PayloadType: 105,
},

// TODO verify 42e01f won't produle bframes
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's remove this before merging

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, this was part of the default set of Pion codecs but I don't think this part even does what we need it to right now - the remote SDPs the browser is setting don't look as constrained as I was expecting. Needs more investigation

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed in a5633d6

@j0sh
Copy link
Collaborator Author

j0sh commented Mar 11, 2025

Addressed PR feedback, and added the following:

Makes WHIP not be blocked by slow (or broken) orch selection: 147af6b

Adds a packet queue limit in the segmenter: 1203f5a

Reject non-H.264 video codecs: 3c10a60

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
AI Issues and PR related to the AI-video branch. dependencies Pull requests that update a dependency file go Pull requests that update Go code
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants