Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support metrics for web UI natively instead of using prometheus #630

Merged
merged 3 commits into from
May 23, 2024

Conversation

mwylde
Copy link
Member

@mwylde mwylde commented May 23, 2024

The web UI has a nice feature that shows commonly-used job metrics directly in the pipeline detail page, including event rate metrics and backpressure metrics. Today, this works like this:

  [Arroyo worker] --pushes--> [Prometheus push gateway] <--reads-- [Prometheus]
                                                                     ↑ queries
                                                                   [Arroyo API]

requiring prometheus and pushgateway in order for the web ui metrics to work.

As part of the effort in 0.11 to remove all external dependencies (following on #626), this PR removes the need for prometheus by adding a simple metrics system within the controller.

Now, the controller queries metrics from the workers, computes rates using a exponentially-weighted moving average, stores a limited history (5 minutes), and responds to queries from the API for those metrics.

@mwylde mwylde merged commit efb0a5a into master May 23, 2024
6 checks passed
@mwylde mwylde deleted the metrics branch May 23, 2024 19:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant