Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use the same password for /metrics in ECS tasks for both prod and dev #187

Closed
LDiazN opened this issue Feb 18, 2025 · 1 comment · Fixed by #199
Closed

Use the same password for /metrics in ECS tasks for both prod and dev #187

LDiazN opened this issue Feb 18, 2025 · 1 comment · Fixed by #199
Assignees
Labels
enhancement New feature request or improvement to existing functionality priority/medium Normal priority issue

Comments

@LDiazN
Copy link
Contributor

LDiazN commented Feb 18, 2025

We can monitor ECS tasks with a Prometheus thanks to this scraping config, but unfortunately we need a different job for both prod and dev because of the basic auth section, the password is different in dev and prod.

In order to scrape metrics for both prod and dev, we would have to create a different job for each environment, duplicating a lot of code

So a possible solution would be to use the same password for both prod and dev (see #186 (comment))

The final result should:

  • have a single job block with all the required settings to scrape ECS tasks, both from prod and dev
  • allow differentiating the environment with a Prometheus label (environment = dev for example)
@LDiazN LDiazN added the enhancement New feature request or improvement to existing functionality label Feb 18, 2025
@LDiazN LDiazN self-assigned this Feb 18, 2025
@LDiazN
Copy link
Contributor Author

LDiazN commented Feb 24, 2025

There's a problem with this task, we need the Prometheus password to be shared between environments but each environment generates its own password:

In the dev environment:
https://github.com/ooni/devops/blob/a4106269f2933fa79c5e1900428533c9e3efd41a/tf/environments/dev/main.tf#L199C5-L212C2

In the prod environment:
https://github.com/ooni/devops/blob/a4106269f2933fa79c5e1900428533c9e3efd41a/tf/environments/prod/main.tf#L221C1-L234C2

Is there any way in which we can access the same credentials in both files?

We could just write the password manually in both environments and change the code to lookup the password instead of actually creating a random password, but this would imply a little more manual maintenance

@hellais hellais added the priority/medium Normal priority issue label Feb 24, 2025
@LDiazN LDiazN linked a pull request Feb 25, 2025 that will close this issue
LDiazN added a commit that referenced this issue Feb 27, 2025
This PR will change the dev environment configuration to use the same
password for /metrics in both prod and dev for ECS tasks

In order to do this we also changed the way the credentials are stored
to use `aws_ssm` parameters instead of `secrets_manager` (also related
to #114)

Since this change was breaking the monitoring for several systems in
prod that were using the old password, I used the old secrets manager
password value for the new ssm manager password, meaning that a change
in the secrets manager password can lead to breaking monitoring for some
systems.

For this reason, we should add a follow up issue to migrate all
`prometheus_metrics_password` occurrences to use ssm and run that
migration carefully to avoid disruptions in production

This PR solves #187
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature request or improvement to existing functionality priority/medium Normal priority issue
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants