docs(prometheus): Document how to set up scraping in kubernetes

[gardleopard]

Make it easier to figure out how to get prometheus scraping of metrics endpoint working

[sgibson91]

Hi @gardleopard! @GeorgianaElena and I have spent some time reviewing this, thanks very much for your contribution! This looks good to us, and we will approve, but we wanted to ask if you were aware of the interNamespaceAccessLabels config option? We believe this will achieve the same outcome as your proposal. Here are a couple of doc pages about it:

• https://z2jh.jupyter.org/en/stable/resources/reference.html#hub-networkpolicy-internamespaceaccesslabels
• https://z2jh.jupyter.org/en/stable/administrator/security.html#introduction-to-the-chart-s-four-network-policies

[gardleopard]

@sgibson91 I did not know about that and those can in fact make it possible to scrape as well. That solution is to my understanding more cluster wide than targeted to the scraping pod.

[sgibson91]

That solution is to my understanding more cluster wide than targeted to the scraping pod.

Network access is only granted to pods in other namespaces that have the corresponding label, so you could still restrict it to just the scraping pod.

[GeorgianaElena]

Thank you very much for this PR @gardleopard ✨ Documenting this is very important.

I've left a few suggestions recommending documenting the interNamespaceAccessLabels approach too as the native solution since it's more straightforward and the explicit ingress rule as the alternative.

I believe that once we'll have both documented, we can merge the PR 🚀

[GeorgianaElena]

I believe the annotations can be omitted as they are already the default

Suggested change

	hub:
	annotations:
	prometheus.io/scrape: "true"
	prometheus.io/path: "/hub/metrics"
	prometheus.io/port: "8081"
	networkPolicy:
	ingress:
	- from:
	- namespaceSelector:
	matchLabels:
	# namespace where your prometheus or vmagent is running
	name: victoriametrics
	- podSelector:
	matchLabels:
	# a valid selector for the pod that needs to reach jupyterhub
	app.kubernetes.io/instance: vmagent
	hub:
	networkPolicy:
	interNamespaceAccessLabels: accept

[GeorgianaElena]

I believe we should recommend the interNamespaceAccessLabels because it is more straightforward.

Suggested change

Network policy needs to be modified in order for prometheus or vmagent to be able to reach the metrics endpoint. Working example configuration is as follows:

Network policy needs to be modified in order for prometheus or vmagent to be able to reach the metrics endpoint. The recommended way is by setting [interNamespaceAccessLabels=accept](https://z2jh.jupyter.org/en/stable/resources/reference.html#hub-networkpolicy-internamespaceaccesslabels). This makes the hub pod's associated NetworkPolicy accept ingress from pods in other namespaces that have specific access labels.```

[GeorgianaElena]

Suggested change

	```
	```
	And then set [`prometheus.server.podLabels`](https://github.com/prometheus-community/helm-charts/blob/0c7bf42ac2265d13845ffe0c499d16e6b8cdedea/charts/prometheus/values.yaml#L554) to be `hub.jupyter.org/network-access-hub: "true"` to enable prometheus to reach the hub.
	Alternatively you can also set an explicit ingress rule to allow the prometheus or vmagent pod to reach the hub pod to scrape metrics.
	```yaml
	hub:
	networkPolicy:
	ingress:
	- from:
	- namespaceSelector:
	matchLabels:
	# namespace where your prometheus or vmagent is running
	name: <namespace>
	- podSelector:
	matchLabels:
	# a valid selector for the pod that needs to reach jupyterhub
	app.kubernetes.io/instance: vmagent
	```

[GeorgianaElena]

I suggest removing these from here, as it's hard to maintain the info in two places and the documentation is the source of truth already.

Suggested change

	# To open for vmagent to scrape metrics from the hub pod, add the following
	# to the ingress list:
	# ingress:
	# - from:
	# - namespaceSelector:
	# matchLabels:
	# name: victoriametrics
	# - podSelector:
	# matchLabels:
	# app.kubernetes.io/instance: vmagent