Skip to content

Commit fc580bb

Browse files
fktkrtfktkrt
authored
Add Grafana integration example (#2408)
* add basic Grafana integration setup and README.md Signed-off-by: fktkrt <fktkrt@gmail.com> * move instructions to README, fix typo Signed-off-by: fktkrt <fktkrt@gmail.com> * add prometheus as datasource and hotrod scrape_config Signed-off-by: fktkrt <fktkrt@gmail.com> * reorg into folders, add hotROD dashboard, add basic description to README Signed-off-by: fktkrt <fktkrt@gmail.com> * rework folder structure, remove duplicated file, accept suggestions Signed-off-by: fktkrt <fktkrt@gmail.com>
1 parent f2de847 commit fc580bb

File tree

6 files changed

+2416
-0
lines changed

6 files changed

+2416
-0
lines changed

‎examples/grafana-integration/README.md

+55
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
# Hot R.O.D. - Rides on Demand - Grafana integration
2+
3+
This example combines the Hot R.O.D. demo application ([examples/hotrod/](../hotrod/)) with Grafana, Loki and Prometheus integration, to demonstrate logs, metrics and traces correlation.
4+
5+
## Running via `docker-compose`
6+
7+
### Prerequisites
8+
9+
* Clone the Jaeger repository `git clone https://github.com/jaegertracing/jaeger.git`, then `cd examples/grafana-integration`
10+
11+
* All services will log to Loki via the [official Docker driver plugin](https://grafana.com/docs/loki/latest/clients/docker-driver/).
12+
Install the Loki logging plugin for Docker:
13+
14+
```bash
15+
docker plugin install \
16+
grafana/loki-docker-driver:latest \
17+
--alias loki \
18+
--grant-all-permissions
19+
```
20+
21+
### Run the services
22+
23+
`docker-compose up`
24+
25+
### Access the services
26+
* HotROD application at http://localhost:8080
27+
* Grafana UI at http://localhost:3000
28+
29+
### Explore with Loki
30+
31+
Currently the most powerful way to correlate application logs with traces can be performed via Grafana's Explore interface.
32+
33+
After setting the datasource to Loki, all the log labels become available, and can be easily filtered using [Loki's LogQL query language](https://grafana.com/docs/loki/latest/logql/).
34+
35+
For example, after selecting the compose project/service under Log labels , errors can be filtered with the following expression:
36+
37+
```
38+
{compose-project="grafana-integration"} |= "error"
39+
```
40+
41+
which will list the redis timeout events.
42+
43+
### HotROD - Metrics and logs overview dashboard
44+
45+
Since the HotROD application can expose its metrics in Prometheus' format, these can be also used during investigation.
46+
47+
This example includes a dashboard that contains a log panel for the selected services in real time. These can be also filtered by a search field, that provides `grep`-like features.
48+
49+
There are also panels to display the ratio/percentage of errors in the current timeframe.
50+
51+
Additionally, there are graphs for each service, visualing the rate of the requests and showing latency percentiles.
52+
53+
### Clean up
54+
55+
`docker-compose down`

‎examples/grafana-integration/docker-compose.yaml

+68
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
version: '2'
2+
3+
services:
4+
grafana:
5+
image: grafana/grafana:7.1.4
6+
ports:
7+
- '3000:3000'
8+
volumes:
9+
- ./grafana/datasources.yaml:/etc/grafana/provisioning/datasources/datasources.yaml
10+
- ./grafana/dashboards.yml:/etc/grafana/provisioning/dashboards/dashboards.yml
11+
- ./grafana/hotrod_metrics_logs.json:/etc/grafana/provisioning/dashboards/hotrod_metrics_logs.json
12+
logging:
13+
driver: loki
14+
options:
15+
loki-url: 'http://localhost:3100/api/prom/push'
16+
17+
loki:
18+
image: grafana/loki:master
19+
ports:
20+
- '3100:3100'
21+
command: -config.file=/etc/loki/local-config.yaml
22+
# send Loki traces to Jaeger
23+
environment:
24+
- JAEGER_AGENT_HOST=jaeger
25+
- JAEGER_AGENT_PORT=6831
26+
- JAEGER_SAMPLER_TYPE=const
27+
- JAEGER_SAMPLER_PARAM=1
28+
logging:
29+
driver: loki
30+
options:
31+
loki-url: 'http://localhost:3100/api/prom/push'
32+
33+
jaeger:
34+
image: jaegertracing/all-in-one:latest
35+
ports:
36+
- '6831:6831'
37+
- '16686:16686'
38+
logging:
39+
driver: loki
40+
options:
41+
loki-url: 'http://localhost:3100/api/prom/push'
42+
43+
hotrod:
44+
image: jaegertracing/example-hotrod:latest
45+
ports:
46+
- '8080:8080'
47+
- '8083:8083'
48+
command: ["-m","prometheus","all"]
49+
environment:
50+
- JAEGER_AGENT_HOST=jaeger
51+
- JAEGER_AGENT_PORT=6831
52+
logging:
53+
driver: loki
54+
options:
55+
loki-url: 'http://localhost:3100/api/prom/push'
56+
57+
prometheus:
58+
image: prom/prometheus:v2.19.1
59+
volumes:
60+
- ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
61+
ports:
62+
- '9090:9090'
63+
command:
64+
- --config.file=/etc/prometheus/prometheus.yml
65+
logging:
66+
driver: loki
67+
options:
68+
loki-url: 'http://localhost:3100/api/prom/push'

‎examples/grafana-integration/grafana/dashboard.yml

+11
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
apiVersion: 1
2+
3+
providers:
4+
- name: 'HotROD'
5+
orgId: 1
6+
folder: ''
7+
type: file
8+
disableDeletion: false
9+
editable: true
10+
options:
11+
path: /etc/grafana/provisioning/dashboards

‎examples/grafana-integration/grafana/datasources.yaml

+18
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
apiVersion: 1
2+
3+
datasources:
4+
- name: Loki
5+
type: loki
6+
access: proxy
7+
url: http://loki:3100
8+
editable: true
9+
isDefault: true
10+
- name: Jaeger
11+
type: jaeger
12+
access: browser
13+
url: http://jaeger:16686
14+
editable: true
15+
- name: Prometheus
16+
type: prometheus
17+
access: proxy
18+
url: http://prometheus:9090

0 commit comments

Comments
 (0)