prometheus+grafana安装和配置

>i 这里我们采用 docker-compose 方式安装

# 1.前期准备

```bash
# 创建目录
mkdir -p /opt/jonnyan404/prometheus        # prometheus主目录
mkdir -p /opt/jonnyan404/prometheus-tsdb   # prometheus数据持久化目录
mkdir -p /opt/jonnyan404/prometheus/rules  # prometheus告警规则目录
mkdir -p /opt/jonnyan404/prometheus/target # 被监控的机器列表,由于我们采用文件发现,需要手动更新文件.
mkdir -p /opt/jonnyan404/alertmanager      # alertmanager配置文件目录
mkdir -p /opt/jonnyan404/grafana-data      # grafana数据持久化目录
mkdir -p /opt/jonnyan404/dingding          # 综合告警配置和数据持久化目录
# 赋权
chmod 777 /opt/jonnyan404/prometheus-tsdb
chmod 777 /opt/jonnyan404/grafana-data
```

# 1.编辑必要的配置文件
## 1.1 prometheus配置

- 基于文件发现的prometheus配置文件
- 配置文件参数:<https://prometheus.io/docs/prometheus/latest/configuration/configuration/>

- vim  /opt/prometheus/prometheus.yml

```bash
# my global config
global:
  scrape_interval:     15s # 采集数据间隔
  evaluation_interval: 15s # 告警规则表执行间隔

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
       - alert:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
   - "/etc/prometheus/rules/*.yml"
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
 # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
 - job_name: 'prometheus'

# metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

static_configs:
    - targets: ['localhost:9090']

- job_name: 'aws_ec2'
  #  scrape_interval: 10s
    file_sd_configs:
    - refresh_interval: 30s # 30s重载配置文件
      files:
      - /etc/prometheus/target/*.yml
      - /etc/prometheus/target/*.json

```

## 1.2 alertmanager配置文件

- vim /opt/jonnyan404/alertmanager/alertmanager.yml

```yaml
global:
  resolve_timeout: 5m

route:
  group_by: ['alertname']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h #未解决的告警间隔
  receiver: 'default'
receivers:
- name: 'default'
  webhook_configs:
          - url: 'http://主机IP:9000/prometheusalert?type=dd&tpl=dingding&ddurl=钉钉告警地址' # 这里依赖3.3prometheusalert配置,请注意!!!
inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'dev', 'instance']

```

## 1.3 配置dingding告警

```bash
cd /opt/jonnyan404/dingding
git clone https://github.com/feiyu563/PrometheusAlert.git
cp PrometheusAlert/conf/app-example.conf app.conf
```

# 2. 配置docker-compose

>i 集成 prometheus/alertmanager/dingding/grafana

- vim docker-compose.yml

```yaml

version: "3"
services:
  promethtus:
    image: prom/prometheus
    container_name: prometheus
    restart: always
    volumes:
      - /opt/jonnyan404/prometheus:/etc/prometheus
      - /opt/jonnyan404/prometheus-tsdb:/prometheus-tsdb
    environment:
      - TZ=Asia/Shanghai
    ports:
      - "9090:9090"
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus-tsdb'
      - '--storage.tsdb.retention.time=90d' #数据保存时间,默认15天
    depends_on:
      - alert

alert:
    image: prom/alertmanager
    container_name: alert
    restart: always
    volumes:
      - /opt/jonnyan404/alertmanager:/etc/alertmanager
    environment:
      - TZ=Asia/Shanghai
    ports:
      - "9093:9093"
    command:
      - "--config.file=/etc/alertmanager/alertmanager.yml"
      - "--web.external-url=http://x.x.x.x:9093/" #外网IP,裸机搭建的可能需要

grafana:
    image: grafana/grafana:latest
    container_name: grafana
    restart: always
    environment:
      - TZ=Asia/Shanghai
      - LANG=zh_CN.UTF-8
    ports:
      - "3000:3000"
    volumes:
      - /opt/jonnyan404/grafana-data:/var/lib/grafana

dingding:
    image: feiyu563/prometheus-alert:latest
    container_name: dingding
    restart: always
    environment:
      - TZ=Asia/Shanghai
    ports:
      - "9000:8080"
    volumes:
      - /opt/jonnyan404/dingding:/app/conf
      - /opt/jonnyan404/dingding:/app/db

# Volumes for persisting data, see https://docs.docker.com/engine/admin/volumes/volumes/
#volumes:
#  prometeus-tsdb:
#    driver: local
#  grafana_data:
#    driver: local
#  alertmanager_journal:
#    driver: local

```

# 3. 启动
1. 执行 `docker-compose up -d` 启动
2. 打开 `http://ip:3000` 访问 grafana 并配置数据源

## 3.3 prometheusalert插件钉钉告警配置

>w 这里如果有疑问,请参考:<https://github.com/feiyu563/PrometheusAlert/blob/master/doc/readme/customtpl.md>

-  打开 `http://ip:9000` 访问prometheusalert界面,点击上方菜单`alerttemplate`-->`添加模板`,配置自定义dingding告警模板.以下是模板内容:

```go
{{ $var := .externalURL}}{{ range $k,$v:=.alerts }}
{{if eq $v.status "resolved"}}
#### [Prometheus恢复信息]({{$v.generatorURL}})

##### 告警名称：[{{$v.labels.alertname}}]({{$var}})
##### 告警级别：{{$v.labels.severity}}
##### 触发时间：{{$v.startsAt}}
##### 结束时间：{{$v.endsAt}}
##### 主机名称：{{$v.labels.nodename}}

**{{$v.annotations.summary}}**
[点击打开grafana]($v.annotations.grafana)
{{else}}
#### [Prometheus告警信息]({{$v.generatorURL}})

##### 告警名称：[{{$v.labels.alertname}}]({{$var}})
##### 告警级别：{{$v.labels.severity}}
##### 触发时间：{{$v.startsAt}}
##### 主机名称：{{$v.labels.nodename}}

**{{$v.annotations.summary}}**
[点击打开grafana]($v.annotations.grafana)
{{end}}
{{ end }}

```

这样就把基础程序都运行起来了,剩下就是安装配置各种exporter了.

# Reference Link
- 官方配置文件示例:<https://github.com/prometheus/prometheus/tree/main/documentation/examples>
- 三方中文手册:<https://www.prometheus.wang/>
- 官方exporter列表: <https://prometheus.io/docs/instrumenting/exporters/>