获取软件包
最新的软件包获取地址
https://prometheus.io/download/
Prometheus
### 1、下载Prometheus $ wget https://github.com/prometheus/prometheus/releases/download/v2.6.0/prometheus-2.6.0.linux-amd64.tar.gz ### 2、解压软件包 $ tar xf prometheus-2.6.0.linux-amd64.tar.gz ### 3、配置Prometheus启动程序 ### 把解压出来的文件移动到/usr/local/目录下，并重命名为prometheu $ mv prometheus-2.6.0.linux-amd64 /usr/local/prometheus ### 生成启动脚本 $ vim /usr/lib/systemd/system/prometheus.service [Unit] Description=Prometheus: the monitoring system Documentation=http://prometheus.io/docs/ [Service] ExecStart=/usr/local/prometheus/prometheus \ --config.file=/usr/local/prometheus/prometheus.yml \ --storage.tsdb.path=/var/lib/prometheus \ --web.console.templates=/usr/local/prometheus/consoles \ --web.console.libraries=/usr/local/prometheus/console_libraries \ --web.listen-address=0.0.0.0:9090 --web.external-url= Restart=always StartLimitInterval=0 RestartSec=10 [Install] WantedBy=multi-user.target ### 创建监控数据存储目录 $ mkdir /var/lib/prometheus ### 4、启动Prometheus $ systemctl daemon-reload $ systemctl enable prometheus $ systemctl start prometheus ### 5、查看端口监听状态 ### Prometheus监听的端口为9090，启动成功后可以通过netstat命令进行查看端口的监听状态 $ netstat -antpu | grep 9090 tcp 0 0 127.0.0.1:33270 127.0.0.1:9090 ESTABLISHED 6426/prometheus tcp6 0 0 :::9090 :::* LISTEN 6426/prometheus tcp6 0 0 ::1:9090 ::1:51821 ESTABLISHED 6426/prometheus tcp6 0 0 ::1:51821 ::1:9090 ESTABLISHED 6426/prometheus tcp6 0 0 127.0.0.1:9090 127.0.0.1:33270 ESTABLISHED 6426/prometheus ### 6、通过浏览器进行访问 ### Prometheus启动成功后，可以通过浏览器访问查看状态和配置信息 http://local:9090/graph
Ceph_export
Ceph_export 需要使用Go进行编译，也可以下载已经编译好的Ceph_exporter直接使用.
### 1、安装软件Go环境 $ yum -y install golang ### 2、查看Go环境变量 $ go env GOARCH="amd64" GOBIN="" GOCACHE="/root/.cache/go-build" GOEXE="" GOFLAGS="" GOHOSTARCH="amd64" GOHOSTOS="linux" GOOS="linux" GOPATH="/root/go" GOPROXY="" GORACE="" GOROOT="/usr/lib/golang" GOTMPDIR="" GOTOOLDIR="/usr/lib/golang/pkg/tool/linux_amd64" GCCGO="gccgo" CC="gcc" CXX="g++" CGO_ENABLED="1" GOMOD="" CGO_CFLAGS="-g -O2" CGO_CPPFLAGS="" CGO_CXXFLAGS="-g -O2" CGO_FFLAGS="-g -O2" CGO_LDFLAGS="-g -O2" PKG_CONFIG="pkg-config" GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build359765015=/tmp/go-build -gno-record-gcc-switches" ### 3、设置Go环境变量 $ vim /etc/profile.d/go.sh export GOROOT=/usr/lib/golang export GOBIN=$GOROOT/bin export GOPATH=/root/go export PATH=$PATH:$GOROOT/bin:$GOPATH/bin $ source /etc/profile.d/go.sh ### 4、下载并编译Ceph_exporter $ mkdir go/src/github.com/digitalocean/ $ cd go/src/github.com/digitalocean/ $ git clone https://github.com/digitalocean/ceph_exporter $ cd ceph_exporter $ go build ### 5、创建Ceph_exporter启动程序 $ mkdir ~/go/bin/ $ cp ~/go/src/github.com/digitalocean/ceph_exporter/ceph_exporter ~/go/bin/ $ vim /usr/lib/systemd/system/ceph_exporter.service [Unit] Description=Prometheus's ceph metrics exporter [Service] User=root Group=root ExecStart=/root/go/bin/ceph_exporter [Install] WantedBy=multi-user.target Alias=ceph_exporter.service ### 6、启动Ceph_exporter $ systemctl daemon-reload $ systemctl enable ceph_exporter $ systemctl start ceph_exporter ### 7、查看端口监听状态 ### Ceph_exporter使用的是9128端口，可以通过netstat进行查看端口的监听状态 $ netstat -antpu | grep 9128 tcp6 0 0 :::9128 :::* LISTEN 6839/ceph_exporter ### 8、修改Prometheus配置 ### 把Ceph_exporter的接口添加到Prometheus的配置中 $ vim /usr/local/prometheus/prometheus.yml scrape_configs: - job_name: 'ceph' honor_labels: true static_configs: - targets: ['192.168.1.10:9128'] labels: instance: Ceph测试集群 ### 9、重启Prometheus进程 $ systemctl restart prometheus ### 10、浏览器访问验证 http://local:9090/graph
Grafana
1、下载软件包
不同系统的最新软件包可以在Grafana的官网获取下载地址https://grafana.com/grafana/download
$ wget https://dl.grafana.com/oss/release/grafana-5.4.3-1.x86_64.rpm
2、安装Grafana
$ yum -y install grafana-5.4.3-1.x86_64.rpm
3、启动Grafana
$ systemctl enable grafana-server $ systemctl start grafana-server
4、查看端口监听状态
Grafana监听端口为3000，可以使用netstat查看监听状态
$ netstat -antpu | grep 3000
tcp6 0 0 :::3000 :::* LISTEN 7147/grafana-server
5、浏览器访问登录
访问地址为http://$IP:3000，初始用户名和密码均为admin，首次登录后会提示设置新的密码
6、配置Dashboard
点击Add data source添加数据源
选择Prometheus
URL地址为Prometheus的访问地址http://$IP:9090
导入Dashboard，模板的编号为917，如果无法连接互联网，也可以在Grafana的官网下载模板后手动导入https://grafana.com/dashboards/917
查看监控状态
AlertManager
### 1、安装Alertmanager
$ wget https://github.com/prometheus/alertmanager/releases/download/v0.16.0/alertmanager-0.16.0.linux-amd64.tar.gz
$ tar xf alertmanager-0.16.0-alpha.0.linux-amd64.tar.gz
$ cd alertmanager-0.16.0-alpha.0.linux-amd64
$ cp alertmanager amtool /usr/bin/
$ cp alertmanager.yml /usr/local/prometheus/
### 2、生成启动程序
$ vim /usr/lib/systemd/system/alertmanager.service
[Unit]
Description=Prometheus: the alerting system
Documentation=http://prometheus.io/docs/
After=prometheus.service
[Service]
ExecStart=/usr/bin/alertmanager --config.file=/usr/local/prometheus/alertmanager.yml
Restart=always
StartLimitInterval=0
RestartSec=10
[Install]
WantedBy=multi-user.target
### 3、启动Alertmanager
$ systemctl enable alertmanager
$ systemctl start alertmanager
### 4、查看端口监听状态
Alertmanager的监听端口为9093，可以使用netstat查看端口监听状态
$ netstat -antpu | grep 9093
tcp6 0 0 :::9093 :::* LISTEN 7381/alertmanager
### 5、配置Prometheus，添加Alertmanager端点
$ vim /usr/local/prometheus/prometheus.yml
alerting:
alertmanagers:
- static_configs:
- targets: ["192.168.1.10:9093"]
### 6、重启Prometheus
$ systemctl restart prometheus
### 配置钉钉告警
### 1、配置webhook
$ mkdir -p /usr/lib/golang/src/github.com/timonwong/
$ cd /usr/lib/golang/src/github.com/timonwong/
$ git clone https://github.com/timonwong/prometheus-webhook-dingtalk.git
$ cd prometheus-webhook-dingtalk
$ make
$ nohup ./prometheus-webhook-dingtalk --ding.profile="webhook=https://oapi.dingtalk.com/robot/send?access_token=8fe12c1a58b0769d7fcbf6ebf3bcd2cfcba825f2c45b4b39055890fd705df543" &> /var/log/dingding.log &
### 2、添加webhook告警
$ vim /usr/local/prometheus/alertmanager.yml
global:
resolve_timeout: 5m
route:
group_by: ['alertname']
group_wait: 10s
group_interval: 10s
repeat_interval: 1h
receiver: 'web.hook'
receivers:
- name: 'web.hook'
webhook_configs:
- url: 'http://192.168.1.10:8060/dingtalk/webhook/send'
inhibit_rules:
- source_match:
severity: 'critical'
target_match:
severity: 'warning'
equal: ['alertname', 'dev', 'instance']
### 3、添加告警规则文件
$ vim /usr/local/prometheus/prometheus.yml
rule_files:
- /usr/local/prometheus/ceph.yml
### 4、配置告警规则
$ vim /usr/local/prometheus/ceph.yml
groups:
- name: ceph-rule
rules:
- alert: Ceph OSD Down
expr: ceph_osd_down > 0
for: 2m
labels:
product: Ceph测试集群
annotations:
Warn: "{{$labels.instance}}: 有{{ $value }}个OSD挂掉了"
Description: "{{$labels.instance}}:{{ $labels.osd }}当前状态为{{ $labels.status }}"
- alert: 集群空间使用率
expr: ceph_cluster_used_bytes / ceph_cluster_capacity_bytes * 100 > 80
for: 2m
labels:
product: Ceph测试集群
annotations:
Warn: "{{$labels.instance}}:集群空间不足"
Description: "{{$labels.instance}}:当前空间使用率为{{ $value }}"
### 5、重启进程使配置生效
$ systemctl restart alertmanager
$ systemctl restart prometheus.service
### 6、钉钉验证
### 停掉一个OSD后，钉钉收到告警
### 重新启动后收到恢复通知