grafana显示无数据，prometheus打不开
背景
重新使用kube-promethues安装之后，发现prometheus打不开，grafana能打开，但是显示没有数据
k8s版本：1.24.1
kube-promethues版本：0.12.0
问题探索
查看prometheus日志发现， promethues-operator未能成功启动，显示16行有错误。
k -n monitoring logs prometheus-operator-7bb9877c54-klzj7
问题解决
使用yaml.cn查看格式
发现确实有缩进不正确,错误案例：
- job_name: 'node-exporter'
static_configs:
- targets:
- 192.168.0.123:20001
- 192.168.0.124:20001
- 192.168.0.153:20001
- 192.168.0.154:20001
- 192.168.0.72:20001
- 192.168.0.199:20001
- 192.168.0.176:20001
- 192.168.0.147:20001
- job_name: 'redis'
static_configs:
- targets: ['192.168.0.153:9121']
- job_name: "HD_cadvisor"
static_configs:
- targets: ["192.168.0.72:20002"]
显示错误如下所示
Error: bad indentation of a mapping entry (16:4)
13 | static_configs:
14 | - targets: ['192.168.0.153:9121']
15 | - job_name: "HD_cadvisor"
16 | static_configs:
---------^
17 | - targets: ["192.168.0.72:20 ...
更正之后，如下：
- job_name: 'node-exporter'
static_configs:
- targets:
- 192.168.0.123:20001
- 192.168.0.124:20001
- 192.168.0.153:20001
- 192.168.0.154:20001
- 192.168.0.72:20001
- 192.168.0.199:20001
- 192.168.0.176:20001
- 192.168.0.147:20001
- job_name: 'redis'
static_configs:
- targets: ['192.168.0.153:9121']
- job_name: "HD_cadvisor"
static_configs:
- targets: ["192.168.0.72:20002"]
此时显示正常,可以正确解析出json：
[
{
"job_name": "node-exporter",
"static_configs": [
{
"targets": [
"192.168.0.123:20001",
"192.168.0.124:20001",
"192.168.0.153:20001",
"192.168.0.154:20001",
"192.168.0.72:20001",
"192.168.0.199:20001",
"192.168.0.176:20001",
"192.168.0.147:20001"
]
}
]
},
{
"job_name": "redis",
"static_configs": [
{
"targets": [
"192.168.0.153:9121"
]
}
]
},
{
"job_name": "HD_cadvisor",
"static_configs": [
{
"targets": [
"192.168.0.72:20002"
]
}
]
}
]
清理掉重新安装，即可解决
#2024-11-24 15:26:27 root
#使用这个文件，是可以额外挂载外部其他node_exporter
vim prometheus-additional.yaml
kubectl -n monitoring logs prometheus-operator-7bb9877c54-klzj7
kubectl delete -f manifests/
kubectl delete secrets -n monitoring additional-scrape-configs
kubectl create secret generic additional-scrape-configs --from-file=prometheus-additional.yaml -n monitoring
kubectl apply -f manifests/
等待一会儿，发现已经有了数据，一切又恢复了正常。