Prometheus监控神技--⾃动发现配置
⼀、⾃动发现类型
：
监控某个statefulset服务的时候，我在service⽂件中定义了个EP，然后把pod的ip写死在配置⽂件中，这样，当pod重启后，IP地址变化，就监控不到数据了，这肯定是不合理的。
如果在我们的 Kubernetes 集中有了很多的 Service/Pod，那么我们都需要⼀个⼀个的去建⽴⼀个对应的 ServiceMonitor 对象来进⾏监控吗？这样岂不是也很⿇烦么？
为解决上⾯的问题，Prometheus Operator 为我们提供了⼀个额外的抓取配置的来解决这个问题，我们可以通过额外的配置来获取k8s的资源监控（pod、service、node等）。
promethues⽀持多种⽂件发现。
其中通过kubernetes_sd_configs，可以达到我们想要的⽬的，监控其各种资源。kubernetes SD 配置允许从kubernetes REST API接受搜集指标，且总是和集保持同步状态，以下任何⼀种role类型都能够配置来发现我们想要的对象，来⾃官⽹翻译的。
1、Node
Node role发现每个集中的⽬标是通过默认的kubelet的HTTP端⼝。⽬标地址默认是kubernetes如下地址中node的第⼀个地址（NodeInternalIP, NodeExternalIP,NodeLegacyHostIP, and NodeHostName.）
可⽤的meta标签有：
__meta_kubernetes_node_name: The name of the node object.
__meta_kubernetes_node_label_<labelname>: Each label from the node object.
__meta_kubernetes_node_labelpresent_<labelname>: true for each label from the node object.
__meta_kubernetes_node_annotation_<annotationname>: Each annotation from the node object.
__meta_kubernetes_node_annotationpresent_<annotationname>: true for each annotation from the node object.
__meta_kubernetes_node_address_<address_type>: The first address for each node address type, if it exists.
此外，node的实例标签将会被设置成从API server传递过来的node的name。
2、Service
service⾓⾊会为每个服务发现⼀个服务端⼝。对于⿊盒监控的服务，这个⽐较有⽤。address将会被设置成service的kubernetes DNS名称以及各⾃的服务端⼝。
Available meta labels:
__meta_kubernetes_namespace: The namespace of the service object.
__meta_kubernetes_service_annotation_<annotationname>: Each annotation from the service object.
__meta_kubernetes_service_annotationpresent_<annotationname>: "true" for each annotation of the service object.
__meta_kubernetes_service_cluster_ip: The cluster IP address of the service. (Does not apply to services of type ExternalName)
__meta_kubernetes_service_external_name: The DNS name of the service. (Applies to services of type ExternalName)
__meta_kubernetes_service_label_<labelname>: Each label from the service object.
__meta_kubernetes_service_labelpresent_<labelname>: true for each label of the service object.
__meta_kubernetes_service_name: The name of the service object.
__meta_kubernetes_service_port_name: Name of the service port for the target.
__meta_kubernetes_service_port_protocol: Protocol of the service port for the target.
3、Pod
Pod role会发现所有pods以及暴露的容器作为target。每个容器声明⼀个端⼝，⼀个单独的target就会⽣成。如果⼀个容器没有指定端⼝，通过relabel⼿动指定⼀个端⼝，⼀个port-free target容器将会⽣成。
Available meta labels:
__meta_kubernetes_namespace: The namespace of the pod object.
__meta_kubernetes_pod_name: The name of the pod object.
__meta_kubernetes_pod_ip: The pod IP of the pod object.
__meta_kubernetes_pod_label_<labelname>: Each label from the pod object.
__meta_kubernetes_pod_labelpresent_<labelname>: truefor each label from the pod object.
__meta_kubernetes_pod_annotation_<annotationname>: Each annotation from the pod object.
__meta_kubernetes_pod_annotationpresent_<annotationname>: true for each annotation from the pod object.
__meta_kubernetes_pod_container_init: true if the container is an InitContainer
__meta_kubernetes_pod_container_name: Name of the container the target address points to.
__meta_kubernetes_pod_container_port_name: Name of the container port.
__meta_kubernetes_pod_container_port_number: Number of the container port.
__meta_kubernetes_pod_container_port_protocol: Protocol of the container port.
__meta_kubernetes_pod_ready: Set to true or false for the pod's ready state.
__meta_kubernetes_pod_phase: Set to Pending, Running, Succeeded, Failed or Unknown in the lifecycle.
__meta_kubernetes_pod_node_name: The name of the node the pod is scheduled onto.
__meta_kubernetes_pod_host_ip: The current host IP of the pod object.
__meta_kubernetes_pod_uid: The UID of the pod object.
__meta_kubernetes_pod_controller_kind: Object kind of the pod controller.
__meta_kubernetes_pod_controller_name: Name of the pod controller.
4、endpoints
endpoints role从每个服务监听的endpoints发现。每个endpoint都会发现⼀个port。如果endpoint是⼀个pod，所有包含的容器不被绑定到⼀个endpoint port，也会被targets被发现。
Available meta labels:
__meta_kubernetes_namespace: The namespace of the endpoints object.
__meta_kubernetes_endpoints_name: The names of the endpoints object.
For all targets discovered directly from the endpoints list (those not additionally inferred from underlying pods), the following labels are attached:
__meta_kubernetes_endpoint_hostname: Hostname of the endpoint.
__meta_kubernetes_endpoint_node_name: Name of the node hosting the endpoint.
__meta_kubernetes_endpoint_ready: Set to true or false for the endpoint's ready state.
__meta_kubernetes_endpoint_port_name: Name of the endpoint port.
__meta_kubernetes_endpoint_port_protocol: Protocol of the endpoint port.
__meta_kubernetes_endpoint_address_target_kind: Kind of the endpoint address target.
__meta_kubernetes_endpoint_address_target_name: Name of the endpoint address target.
If the endpoints belong to a service, all labels of the role: service discovery are attached.
For all targets backed by a pod, all labels of the role: pod discovery are attached.
5、ingress
ingress role将会发现每个ingress。ingress在⿊盒监控上⽐较有⽤。address将会被设置成ingress指定的配置。
Available meta labels:
__meta_kubernetes_namespace: The namespace of the ingress object.
__meta_kubernetes_ingress_name: The name of the ingress object.
__meta_kubernetes_ingress_label_<labelname>: Each label from the ingress object.
__meta_kubernetes_ingress_labelpresent_<labelname>: true for each label from the ingress object.
__meta_kubernetes_ingress_annotation_<annotationname>: Each annotation from the ingress object.
__meta_kubernetes_ingress_annotationpresent_<annotationname>: true for each annotation from the ingress object. __meta_kubernetes_ingress_scheme: Protocol scheme of ingress, https if TLS config is set. Defaults to http.
__meta_kubernetes_ingress_path: Path from ingress spec. Defaults to /.
⼆、⾃动发现Pod配置
apiVersion: apps/v1
kind: StatefulSet
metadata:
labels:
run: jx3recipe
name: jx3recipe
annotations:
prometheus.io/scrape: "true"
spec:
selector:
matchLabels:
app: jx3recipe
serviceName: jx3recipe-service
replicas: 2
template:
metadata:
labels:
app: jx3recipe
appCluster: jx3recipe-cluster
spec:
terminationGracePeriodSeconds: 20
containers:
- image: /jx3pvp/jx3recipe:qa-latest
imagePullPolicy: Always
securityContext:
runAsUser: 1000
name: jx3recipe
lifecycle:
preStop:
exec:
command: ["kill","-s","SIGINT","1"]
volumeMounts:
- name: config-volume
mountPath: /l
subPath: l
resources:
requests:
cpu: "100m"
memory: "500Mi"
env:
- name: JX3PVP_ENV
value: "qa"
-
name: JX3PVP_RUN_MODE
value: "k8s"
- name: JX3PVP_SERVICE_ID
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: JX3PVP_LOCAL_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: JX3PVP_CONSUL_IP
value: $(CONSUL_AGENT_SERVICE_HOST)
ports:
- name: biz
containerPort: 8000
protocol: "TCP"
- name: admin
containerPort: 7000
protocol: "TCP"
volumes:
- name: config-volume
configMap:
name: app-configure-file-jx3recipe
items:
- key: l
path: l
1、创建发现规则
设定发现pod规则：⽂件名为promethues-additional.yaml
pod名称的label为jx3recipe
pod的label_appCluster匹配为 jx3recipe-cluster
pod的address为.*:7000/metrics格式
- job_name: 'kubernetes-service-pod'
kubernetes_sd_configs:
-
role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_container_name]
action: replace
target_label: jx3recipe
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- source_labels:  ["__meta_kubernetes_pod_label_appCluster"]
regex: "jx3recipe-cluster"
action: keep
- source_labels: [__address__]
action: keep
regex: '(.*):7000'
2、创建对应的Secret对象
kubectl create secret generic additional-configs --from-file=prometheus-additional.yaml -n monitoring
创建完成后，会将上⾯配置信息进⾏ base64 编码后作为 prometheus-additional.yaml 这个 key 对应的值存在：
apiVersion: v1
data:
prometheus-additional.yaml: LSBqb2JfbmFtZTogJ2t1YmVybmV0ZXMtc2VydmljZS1wb2QnCiAga3ViZXJuZXRlc19zZF9jb25maWdzOgogIC0gcm9sZTogcG9kCiAgcmVsYWJlbF9jb25maWdzOgogIC0gc291cmNlX2x kind: Secret
metadata:
creationTimestamp: "2019-09-10T09:32:22Z"
name: additional-configs
namespace: monitoring
resourceVersion: "1004681"
selfLink: /api/v1/namespaces/monitoring/secrets/additional-configs
uid: e455d657-d3ad-11e9-95b4-fa163e3c10ff
type: Opaque
  然后我们只需要在声明 prometheus 的资源对象⽂件中添加上这个额外的配置：(prometheus-prometheus.yaml)
3、promethues添加资源对象
修改prometheus-prometheus.yaml⽂件
apiVersion: s/v1
kind: Prometheus
metadata:
labels:
prometheus: k8s
name: k8s
namespace: monitoring
spec:
alerting:
alertmanagers:
- name: alertmanager-main
namespace: monitoring
port: web
baseImage: quay.io/prometheus/prometheus
nodeSelector:
beta.kubernetes.io/os: linux
replicas: 2
secrets:
- etcd-certs
resources:
requests:
memory: 400Mi
ruleSelector:
matchLabels:
prometheus: k8s
role: alert-rules
securityContext:
fsGroup: 2000
runAsNonRoot: true
runAsUser: 1000
additionalScrapeConfigs:
name: additional-configs
key: prometheus-additional.yaml
serviceAccountName: prometheus-k8s
serviceMonitorNamespaceSelector: {}
serviceMonitorSelector: {}
version: v2.5.0
  增加了下⾯这⼀段：
additionalScrapeConfigs:
name: additional-configs
key: prometheus-additional.yaml
4、应⽤配置
kubectl apply -f prometheus-prometheus.yaml
  过⼀段时间，刷新promethues上的config，将会看到下⾯红⾊框框的配置。
5、添加权限
　在 Prometheus Dashboard 的配置页⾯下⾯我们可以看到已经有了对应的的配置信息了，但是我们切换到 targets 页⾯下⾯却并没有发现对应的监控任务，查看 Prometheus 的 Pod ⽇志：
可以看到有很多错误⽇志出现，都是xxx is forbidden，这说明是 RBAC 权限的问题，通过 prometheus 资源对象的配置可以知道 Prometheus 绑定了⼀个名为 prometheus-k8s 的 ServiceAccount 对象，⽽这个对象绑
定的是⼀个名为 prometheus-k8s 的 ClusterRole：（prometheus-clusterRole.yaml）
修改为：
nodeselectorapiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: prometheus-k8s
rules:
-
apiGroups:
- ""
resources:
- nodes
- services
- endpoints
- pods
- nodes/proxy
verbs:
- get
- list
-
watch
- apiGroups:
- ""
resources:
- configmaps
- nodes/metrics
verbs:
- get
- nonResourceURLs:
- /metrics
verbs:
-
get
  更新上⾯的 ClusterRole 这个资源对象，然后重建下 Prometheus 的所有 Pod，正常就可以看到 targets 页⾯下⾯有 kubernetes-service-pod这个监控任务了：⾄此，⼀个⾃动发现pod的配置就完成了，其他资源（service、endpoint、ingress、node同样也可以通过⾃动发现的⽅式实现。）

版权声明：本站内容均来自互联网，仅供演示用，请勿用于商业和其他非法用途。如果侵犯了您的权益请与我们联系QQ:729038198，我们将在24小时内删除。