Scrape Custom metrics with Prometheus

Annotations

Annotations on pods allow Prometheus to automatically retrive metrics from the configure endpoints. The following annotations can be configured :

• prometheus.io/scrape: The default configuration will scrape all pods and, if set to false, this annotation will exclude the pod from the scraping process.
• prometheus.io/path: If the metrics path is not /metrics, define it with this annotation.
• prometheus.io/port: Scrape the pod on the indicated port instead of the pod’s declared ports.

These annotations need to be part of the pod metadata. They will have no effect if set on other objects such as Services or DaemonSets.

See example below:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: prometheus-example-app
  name: prometheus-example-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: prometheus-example-app
  template:
    metadata:
      labels:
        app.kubernetes.io/name: prometheus-example-app
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
        prometheus.io/path: "/metrics"
        prometheus.io/scrape_interval: "60s"
    spec:
      containers:
      - name: prometheus-example-app
        image: quay.io/brancz/prometheus-example-app:v0.3.0
        ports:
        - name: web
          containerPort: 8080

ServiceMonitor and PodMonitor

Prometheus operator provides a set od custom resources which can be configured to scrape Services (opens new window) and Pods (opens new window). The same example as above can then be re-written as follows:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: prometheus-example-app
  name: prometheus-example-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: prometheus-example-app
  template:
    metadata:
      labels:
        app.kubernetes.io/name: prometheus-example-app
    spec:
      containers:
      - name: prometheus-example-app
        image: quay.io/brancz/prometheus-example-app:v0.3.0
        ports:
        - name: web
          containerPort: 8080
---
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  namespace: prometheus-demo
  labels:
    app.kubernetes.io/name: prometheus-example-app
  name: prometheus-pod-monitor
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: prometheus-example-app
  podMetricsEndpoints:
  - port: web