How to expose Prometheus metrics from your application ¶

Before you can monitor your services using Prometheus, you need to add instrumentation to their code using one of the Prometheus client libraries. These implement the Prometheus metric types.

Step 0: Make your namespace eligible for metrics scrapping ¶

Please make sure the label infra.webex.com/app-group={app-group} is existing on your namespace which will expose metrics. Please use kubectl to add required label:

kubectl label ns {your-namespace} infra.webex.com/app-group={your-app-group}

Step 1: Instrument your application ¶

Choose a Prometheus client library that matches the language in which your application is written. This lets you define and expose internal metrics via an HTTP endpoint on your application’s instance. Refer to the upstream Prometheus documentation on how to implement this.

Step 2: Expose your metrics ¶

After implementing the API for metrics collection, the recommended approach for exposing metrics in a Kubernetes environment is through ServiceMonitor or PodMonitor, rather than using annotations. These methods offer more flexibility and better integration with the Prometheus Operator.

Using ServiceMonitor ¶

A ServiceMonitor is a resource in Kubernetes that specifies how groups of services should be monitored. The Prometheus Operator uses this resource to identify and scrape metrics from the specified services. This method is more dynamic compared to annotations and allows for more complex configurations.

To set up a ServiceMonitor, you need to define it in your Kubernetes cluster. For instance, consider the following example where an application is exposed via a Kubernetes Service:

kind: Service
apiVersion: v1
metadata:
  name: example-app
  labels:
    app: example-app
spec:
  selector:
    app: example-app
  ports:
  - name: web
    port: 8080

You can expose the metrics for this service with the following ServiceMonitor definition:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: example-app
  labels:
    team: frontend
spec:
  selector:
    matchLabels:
      app: example-app
  endpoints:
  - port: web

Using PodMonitor ¶

Alternatively, PodMonitor is another resource that offers a similar function to ServiceMonitor but is specifically designed for targeting Pods directly. This is particularly useful when you want to monitor specific Pods rather than a service that might encompass a broader range of Pods.

To use PodMonitor, first ensure your Pods are correctly labeled. Then, define a PodMonitor resource that specifies which Pods to monitor. For example, if you have the following deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: example-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: example-app
  template:
    metadata:
      labels:
        app: example-app
    spec:
      containers:
      - name: example-app
        image: fabxc/instrumented_app
        ports:
        - name: web
          containerPort: 8080

You can create a PodMonitor like this:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: example-app-podmonitor
  labels:
    team: frontend
spec:
  podMetricsEndpoints:
  - port: web
  selector:
    matchLabels:
      app: example-app