EMS Application Metrics Architecture ¶

This document outlines the observability architecture of the EMS system, focusing on custom Prometheus metrics. It explains how metrics are defined, exported, and scraped.

Overview ¶

The system uses Prometheus for collecting and exposing application metrics. The primary entry point for scraping is the /metrics/processing API endpoint.

1. API Endpoint ¶

The entry point for retrieving internal metrics is defined in keep/api/routes/metrics.py. Endpoint: [GET] /metrics/processing

This endpoint collects all multi-process metrics from the application context.

@router.get("/metrics/processing", include_in_schema=False)
async def get_processing_metrics(
    request: Request,
    ...

By default, these metrics use the prometheus_client and support a multiprocess environment using the CollectorRegistry and MultiProcessCollector.

2. Metrics Definitions ¶

All custom Prometheus metrics are centrally defined in the following file:

keephq/keep/api/core/metrics.py

This file creates various standard Prometheus metrics (Counters, Gauges, Summaries, Histograms), utilizing prefix identifiers like keep_ and keep_workflows_. Examples of collected stats include:

• Events:
  • keep_events_in_total (Counter)
  • keep_events_processed_total (Counter)
  • keep_events_error_total (Counter)
  • keep_processing_time_seconds (Summary)
  • keep_running_tasks_current (Gauge)
• Workflows:
  • keep_workflows_executions_total (Counter)
  • keep_workflows_execution_errors_total (Counter)
  • keep_workflows_execution_status_total (Counter)
  • keep_workflows_execution_duration_seconds (Histogram)
  • keep_workflows_running (Gauge)

3. Exposing Metrics to Prometheus (PodMonitor) ¶

To scrape the metrics from Kubernetes running workloads, a Prometheus PodMonitor is used. This allows Prometheus to automatically discover and scrape the EMS backend target.

The configuration file is located at: backend-podmonitor.yaml

PodMonitor Details ¶

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: {{ include "keep.fullname" . }}-backend-pod-monitor
  labels:
    metrics: {{ include "keep.fullname" . }}-backend-metrics
spec:
  selector:
    matchLabels:
      app.kubernetes.io/component: backend
  podMetricsEndpoints:
    - path: /metrics/processing
      port: http
      basicAuth:
        username:
          name: {{ include "keep.fullname" . }}-app-secrets
          key: basic_auth_username
        password:
          name: {{ include "keep.fullname" . }}-app-secrets
          key: basic_auth_password
      metricRelabelings:
        - sourceLabels: [__name__]
          action: keep
          regex: '^({{ join "|" .Values.metricsAllowList.keepMetrics }})$'

• Path: Uses /metrics/processing.
• Authentication: The PodMonitor is configured with basicAuth referencing standard application secrets, securing the endpoint from unauthenticated access.
• Relabeling/Whitelist: The endpoint employs metricRelabelings with a regex matching against .Values.metricsAllowList.keepMetrics. This restricts the scraped data to explicitly allowed metrics only.

4. Adding New Metrics ¶

If you wish to introduce a new custom metric to EMS, follow these steps:

1. Define the metric in code: Update keephq/keep/api/core/metrics.py to instantiate your new metric object (e.g., Counter(...), Gauge(...)). Instrument your application code to update this metric.

2. Allow-list the metric for Prometheus scraping: Add the exact metric name to the keep metrics allowlist in the helm values.

   Location: ems-helm-values.yaml

   (e.g., if you added keep_workflows_retries_total, ensure keep_workflows_retries_total is appended to the metricsAllowList.keepMetrics array so the PodMonitor successfully ingests it).