EMS Observability Architecture

This page explains how Keep collects metrics, logs, and traces in the local OpenTelemetry stack (docker-compose-with-otel.yaml). It also explains how to extend custom metrics and custom traces at code and collector levels.

Architecture overview ¶

The current stack uses these components:

• keep-backend-dev emits OpenTelemetry telemetry
• otel-collector receives and routes telemetry by signal
• prometheus stores and serves metrics (Kubed Platform)
• tempo stores and serves traces (Self Hosted)
• loki stores and serves logs (Kubed Platform)
• grafana queries Prometheus, Tempo, and Loki (LMA)

flowchart LR
  A[keep-backend-dev\nOTel SDK in keep/api/observability.py] --> B[otel-collector\notlp receiver]
  B --> C[metrics pipeline] --> D[prometheus exporter :9100] --> E[prometheus]
  B --> F[traces pipeline] --> G[otlp exporter -> tempo:4317] --> H[tempo]
  B --> I[logs pipeline] --> J[loki exporter -> /loki/api/v1/push] --> K[loki]
  E --> L[grafana datasource: Prometheus]
  H --> M[grafana datasource: Tempo]
  K --> N[grafana datasource: Loki]

Metrics flow ¶

Keep creates metrics in the OpenTelemetry Python SDK and exports them to the collector over OTLP gRPC.

flowchart TD
  A[keep/api/api.py\nInstrumentator metric namespace: keep] --> B[keep/api/observability.py\nMeterProvider + OTLPMetricExporter]
  B -->|OTLP gRPC :4317| C[otel-collector receiver: otlp]
  C --> D["metrics pipeline\nreceivers:[otlp]\nprocessors:[]\nexporters:[prometheus]"]
  D --> E[collector prometheus endpoint :9100]
  E --> F[prometheus scrape job: wordpress]
  F --> G[grafana prometheus datasource]

  T1[tempo metrics_generator\nservice-graphs/span-metrics] -->|remote_write| F

Key code and config:

• App metrics instrumentation:
  • keep/api/api.py (KEEP_METRICS, Instrumentator(..., metric_namespace="keep"))
  • keep/api/observability.py (MeterProvider, OTLPMetricExporter)
• App endpoint env:
  • docker-compose-with-otel.yaml OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=http://otel-collector:4317
  • docker-compose-with-otel.yaml METRIC_OTEL_ENABLED=true
• Collector metrics pipeline:
  • otel-shared/otel-collector-config.yaml (service.pipelines.metrics)
  • Prometheus exporter endpoint 0.0.0.0:9100
• Prometheus scrape:
  • otel-shared/prometheus.yaml (job_name: 'wordpress', target otel-collector:9100)
• Tempo generated metrics:
  • otel-shared/tempo.yaml (metrics_generator, remote write to Prometheus)

Trace flow ¶

Keep creates spans through FastAPI and Requests instrumentations and through manual spans, then exports to collector via OTLP HTTP.

flowchart TD
  A[keep/api/observability.py\nTracerProvider + BatchSpanProcessor] --> B[FastAPIInstrumentor + RequestsInstrumentor]
  B -->|OTLP HTTP :4318 /v1/traces| C[otel-collector receiver: otlp]
  C --> D["traces pipeline\nprocessors:[memory_limiter,batch]"]
  D -->|otlp exporter| E[tempo distributor receiver :4317]
  E --> F[tempo ingester + local storage\n/tmp/tempo/wal + /tmp/tempo/blocks]
  F --> G[tempo query API :3200]
  G --> H[grafana tempo datasource]

Key code and config:

• App trace setup:
  • keep/api/observability.py (TracerProvider, BatchSpanProcessor)
  • keep/api/observability.py (FastAPIInstrumentor, RequestsInstrumentor)
• App endpoint env:
  • docker-compose-with-otel.yaml OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://otel-collector:4318/v1/traces
• Collector traces pipeline:
  • otel-shared/otel-collector-config.yaml (service.pipelines.traces)
  • OTLP exporter to tempo:4317
• Tempo receiver and storage:
  • otel-shared/tempo.yaml (OTLP receiver on 0.0.0.0:4317/4318)

Log flow ¶

Keep exports logs through OpenTelemetry logs SDK and collector sends them to Loki.

flowchart TD
  A[python logging + LoggingInstrumentor] --> B[LoggerProvider + BatchLogRecordProcessor]
  B -->|OTLP HTTP :4318 /v1/logs| C[otel-collector receiver: otlp]
  C --> D["logs pipeline\nprocessors:[memory_limiter,batch,attributes]"]
  D --> E[loki exporter\nhttp://loki:3100/loki/api/v1/push]
  E --> F[grafana loki datasource]

  V1[vector docker_logs source] -. configured path, not active in current podman permissions .-> E

Key code and config:

• App logs setup:
  • keep/api/observability.py (LoggerProvider, BatchLogRecordProcessor)
  • keep/api/observability.py (LoggingHandler(level=logging.INFO, ...))
  • keep/api/observability.py (LoggingInstrumentor().instrument())
• App endpoint env:
  • docker-compose-with-otel.yaml OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=http://otel-collector:4318/v1/logs
• Collector logs pipeline:
  • otel-shared/otel-collector-config.yaml (service.pipelines.logs)
  • attributes processor adds Loki attributes (loki.attribute.labels, loki.format)
• Vector alternative path:
  • otel-shared/vector.toml

Where telemetry initialization happens ¶

Both API and ARQ worker initialize observability when KEEP_OTEL_ENABLED=true:

• keep/api/api.py -> keep.api.observability.setup(app)
• keep/api/arq_worker_gunicorn.py -> keep.api.observability.setup(app)

How to extend custom metrics ¶

Use this path when you need domain metrics like workflow counts, provider latency, or alert classification rates.

Step 1: create a meter and instruments in backend code ¶

Add code in the module where the business event happens. Keep metric names and attributes stable.

from opentelemetry import metrics

meter = metrics.get_meter("keep.custom", version="1.0.0")
workflow_run_counter = meter.create_counter(
    name="keep_workflow_runs_total",
    description="Total workflow runs",
    unit="1",
)
workflow_duration = meter.create_histogram(
    name="keep_workflow_duration_seconds",
    description="Workflow run duration",
    unit="s",
)

# usage
workflow_run_counter.add(1, {"tenant_id": tenant_id, "status": "success"})
workflow_duration.record(elapsed_seconds, {"tenant_id": tenant_id})

Step 2: validate endpoint and exporter settings ¶

Confirm these are set for the running backend container:

• METRIC_OTEL_ENABLED=true
• OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=http://otel-collector:4317

Step 3: expose and query in Prometheus and Grafana ¶

No collector change is needed for basic passthrough because service.pipelines.metrics already exports to Prometheus.

Query in Grafana with PromQL, for example:

sum(rate(keep_workflow_runs_total[5m])) by (status)

Step 4: add metric processing only if needed ¶

If you need filtering, renaming, or attribute normalization, add processors in otel-shared/otel-collector-config.yaml and include them in service.pipelines.metrics.processors.

How to extend custom traces ¶

Use this path when you need to trace specific business operations in more detail.

Step 1: create manual spans around business logic ¶

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("workflow.execute") as span:
    span.set_attribute("keep.tenant_id", tenant_id)
    span.set_attribute("keep.workflow_id", workflow_id)
    try:
        execute_workflow()
        span.set_attribute("keep.result", "success")
    except Exception as exc:
        span.record_exception(exc)
        span.set_attribute("keep.result", "error")
        raise

Step 2: use semantic attributes and low-cardinality tags ¶

Use consistent keys and avoid high-cardinality values where possible:

• Prefer IDs, enums, booleans
• Avoid full SQL, raw payloads, long free-form strings

Step 3: verify routing and storage ¶

The existing path is already wired:

• App -> OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
• Collector service.pipelines.traces
• Tempo OTLP receiver (tempo:4317)

Step 4: add collector processors for trace policy ¶

When you need advanced behavior (sampling, redaction, enrichment), add processors to collector and include them in the traces pipeline before export.

Typical examples:

• tail_sampling for cost control
• attributes for sanitization or normalization
• filter to drop noise

Troubleshooting quick checks ¶

Use these checks to isolate issues by stage:

1. App emits telemetry:
   • Check keep-backend-dev logs for OTLP endpoint lines.
2. Collector receives and exports:
   • Check otel-collector logs for exporter errors.
3. Backend stores telemetry:
   • Tempo: query tags API
   • Loki: query labels API
   • Prometheus: query metric names

Next steps ¶

If you want, add a dedicated collector profile for production-like behavior with sampling and stricter attribute controls, then keep this current profile as the local debug profile.

Hands-on example: custom metrics and custom trace ¶

This repository now includes a runnable example endpoint that emits custom metrics and a custom trace span:

• Route: GET /observability-examples/custom
• Code: keep/api/routes/observability_examples.py
• Feature toggle: KEEP_OBSERVABILITY_EXAMPLE_ENABLED (default true, set false to disable)

1. Generate telemetry ¶

Call the endpoint a few times to generate both success and error samples:

curl "http://localhost:8080/observability-examples/custom?tenant=demo&work_ms=180"
curl "http://localhost:8080/observability-examples/custom?tenant=demo&work_ms=320"
curl "http://localhost:8080/observability-examples/custom?tenant=demo&work_ms=120&fail=true"

The endpoint emits these custom metrics:

• keephq_demo_requests
• keephq_demo_failures
• keephq_demo_duration_ms

Because the collector Prometheus exporter is configured with namespace: keep, the names you query in Prometheus and Grafana become:

• keep_keephq_demo_requests
• keep_keephq_demo_failures
• keep_keephq_demo_duration_ms_bucket|sum|count

It also emits this custom span name:

• keephq.demo.custom_trace

2. Verify custom metrics in Grafana ¶

Open Grafana Explore and choose Prometheus datasource.

Use these PromQL queries:

sum(keep_keephq_demo_requests) by (status)

sum(keep_keephq_demo_failures) by (tenant)

histogram_quantile(0.95, sum(rate(keep_keephq_demo_duration_ms_bucket[5m])) by (le))

To discover names quickly:

curl -s 'http://localhost:9090/api/v1/label/__name__/values' | grep demo

3. Verify custom traces in Grafana ¶

Open Grafana Explore and choose Tempo datasource.

Use this TraceQL query:

{ name = "keephq.demo.custom_trace" }

You can also filter by your custom span attributes:

{ name = "keephq.demo.custom_trace" && span.keep.demo.tenant = "demo" }

4. Troubleshooting this example ¶

If you do not see data:

1. Confirm the endpoint is not disabled via KEEP_OBSERVABILITY_EXAMPLE_ENABLED=false.
2. Confirm OTLP endpoints are set in docker-compose-with-otel.yaml.
3. Check otel-collector logs for exporter errors.
4. Re-run the endpoint to generate fresh samples.

Reference ¶

Code files mentioned in this page are all under Monitoring/keephq.