Skip to content

Deployment

Overview

Observability services are deployed from version-controlled Helm charts and runtime configuration. Changes should be reproducible through the managed pipeline and verified through live telemetry.

CI/CD Pipeline

sequenceDiagram
    participant Dev as Engineer
    participant Repo as Source Repository
    participant CI as CI Pipeline
    participant Argo as WBX3 / Argo
    participant DFW as DFW Cluster
    participant SJC as SJC Cluster
    participant Obs as Telemetry Validation

    Dev->>Repo: Open pull request
    Repo->>CI: Run tests and validation
    CI-->>Repo: Report result
    Dev->>Repo: Merge approved change
    Repo->>Argo: Publish chart or configuration
    Argo->>DFW: Reconcile deployment
    Argo->>SJC: Reconcile deployment
    DFW->>Obs: Report runtime and telemetry health
    SJC->>Obs: Report runtime and telemetry health
    Obs-->>Dev: Confirm or roll back

Inventory changes follow a separate generation/review path and are consumed by git-sync or exporter-specific configuration.

Environments

Production services run in the documented DFW and SJC clusters. Exact development/staging environments vary by service and must be confirmed in the current deployment repository before testing or promoting a change.

Deployment Process

  1. Identify the owning repository: inventory, exporter code, chart, runtime config, dashboard, or alert definition.
  2. Confirm current deployed version and active/standby or active/active behavior.
  3. Make the smallest source-controlled change and add tests/rendered output.
  4. Review the pull request and deployment plan for both clusters.
  5. Deploy through the managed pipeline.
  6. Verify Kubernetes health, service self-metrics, target discovery/ingress, downstream data freshness, dashboards, and affected alerts.
  7. Record the version, evidence, and rollback decision.

Rollback Procedures

Define the previous chart/image/config revision before deployment. Roll back through source control and the managed pipeline. Avoid manual in-cluster edits except through an approved emergency procedure, and reconcile any emergency change immediately afterward.

After rollback, verify replica/scale state, scrape or ingress health, Mimir/Splunk freshness, dashboard queries, and notification delivery when affected.