Skip to content

Exporter Update Deployment: Method of Procedure

This procedure ports the old Deployment MoP entry and expands it into a usable rollout and rollback checklist. The detailed internal source remains the WxCDI Custom Exporter Kubed Deployment Process.

Scope

Use this MoP for a version or configuration update to a Prometheus exporter deployed in WxCDI Webex Kubed. Adjust it when the change affects shared Prometheus, ingress, credentials, inventory generation, or multiple exporter families.

Required Change Record

Document:

  • exporter/service and operational reason;
  • source commit, image tag, and image digest;
  • chart version and configuration pull request;
  • namespaces, customer segments, clusters, and regions affected;
  • expected telemetry or behavior change;
  • maintenance/alert handling requirements;
  • validation owner and evidence location;
  • previous known-good image/chart/config versions;
  • abort and rollback criteria.

Pre-Deployment Checks

  1. Confirm the source change passed tests, review, and security scanning.
  2. Confirm Harness published the expected image and digest.
  3. Confirm the Helm chart version exists and renders with the target values.
  4. Review configuration differences for DFW and SJC.
  5. Confirm Keeper secret references exist without displaying their values.
  6. Record current deployment images, replicas, readiness, restart counts, and resource use.
  7. Record current Prometheus target health and latest metric timestamps.
  8. Check active incidents, maintenance, and relevant alert state.
  9. Confirm the rollback version is still available in the registry.

Deployment Flow

sequenceDiagram
    participant Eng as Deployment Engineer
    participant PR as Configuration Pull Request
    participant Argo as WBX3 or Argo
    participant Canary as First Segment or Cluster
    participant Remaining as Remaining Segments or Cluster
    participant Obs as Prometheus, Mimir, and Grafana

    Eng->>PR: Submit reviewed version/config change
    PR->>Argo: Trigger approved deployment
    Argo->>Canary: Deploy first bounded target
    Canary->>Obs: Expose health and metrics
    Obs-->>Eng: Validate acceptance criteria
    Eng->>Argo: Authorize remaining rollout
    Argo->>Remaining: Deploy approved version
    Remaining->>Obs: Expose health and metrics
    Obs-->>Eng: Confirm completion or rollback

Deployment Steps

  1. Authenticate to the correct cluster and verify the current context.
  2. Trigger the deployment using the configuration repository's current WBX3/Argo workflow.
  3. Deploy the smallest safe customer segment or one cluster first when supported.
  4. Watch the pipeline and Kubernetes rollout to completion.
  5. Do not proceed while pods are unavailable, crash-looping, or failing readiness.
  6. Run canary validation below.
  7. Continue with the remaining segments/cluster only after acceptance criteria pass.
  8. Repeat validation for every production target.

Kubernetes Validation

kubectl rollout status deployment/<deployment> -n <namespace>
kubectl get deployments,pods,services,endpoints -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp
kubectl logs <pod> -n <namespace> --all-containers --since=15m

Confirm:

  • expected image digest and replica count;
  • ready/available replicas and stable restarts;
  • correct ConfigMaps, Secret references, and mounted files;
  • service endpoints and ServiceMonitor discovery;
  • resource use within normal range;
  • no authentication, collection, parsing, or remote-write errors.

Telemetry Validation

  1. Confirm the expected targets are present and up in Prometheus.
  2. Compare scrape duration and errors with the pre-deployment baseline.
  3. Query at least one expected metric for each collection method affected.
  4. Confirm new samples appear in Mimir/Grafana with correct labels.
  5. Validate dashboards and alert queries impacted by schema/label changes.
  6. Confirm no duplicate series/customer segments were introduced.
  7. If alert routing changed, test firing and resolved delivery with approved synthetic data.

Abort Criteria

Stop the rollout when any of these occurs:

  • readiness or availability does not recover in the approved window;
  • material increase in collection failures or duration;
  • missing or duplicate customer coverage;
  • unexpected credential or network failures;
  • remote-write backlog/rejections attributable to the change;
  • dashboard or alert breakage that removes operational visibility;
  • resource saturation or crash loops.

Rollback

  1. Stop further rollout.
  2. Restore the previous image/chart/config version in the owning repository.
  3. Trigger the managed deployment workflow.
  4. Verify Kubernetes availability and image digest.
  5. Verify Prometheus targets, latest samples, dashboards, and alert recovery.
  6. Check whether any delayed/backlogged data requires follow-up.
  7. Record the failure evidence, rollback time, and customer impact.

Avoid direct kubectl rollback unless the approved incident procedure requires it. Reconcile any emergency live change back to source control immediately.

Completion

Attach or link:

  • pipeline/deployment result;
  • before/after workload evidence;
  • Prometheus/Mimir/Grafana validation;
  • alert validation when applicable;
  • final deployed digest/chart/config revision;
  • rollback decision and follow-up actions.