Skip to content

Certificate Management

Business Domain

Certificate management in Kubernetes refers to the process of creating, distributing, and rotating cryptographic certificates that are used within a Kubernetes cluster. These certificates are crucial for securing communication between the various components of the cluster, such as nodes, pods, services, and the Kubernetes API server.

Related Components of Certificate management from Backstage:

  • Cert Manager: The cert-manager is a Kubernetes operator that runs within every cluster, and is responsible for provisioning certificates for all workloads in the cluster.
  • cert-manager-cainjector
  • cert-manager-controller
  • cert-manager-package-debian
  • cert-manager-webhook
  • trust-manager

Customer Impact Cases

Proper certificate management is crucial for maintaining the security of a Kubernetes cluster. It's important to monitor certificate expiration dates and have policies in place for regular certificate rotation to minimize the risk of security breaches and service interruptions.

The following cases require the attention of the SRE team and need immediate action.

  1. The certificate issued by Cert-manager has currently expired and cluster certificate has expired
  2. The Cert-manager pod is crashing in a loop or its deployment has no functioning replica.
  3. (Operational) The visits to Istio ingress-gateway or ingress TLS are failed due to bad certificate issues

Case 1 - Certificate Expiration

Certificate issued by Cert-manager

Certificates issued by Cert-manager are used to provide TLS certificates for Kubernetes Ingress resources, Webhook. Cert-Manager also issue and manage certificates for mutual TLS (mTLS) between services of Service Mesh.

This indicator is provided to detect the expiration of all certificates issued by Cert-manager which is measured by Mimir alert rules based on metrics certmanager_certificate_expiration_timestamp_seconds

a. Indicator for Warning Severity 

avg by (exported_namespace, namespace, name) (certmanager_certificate_expiration_timestamp_seconds - time()) < (24 * 3600)

b. Indicator for Critical Severity

This indicator can be considered as SLI present in Core Component Dashboard

# Major Impact
avg by (exported_namespace, namespace, name) (certmanager_certificate_expiration_timestamp_seconds - time()) < 0

Kubeadm Certificate expiration

Kubeadm takes care of several setup and configuration tasks, one of which is the management of certificates required for the secure operation of a Kubernetes cluster. The certificates used by Kubeadm are for secures communication to the Kubernetes API server from kubectl, kubelet, scheduler, controller manager, etcd, etc.

Indicator: calculate the remaining time before the expiration of Kubernetes certificates on master nodes and determine if any of those certificates will expire within the next 24 hours.

a. Indicator for Warning Severity 

# Degraded Performance
min by (node, cert) (kubeadm_cert_expiration * on (pod) group_left (node) kube_pod_info{node=~".*master.*"}) - time() < (24 * 3600)

b. Indicator for Critical Severity

This indicator can be considered as SLI present in Core Component Dashboard

# Major Impact
min by (node, cert) (kubeadm_cert_expiration * on (pod) group_left (node) kube_pod_info{node=~".*master.*"}) - time() < 0

Case 2 - Cert-manager Pod Crash

Indicator: calculating the rate of container restarts within the cert-manager-v1 namespace over the past hour and then determining if this rate is greater than zero.

  • Indicator for Critical Severity

This indicator can be considered as SLI present in Core Component/Feature Dashboard

rate(kube_pod_container_status_restarts_total{job="kube-state-metrics",namespace="cert-manager-v1"}[1h]) * 60 * 60 > 0

SLI/SLO of Certificate Management

SLI Definition and SLO

According indicators mentioned in customer impact cases, SLI of Certificate Management:

  • Service certificate expiration rate per cluster < 1%
  • Cluster node certificate expiration rate per cluster < 1%
  • Availability of Cert Manager > 99%
  • Number of expire certificate < 1

Alert integration with ISS

  • SLI data
  • Specific certificate expiration
  • Cert manager pod crash
  • MCT certificate check

Create SLO Grafana Alert Rule based on PromQL query

Please refer to operational doc Metrics 2.0 Alert Provisioning

Grafana SLI Dashboard for Cert Management (core component/feature)

This dashboard should include monitoring visualizations as below:

  • SLI/SLO
  • Customer cases indicator trend
  • External ping check (MCT certificate check if exist)
  • Synthetic Test (TAP test cases if exist)

SLI dashboards are created in dashboard folder SLI of Core Components - wbx3