Skip to content

Create an Alert Rule

Start by defining the service impact, first owner, urgency, and expected response action. If a responder cannot act on the notification, improve the signal before creating the alert.

Grafana Alert

Step 1: Define the Query

Develop PromQL over representative healthy, degraded, and missing-data windows. Preserve labels needed for customer/device ownership and routing.

Step 2: Set Conditions

Choose an evaluation interval, pending duration, threshold, no-data behavior, and evaluation-error behavior that detect sustained impact without creating transient noise.

Step 3: Configure Notifications

Add severity, owner, runbook/dashboard links, and the correct contact-point label. Confirm grouping does not combine unrelated customers.

Step 4: Test

Test firing, pending-to-alerting transition, notification delivery, acknowledgement, and resolved notification. Deploy through the managed alert repository.

Splunk Alert

Use an explicit index and bounded time window. Confirm all selected sourcetypes provide the fields required for routing and payloads.

Step 2: Set Conditions

Define schedule, trigger type, comparator, threshold, severity, suppression fields/period, and per-result versus digest behavior.

Step 3: Configure Notifications

Select the approved action/contact point and remove secrets or customer-sensitive content from the payload.

Create and deploy the alert through the managed Terraform repository; avoid untracked UI-only production alerts.