CUCM - Cisco Unified Call Manager
Overview
Cisco Unified Communications Manager (CUCM) is the core call processing platform for the collaboration environment. It handles call routing, signaling, and endpoint registration.
Monitoring
Method
- SNMP Exporter — standard SNMP polling for infrastructure metrics
- Custom Prometheus Exporter — in-house exporter using three collection methods:
- API (AXL/SOAP/REST) — device registration, service status, trunk/gateway status, RTMT alerts
- CLI (SSH) — NTP, backup, license, coredump, ungraceful shutdown detection
- Perfmon (SOAP PerfmonService) — 90+ performance counters across 40+ objects
Exporter Repository: cucm-prometheus-exporter v2.0.0
SNMP Metrics (SNMP Exporter)
Collected via the Prometheus SNMP Exporter using three modules: cucm_metrics, cucm_metrics_01, and cucm_metrics_02. Covers network interfaces, host resources, TCP/UDP stats, and Cisco CCM MIB device counters.
Network Interfaces (IF-MIB) — click to expand
| Metric | Type | Description |
|---|---|---|
ifIndex |
gauge | Unique interface index |
ifDescr |
DisplayString | Interface description |
ifSpeed |
gauge | Interface bandwidth (bits/sec) |
ifPhysAddress |
PhysAddress | Interface MAC address |
ifAdminStatus |
gauge | Desired interface state (1=up, 2=down, 3=testing) |
ifOperStatus |
gauge | Current operational state (1=up, 2=down, 3=testing, 4=unknown, 5=dormant, 6=notPresent, 7=lowerLayerDown) |
ifInOctets |
counter | Total octets received |
ifOutOctets |
counter | Total octets transmitted |
ifAlias |
DisplayString | Interface alias name |
Labels: ifIndex, ifDescr
Host Resources (HOST-RESOURCES-MIB) — click to expand
Storage:
| Metric | Type | Description |
|---|---|---|
hrStorageDescr |
DisplayString | Storage description (e.g., Physical memory, /, /common) |
hrStorageAllocationUnits |
gauge | Allocation unit size in bytes |
hrStorageSize |
gauge | Total storage size (in allocation units) |
hrStorageUsed |
gauge | Storage used (in allocation units) |
hrStorageAllocationFailures |
counter | Storage allocation failures |
Labels: hrStorageIndex, hrStorageDescr, hrStorageType
Devices & Processors:
| Metric | Type | Description |
|---|---|---|
hrDeviceIndex |
gauge | Unique device index |
hrDeviceDescr |
DisplayString | Device description (manufacturer, model) |
hrDeviceStatus |
gauge | Device state (1=unknown, 2=running, 3=warning, 4=testing, 5=down) |
hrProcessorLoad |
gauge | Average CPU load over last minute (%) |
Labels: hrDeviceIndex
TCP Statistics — click to expand
| Metric | Type | Description |
|---|---|---|
tcpActiveOpens |
counter | Connections transitioned to SYN-SENT from CLOSED |
tcpPassiveOpens |
counter | Connections transitioned to SYN-RCVD from LISTEN |
tcpAttemptFails |
counter | Failed connection attempts |
tcpCurrEstab |
gauge | Current ESTABLISHED or CLOSE-WAIT connections |
tcpInSegs |
counter | Total segments received (including errors) |
tcpOutSegs |
counter | Total segments sent (excluding retransmissions) |
tcpRetransSegs |
counter | Total retransmitted segments |
tcpInErrs |
counter | Segments received with errors (bad checksums) |
tcpOutRsts |
counter | Segments sent with RST flag |
UDP Statistics — click to expand
| Metric | Type | Description |
|---|---|---|
udpInDatagrams |
counter | Total UDP datagrams delivered |
udpInErrors |
counter | UDP datagrams that could not be delivered |
udpOutDatagrams |
counter | Total UDP datagrams sent |
System Information — click to expand
| Metric | Type | Description |
|---|---|---|
sysDescr |
DisplayString | System description |
sysUpTime |
gauge | Uptime in hundredths of a second |
sysName |
DisplayString | System hostname |
Cisco CCM MIB — Phone Registration — click to expand
| Metric | Type | Description |
|---|---|---|
ccmRegisteredPhones |
counter | Phones registered with local CallManager |
ccmUnregisteredPhones |
counter | Phones unregistered or lost contact |
ccmRejectedPhones |
counter | Phones whose registration was rejected |
ccmPartiallyRegisteredPhones |
counter | Phones partially registered |
Cisco CCM MIB — Gateway Registration — click to expand
| Metric | Type | Description |
|---|---|---|
ccmRegisteredGateways |
counter | Gateways registered with local CallManager |
ccmUnregisteredGateways |
counter | Gateways unregistered or lost contact |
ccmRejectedGateways |
counter | Gateways whose registration was rejected |
Cisco CCM MIB — Media Device Table — click to expand
Aggregate Counters:
| Metric | Type | Description |
|---|---|---|
ccmRegisteredMediaDevices |
counter | Media devices registered |
ccmUnregisteredMediaDevices |
counter | Media devices unregistered |
ccmRejectedMediaDevices |
counter | Media devices rejected |
Per-Device Table (ccmMediaDeviceTable):
| Metric | Type | Description |
|---|---|---|
ccmMediaDeviceIndex |
gauge | Media device table index |
ccmMediaDeviceName |
DisplayString | Device name |
ccmMediaDeviceStatus |
gauge | Status (2=registered, 3=unregistered, 4=rejected, 5=partiallyregistered) |
ccmMediaDeviceStatusReason |
gauge | Reason code for status change |
ccmMediaDeviceTimeLastStatusUpdt |
DateAndTime | Time of last status change |
ccmMediaDeviceTimeLastRegistered |
DateAndTime | Time of last registration |
ccmMediaDeviceUnregReason |
gauge | Unregistration reason code |
ccmMediaDeviceRegFailReason |
gauge | Registration failure reason code |
Labels: ccmMediaDeviceIndex
Cisco CCM MIB — CTI Device Table — click to expand
Aggregate Counters:
| Metric | Type | Description |
|---|---|---|
ccmRegisteredCTIDevices |
counter | CTI devices registered |
ccmUnregisteredCTIDevices |
counter | CTI devices unregistered |
ccmRejectedCTIDevices |
counter | CTI devices rejected |
Per-Device Table (ccmCTIDeviceTable):
| Metric | Type | Description |
|---|---|---|
ccmCTIDeviceIndex |
gauge | CTI device table index |
ccmCTIDeviceName |
DisplayString | Device name |
ccmCTIDeviceStatus |
gauge | Status (2=registered, 3=unregistered, 4=rejected, 5=partiallyregistered) |
ccmCTIDeviceStatusReason |
gauge | Reason code for status change |
ccmCTIDeviceTimeLastStatusUpdt |
DateAndTime | Time of last status change |
ccmCTIDeviceTimeLastRegistered |
DateAndTime | Time of last registration |
ccmCTIDeviceUnregReason |
gauge | Unregistration reason code |
ccmCTIDeviceRegFailReason |
gauge | Registration failure reason code |
Labels: ccmCTIDeviceIndex
Cisco CCM MIB — VoiceMail Devices — click to expand
| Metric | Type | Description |
|---|---|---|
ccmRegisteredVoiceMailDevices |
counter | VoiceMail devices registered |
ccmUnregisteredVoiceMailDevices |
counter | VoiceMail devices unregistered |
ccmRejectedVoiceMailDevices |
counter | VoiceMail devices rejected |
API Metrics (AXL / SOAP / REST)
Collected via SOAP calls to CUCM RIS, ControlCenter, AXL, and REST endpoints. 9 metrics covering device registration, service health, and infrastructure status.
API Metrics — click to expand
| Metric | Description | Availability |
|---|---|---|
cucm_device_registration_count |
Count of registered SIP devices | Publisher & Subscriber |
cucm_service_status |
Status of CUCM services (Started=1, Stopped=0, Not Activated=2) | Publisher & Subscriber |
cucm_system_version |
CUCM version info (value=1, version in label) | Publisher & Subscriber |
cucm_sip_trunk_status |
SIP trunk registration (1=Registered, 0=Unregistered) | Publisher only |
cucm_cisco_gateway_registration |
Gateway registration status | Publisher only |
cucm_cisco_cti_port_registration |
CTI Port registration status | Publisher only |
cucm_cisco_cti_route_point_registration |
CTI Route Point registration status | Publisher only |
cucm_cisco_media_resource_registration |
Media resource registration status | Publisher only |
cucm_rtmt_active_alerts |
Active RTMT alerts (value=1 per alert) | Publisher only |
Services monitored by cucm_service_status:
- Cisco CallManager
- Cisco CTIManager
- Cisco Tftp
- Cisco IP Voice Media Streaming App
- Cisco Extension Mobility
- Cisco AXL Web Service
- Cisco RIS Data Collector
- Cisco DRF Master / Local
- Cisco CallManager SNMP Service
- SOAP - Performance Monitoring APIs
- Cisco Smart License Manager
CLI Metrics (SSH)
Collected by executing CLI commands over SSH and parsing the output. 6 metrics covering system health and operational status.
CLI Metrics — click to expand
| Metric | CLI Command | Description | Availability |
|---|---|---|---|
cucm_ntp_status |
utils ntp status |
NTP sync status (1=synchronised, 0=unsynchronised) | Publisher & Subscriber |
cucm_coredump_file_status |
utils core active list |
Core dumps in last 24h (1=found, 0=none) | Publisher & Subscriber |
cucm_last_backup_status |
utils disaster_recovery status backup |
Last backup result (1=success, 0=failed) | Publisher only |
cucm_license_status |
show license status |
License registration + authorization (1=valid, 0=invalid) | Publisher only |
cucm_file_list_activelog_core_status |
file list activelog /core/ |
Directory count in activelog core | Publisher & Subscriber |
cucm_ungraceful_shutdown_detected |
SSH banner detection | Ungraceful shutdown warning (1=detected, 0=clear) | Publisher & Subscriber |
Perfmon Counters (SOAP PerfmonService)
Collected via the CUCM Perfmon SOAP service. 90+ counters across 40+ objects. All exposed as cucm_perfmon_counters with labels object_name, counter_name, and perfmon_instance_name.
CallManager Core — click to expand
Cisco CallManager:
| Counter | Description |
|---|---|
| CallsActive | Currently active calls |
| CallsCompleted | Total completed calls |
| CallsAttempted | Total attempted calls |
| CallManagerHeartBeat | CallManager process heartbeat |
| RegisteredAnalogAccess | Registered analog access devices |
| RegisteredOtherStationDevices | Other registered station devices |
Cisco CallManager System Performance:
| Counter | Description |
|---|---|
| CodeRedEntryExit | Code Red throttling events |
| CodeYellowEntryExit | Code Yellow throttling events |
| CallsRejectedDueToThrottling | Calls rejected under load |
| SkinnyDevicesThrottled | SCCP devices throttled |
| QueueSignalsPresent 1-High | High priority queue depth |
| QueueSignalsPresent 2-Normal | Normal priority queue depth |
| QueueSignalsPresent 3-Low | Low priority queue depth |
Cisco Call Restriction:
| Counter | Description |
|---|---|
| TransferFailures | Failed call transfers |
| SharedLineFailures | Shared line failures |
| PickUpFailures | Call pickup failures |
| ParkRetrievalFailures | Park retrieval failures |
| MidCallFailures | Mid-call feature failures |
| MeetMeConferenceFailures | Meet-Me conference failures |
| LogicalPartitionFailuresTotal | Logical partition failures |
| ForwardingFailures | Call forwarding failures |
| BasicCallFailures | Basic call failures |
| AdHocConferenceFailures | Ad-hoc conference failures |
Media Resources — click to expand
Cisco HW Conference Bridge Device (multi-instance):
| Counter | Description |
|---|---|
| ResourceAvailable | Available conference bridge resources |
| ResourceActive | Active conference bridge resources |
| OutOfResources | Out-of-resource events |
Cisco MOH Device (multi-instance):
| Counter | Description |
|---|---|
| MOHTotalUnicastResources | Total MOH unicast resources |
| MOHUnicastResourceAvailable | Available MOH unicast resources |
Cisco Transcode Device (multi-instance):
| Counter | Description |
|---|---|
| ResourceActive | Active transcoder resources |
| ResourceAvailable | Available transcoder resources |
| OutOfResources | Out-of-resource events |
Cisco Annunciator Device (multi-instance):
| Counter | Description |
|---|---|
| ResourceActive | Active annunciator resources |
| ResourceAvailable | Available annunciator resources |
| OutOfResources | Out-of-resource events |
Cisco MTP Device (multi-instance):
| Counter | Description |
|---|---|
| ResourceActive | Active MTP resources |
| ResourceAvailable | Available MTP resources |
| OutOfResources | Out-of-resource events |
Cisco Video Conference Bridge Device (multi-instance):
| Counter | Description |
|---|---|
| ConferencesActive | Active video conferences |
| ConferencesTotal | Total video conferences |
| ResourceActive | Active resources |
| ResourceTotal | Total resources |
| OutOfResources | Out-of-resource events |
Cisco Telepresence MCU Conference Bridge Device (multi-instance):
| Counter | Description |
|---|---|
| HttpConnectionErrors | HTTP connection errors to MCU |
| HttpNon200OkResponse | Non-200 HTTP responses |
| OutOfResources | Out-of-resource events |
Trunks & Gateways — click to expand
Cisco SIP (multi-instance):
| Counter | Description |
|---|---|
| CallsCompleted | Completed SIP calls |
| CallsAttempted | Attempted SIP calls |
Cisco H323 (multi-instance):
| Counter | Description |
|---|---|
| CallsCompleted | Completed H.323 calls |
| CallsAttempted | Attempted H.323 calls |
| CallsRejectedDueToICTCallThrottling | Calls rejected by ICT throttling |
Cisco MGCP PRI Device (single-instance):
| Counter | Description |
|---|---|
| CallsActive | Active MGCP PRI calls |
| CallsCompleted | Completed MGCP PRI calls |
Cisco MGCP FXS Device (multi-instance):
| Counter | Description |
|---|---|
| PortStatus | FXS port status |
Cisco MGCP FXO Device (multi-instance):
| Counter | Description |
|---|---|
| PortStatus | FXO port status |
Cisco MGCP T1CAS Device (multi-instance):
| Counter | Description |
|---|---|
| Channel 1-24 Status | Status of each T1 channel |
| CallsActive | Active T1CAS calls |
| CallsCompleted | Completed T1CAS calls |
Cisco MGCP BRI Device (multi-instance):
| Counter | Description |
|---|---|
| Channel 1-2 Status | BRI channel status |
| DatalinkInService | Datalink service status |
| OutboundBusyAttempts | Outbound busy attempts |
Infrastructure & Platform — click to expand
Cisco Tomcat JVM (single-instance):
| Counter | Description |
|---|---|
| KBytesMemoryTotal | JVM total memory (KB) |
| KBytesMemoryFree | JVM free memory (KB) |
| KBytesMemoryMax | JVM max memory (KB) |
Cisco Tomcat Connector (multi-instance):
| Counter | Description |
|---|---|
| ThreadsMax | Max connector threads |
| ThreadsBusy | Busy connector threads |
Cisco TFTP:
| Counter | Description |
|---|---|
| HeartBeat | TFTP service heartbeat |
| HttpRequests | HTTP requests received |
| HttpRequestsProcessed | HTTP requests processed |
| Requests | Total requests received |
| RequestsProcessed | Total requests processed |
System:
| Counter | Description |
|---|---|
| IOPerSecond | Disk I/O operations per second |
Process (multi-instance):
| Counter | Description |
|---|---|
| % CPU Time | Process CPU usage |
| Total CPU Time Used | Cumulative CPU time |
| % Memory Usage | Process memory usage |
| Process Status | Process running status |
Partition (multi-instance):
| Counter | Description |
|---|---|
| % CPU Time | Partition CPU usage |
| % Used | Partition disk usage |
Ramfs (multi-instance):
| Counter | Description |
|---|---|
| SpaceUsed | RAM filesystem space used |
| SpaceFree | RAM filesystem space free |
Network Interface (multi-instance):
| Counter | Description |
|---|---|
| Tx Dropped | Transmitted packets dropped |
| Rx Dropped | Received packets dropped |
| Tx Errors | Transmit errors |
| Rx Errors | Receive errors |
Other Services — click to expand
Cisco SIP Station (single-instance):
| Counter | Description |
|---|---|
| TokensRejected | SIP registration tokens rejected |
| NewRegRejected | New registrations rejected |
| DBTimeouts | Database timeout events |
Cisco CTI Manager (single-instance):
| Counter | Description |
|---|---|
| CTIConnectionActive | Active CTI connections |
| CcmLinkActive | Active CCM links |
Cisco Extension Mobility (single-instance):
| Counter | Description |
|---|---|
| Requests Throttled | Throttled EM requests |
| Successful Logins | Successful EM logins |
| Successful Logouts | Successful EM logouts |
| Total Attempted Login/Logout Requests | Total EM attempts |
Cisco WebDialer (single-instance):
| Counter | Description |
|---|---|
| CallsCompleted | Completed WebDialer calls |
| CallsFailed | Failed WebDialer calls |
| RequestsCompleted | Completed WebDialer requests |
| RequestsFailed | Failed WebDialer requests |
Cisco Hunt Lists (multi-instance):
| Counter | Description |
|---|---|
| CallsRingNoAnswer | Hunt list ring-no-answer calls |
| CallsAbandoned | Abandoned hunt list calls |
Cisco Gatekeeper (multi-instance):
| Counter | Description |
|---|---|
| VideoOutOfResources | Video out-of-resource events |
Cisco Locations LBM (multi-instance):
| Counter | Description |
|---|---|
| BandwidthMaximum / BandwidthAvailable | Audio bandwidth limits |
| VideoBandwidthMaximum / VideoBandwidthAvailable | Video bandwidth limits |
| ImmersiveVideoBandwidthMaximum / ImmersiveVideoBandwidthAvailable | Immersive video bandwidth |
| OutOfResources / VideoOutOfResources / ImmersiveOutOfResources | Out-of-resource events |
Cisco Locations RSVP (multi-instance):
| Counter | Description |
|---|---|
| RSVP TotalCallsFailed | Total RSVP call failures |
| RSVP VideoCallsFailed | RSVP video call failures |
Replication (Publisher Only) — click to expand
Number of Replicates Created and State of Replication (multi-instance):
| Counter | Description |
|---|---|
| Replicate_State | Database replication state per subscriber |
Common Exporter Metrics
Present in every scrape response regardless of collection method.
| Metric | Description |
|---|---|
cucm_device_up |
Device reachability (1=up, 0=down) |
cucm_device_scrape_success |
Scrape result (1=success, 0=failure) |
cucm_device_scrape_duration_seconds |
Time taken for the scrape |
Grafana Dashboards
| Dashboard | Description |
|---|---|
| CUCM Detailed | Detailed CUCM metrics — API, CLI, and Perfmon counters |
| SNMP Metrics - Customer Level | SNMP metrics aggregated at customer level |
| SNMP Metrics | Per-node SNMP metrics (interfaces, storage, CPU, TCP/UDP) |
| HX Cluster View | CUCM devices per HyperFlex cluster |
Logging
Log Source
CUCM nodes send syslogs to a centralized syslog-ng server. The syslog-ng pipeline enriches each log event with metadata before forwarding to Splunk:
- Transport: Syslog (RFC 3164) from CUCM to central syslog-ng collector
- Enrichment: syslog-ng injects metadata fields:
tenant,region,node_type,ip_address,device_type,data_center,customer_name,customer_id - Format: JSON-structured events with both parsed fields and original
RAWMSG - Collector hosts: Regional syslog-ng instances (e.g.,
cdi-syslog-*.prod.infra.webex.com)
Splunk Index
| Field | Value |
|---|---|
| Index | vos_syslog |
| Sourcetype | syslog |
| Source pattern | /var/log/syslog-ng/syslog/cucm/YYYY/MM/DD/HH.log |
Splunk Cloud Instances:
| Instance | Regions | URL |
|---|---|---|
| US/APAC | US, APAC | cisco-webxbu-wxcdi.splunkcloud.com |
| EMEA | EMEA | cisco-webxbu-wxcdieu.splunkcloud.com |
Data Centers:
| Data Center | Region |
|---|---|
| iad1, lax1, sjc1, dfw1 | US |
| sin1, tky2, mel1, syd1 | APAC |
Node Types in this index:
| Node Type | Description |
|---|---|
cucm_pub |
CUCM Publisher |
cucm_sub |
CUCM Subscriber |
Enriched Metadata Fields:
| Field | Description | Example |
|---|---|---|
tenant |
Full tenant identifier | ucmc_us_ibm_cucm_uc_cluster_21 |
region |
Deployment region | us, apac |
node_type |
Node role | cucm_pub, cucm_sub |
ip_address |
Node IP address | 10.10.117.20 |
device_type |
Device type | cucm |
data_center |
Data center code | lax1, iad1, mel1 |
customer_name |
Customer name | ibm, dukeenergy |
customer_id |
Customer ID | c3011, c6801 |
Syslog Fields:
| Field | Description | Example |
|---|---|---|
HOST / HOST_FROM |
Originating CUCM hostname | c6801011ccm104 |
FACILITY |
Syslog facility | local7, authpriv, daemon |
PRIORITY |
Syslog priority | err, warning, notice, info, crit |
MESSAGE |
Parsed syslog message | Full message text |
RAWMSG |
Original raw syslog message | Unparsed syslog line |
ISODATE |
Event timestamp (ISO 8601) | 2026-04-16T06:14:11+00:00 |
COLLECTOR_ID |
Syslog-ng collector ID | 136ecab4 |
COLLECTOR_HOST |
Syslog-ng collector hostname | cdi-syslog-25b1a |
Key Log Queries
Common SPL queries — click to expand
All CUCM logs for a specific customer (last 24h):
index=vos_syslog customer_id="c6801" node_type="cucm_*"
Error and critical logs only:
index=vos_syslog device_type="cucm" (PRIORITY="err" OR PRIORITY="crit")
| spath FACILITY
| stats count by FACILITY, PRIORITY, HOST
| sort -count
Device registration failures:
index=vos_syslog device_type="cucm"
"*DeviceTransientConnection*" OR "*EndPointTransientConnection*"
| spath HOST
| stats count by HOST, customer_name
| sort -count
Logs by facility and priority distribution:
index=vos_syslog device_type="cucm"
| spath FACILITY | spath PRIORITY
| stats count by FACILITY, PRIORITY
| sort -count
Logs by data center:
index=vos_syslog device_type="cucm"
| spath data_center
| stats count by data_center
| sort -count
Search for a specific CUCM node:
index=vos_syslog HOST="c6801011ccm104"
| sort -_time
Alerting
Grafana Alert Rules
Alert Group: cucm-alerts-5m (evaluated every 5 minutes)
Notification Channel: EMS v2 (wxdipo_wxcdi_ems_v2)
Alert Source: metrics-grafana-alerts-prd-commercial
Critical Alerts — click to expand
| Alert ID | Alert Name | Description | Metric / Condition |
|---|---|---|---|
| wxcdi006 | Critical Service Not Started | A critical CUCM service stopped that was previously running | cucm_service_status == 0 for critical services (CallManager, CTIManager, Tftp, AXL, Tomcat, etc.) with max_over_time [12h] == 1 |
| wxcdi018 | Storage Utilization Exceeded Critical Threshold | Disk storage usage above 95% | hrStorageUsed / hrStorageSize * 100 > 95 for physical/virtual storage |
| wxcdi024 | Replicate State Issue | Database replication is no longer in good state | cucm_perfmon_counters{Replicate_State} != 2 when it was previously healthy within 7 days |
| wxcdi032 | Virtual Memory Utilization High | Virtual memory usage above 90% | (Physical RAM + Virtual Memory Used) / (Physical RAM + Virtual Memory Size) * 100 > 90 |
High Severity Alerts — click to expand
| Alert ID | Alert Name | Description | Metric / Condition |
|---|---|---|---|
| wxcdi003 | Code Red Entry | CallManager entered Code Red throttling state | cucm_perfmon_counters{CodeRedEntryExit} == 1 when min_over_time [12h] == 0 |
| wxcdi005 | Coredump File Detected | New core dump file found in last 24h | cucm_coredump_file_status == 1 when min_over_time [12h] == 0 |
| wxcdi008 | CTI Route Point Unregistered | E911 CTI Route Point (E911/E912/E913) lost registration | cucm_cisco_cti_route_point_registration{cti_name=~"E91[1-3]_RP"} == 0 when previously registered |
| wxcdi015 | CPU Exceeded Major Threshold | Average CPU load above 90% | avg(hrProcessorLoad{device_type="cucm"}) > 90 |
| wxcdi026 | SIP Trunk Partially Registered (DI) | DI customer SIP trunk moved to partial registration | cucm_sip_trunk_status == 2 for DI customers (c9xxx) |
| wxcdi027 | SIP Trunk Unregistered (DI) | DI customer SIP trunk unregistered | cucm_sip_trunk_status == 0 for DI customers when previously registered |
| wxcdi028 | SIP Trunk Partially Registered (Non-DI) | Non-DI customer SIP trunk moved to partial registration | cucm_sip_trunk_status == 2 for non-DI customers |
| wxcdi029 | SIP Trunk Unregistered (Non-DI) | Non-DI customer SIP trunk unregistered | cucm_sip_trunk_status == 0 for non-DI customers when previously registered |
| wxcdi031 | Tomcat JVM Memory Utilization High | JVM heap usage above 97% | (KBytesMemoryTotal - KBytesMemoryFree) / KBytesMemoryMax * 100 > 97 |
Warning Alerts — click to expand
| Alert ID | Alert Name | Description | Metric / Condition |
|---|---|---|---|
| wxcdi009 | CTI Manager Active Links Decreased | CTI Manager lost all active connections | cucm_perfmon_counters{CTIConnectionActive} == 0 when max_over_time [16m] > 0 |
| wxcdi025 | Service Not Started (Non-Critical) | A non-critical CUCM service stopped that was previously running | cucm_service_status == 0 for non-critical services with max_over_time [12h] == 1 |
Alert Summary: 15 Grafana alert rules total — 4 critical, 9 high, 2 warning
Splunk Alerts
Index: vos_syslog | Schedule: Every 2-15 minutes | Notification: EMS v2 (webhook)
Alert Source: terraform-splunk-wxcdi-alerts
Active alerts: 62 | Disabled: 16
Critical / High Severity (Splunk severity 4-5) — click to expand
| Alert ID | Alert Name | Description | Schedule |
|---|---|---|---|
| WxCDI101 | SDL Link OOS | CallManager SDL Link Out Of Service | */10 min |
| WxCDI148 | Remote Survivable Node Not Reachable | RTMT RemoteSurvivableNodeNotReachable | */10 min |
| WxCDI122 | Push Notification Service Unavailable | Push Notification Service Unavailable | */10 min |
| WxCDI124 | TVS Threads Exhausted | TVS Threads Exhausted | */10 min |
| WxCDI127 | DBL Exception | DBL Exception | */10 min |
| WxCDI129 | System Resource Error | System Resource Error | */10 min |
| WxCDI131 | Push Notification Access Token Unavailable | Push Notification Service Access Token Unavailable | */10 min |
| WxCDI141 | Device Unregistered | Device Unregistered | */10 min |
| WxCDI147 | Timer Thread Slowed | TimerThreadSlowed | */10 min |
Medium Severity (Splunk severity 3) — click to expand
| Alert ID | Alert Name | Description | Schedule |
|---|---|---|---|
| ControlHub001 | CDR File Delivery Failed | RTMT CDRFileDeliveryFailed | */15 min |
| ControlHub013 | Registered Gateways Decreased | RTMT NumberOfRegisteredGatewaysDecreased | */15 min |
| ControlHub014 | Registered Gateways Increased | RTMT NumberOfRegisteredGatewaysIncreased | */15 min |
| ControlHub015 | Registered Media Devices Decreased | RTMT NumberOfRegisteredMediaDevicesDecreased | */15 min |
| ControlHub016 | Registered Media Devices Increased | RTMT NumberOfRegisteredMediaDevicesIncreased | */15 min |
| ControlHub017 | Remote Survivable Node Not Reachable | RTMT RemoteSurvivableNodeNotReachable | */15 min |
| ControlHub023 | Dir Sync Process Fail To Start | Cisco Dir Sync Process Fail To Start | */15 min |
| WxCDI115 | Server Down | RTMT ServerDown | */10 min |
| WxCDI116 | Registered Devices Exceeded | RTMT NumberOfRegisteredDevicesExceeded | */10 min |
| WxCDI117 | CDR File Delivery Failure Recover | CDRFileDeliveryFailureContinues Recover | */10 min |
| WxCDI119 | Error Change Notify Client Block | Error Change Notify Client Block | */10 min |
| WxCDI120 | Signal Congestion Entry | Signal Congestion Entry | */10 min |
| WxCDI121 | CDR Agent Send File Failure | CDRAgentSendFileFailureContinues | */10 min |
| WxCDI123 | Local Push Notification Unavailable | Local Push Notification Service Unavailable | */10 min |
| WxCDI125 | Code Yellow Entry | Code Yellow Entry | */10 min |
| WxCDI126 | Service Port Offline | Service Port Offline | */10 min |
| WxCDI128 | CDR Max Disk Space Exceeded | CDR Maximum Disk Space Exceeded | */10 min |
| WxCDI130 | Serviceability Connectivity Interruptions | Serviceability Connectivity Interruptions | */10 min |
| WxCDI132 | CallManager Failure | CallManagerFailure | */10 min |
| WxCDI138 | Voicemail Port Unregistered | Voicemail Port Unregistered | */10 min |
| WxCDI142 | CDR File Delivery Failure | CDRFileDeliveryFailureContinues | */10 min |
| WxCDI146 | Low Inactive Partition Disk Space | LowInactivePartitionAvailableDiskSpace | */10 min |
| WxCDI149 | CDR High Water Mark Exceeded | CDRHighWaterMarkExceeded | */10 min |
| WxCDI150 | TFTP Cluster Node Down | TFTP Cluster Node Offline | */2 min |
| WxCDI151 | Excessive Voice Quality Reports | ExcessiveVoiceQualityReports | */10 min |
| WxCDI152 | ILS Pwd Authentication Failed | ILSPwdAuthenticationFailed | */10 min |
| WxCDI153 | ILS Duplicate URI | ILSDuplicateURI | */10 min |
Low / Informational Severity (Splunk severity 1-2) — click to expand
| Alert ID | Alert Name | Description | Schedule |
|---|---|---|---|
| ControlHub002 | Recording Gateway Session Failed | RecordingGatewaySessionFailed | */15 min |
| ControlHub003 | EMCC Failed In Local Cluster | EMCCFailedInLocalCluster | */15 min |
| ControlHub004 | EMCC Failed In Remote Cluster | EMCCFailedInRemoteCluster | */15 min |
| ControlHub005 | ILS Duplicate URI | ILSDuplicateURI | */15 min |
| ControlHub006 | ILS Peer Limit Approaching | ILSPeerLimitApproachingWarning | */15 min |
| ControlHub007 | ILS Peer Limit Exceeded | ILSPeerLimitExceeded | */15 min |
| ControlHub008 | ILS Pwd Authentication Failed | ILSPwdAuthenticationFailed | */15 min |
| ControlHub009 | ILS TLS Authentication Failed | ILSTLSAuthenticationFailed | */15 min |
| ControlHub010 | Recording Gateway Registration Timeout | RecordingGatewayRegistrationTimeout | */15 min |
| ControlHub011 | Recording Resources Not Available | RecordingResourcesNotAvailable | */15 min |
| ControlHub012 | Location Out Of Resource | LocationOutOfResource | */15 min |
| ControlHub019 | Duplicate Learned Pattern | DuplicateLearnedPattern | */15 min |
| ControlHub021 | Dir Sync Connection Failed | Dir Sync Process Connection Failed | */15 min |
| ControlHub022 | Dir Sync Failed Retry | Dir Sync Process Failed Retry | */15 min |
| ControlHub024 | ILS Hub Cluster Unreachable | ILSHubClusterUnreachable | */15 min |
| ControlHub025 | Analog POTS Cable Reinserted | Analog POTS Cable Reinserted On Port | */15 min |
| ControlHub026 | Analog POTS Cable Removed | Analog POTS Cable Removed On Port | */15 min |
| ControlHub027 | VGW MGCP Process Shutdown | VGW MGCP Process Shutdown | */15 min |
| ControlHub029 | Media List Exhausted | MediaListExhausted | */15 min |
| ControlHub030 | Recording Session Terminated | RecordingSessionTerminatedUnexpectedly | */15 min |
| ControlHub031 | Registered Devices Exceeded | NumberOfRegisteredDevicesExceeded | */15 min |
| ControlHub032 | Route List Exhausted | RouteListExhausted | */15 min |
| ControlHub035 | Recording Gateway Reg Rejected | RecordingGatewayRegistrationRejected | */15 min |
| ControlHub037 | Excessive Voice Quality Reports | ExcessiveVoiceQualityReports | */15 min |
| ControlHub038 | Call Attempt Blocked By Policy | CallAttemptBlockedByPolicy | */15 min |
| WxCDI010 | Last Out Of Service Information | Last Out Of Service Information | */15 min |
| WxCDI118 | CDR Agent Send File Failure Recover | CDRAgentSendFileFailureContinues Recover | */10 min |
| WxCDI137 | Voicemail Port Registered | Voicemail Port Registered | */10 min |
| WxCDI143 | CPNS Access Token Invalid | AccessTokenInvalid | */10 min |
| WxCDI145 | CAR Scheduler Job Failed | CARSchedulerJobFailed | */10 min |
Disabled Alerts (16) — click to expand
| Alert ID | Alert Name | Reason |
|---|---|---|
| ControlHub018 | TranConnError | Disabled |
| ControlHub020 | CallManagerFailure | Disabled |
| ControlHub028 | CARSchedulerJobFailed | Disabled |
| ControlHub033 | CDRHighWaterMarkExceeded | Disabled |
| ControlHub034 | CDRMaximumDiskSpaceExceeded | Disabled |
| ControlHub036 | Extension Mobility Login | Disabled |
| WxCDI001 | Sample Alert | Test alert |
| WxCDI009 | SIP Trunk OOS | Duplicate of Mimir alert |
| WxCDI133 | DeviceTypeMismatch | Disabled |
| WxCDI134 | PhoneUnregThresholdExceeded | Disabled |
| WxCDI135 | SystemVersionMismatched | Disabled |
| WxCDI136 | TranConnError | Disabled |
| WxCDI139 | Trunk is in service | Informational, disabled |
| WxCDI140 | Backup completed successfully | Informational, disabled |
| WxCDI144 | RTMTSessionsExceedsThreshold | Disabled |
Notification Flow
Grafana Alert Rules → EMS v2 (KeepHQ) → Notification Channels
Splunk Alerts → EMS v2 (KeepHQ) → Notification Channels
All alerts include runbook and metrics documentation links in annotations.
Inventory
- Repository: wxcdi-inventory-files
- Update Method: Git commit (PR merge to
main) - Sync Mechanism: git-sync process loads inventory to Prometheus disk within ~10 minutes of commit
- Audit Trail: Fully audited via Git history
Regional Inventory Paths:
| Region | Inventory Path |
|---|---|
| US | inventory/us/cucm |
| EMEA | inventory/emea/cucm |
| APAC | inventory/apac/cucm |
| UAE | inventory/uae/cucm |
Onboarding / Offboarding: Add or remove inventory files via PR, merge to main, and the device is picked up (or dropped) by Prometheus within 10 minutes via git-sync. No manual restart required.