Observability ¶

Where to watch EAP at runtime and what it emits — metrics, logs, traces, and alerting.

Metrics ¶

All EAP dashboards live in the EAP folder on Grafana.

Resource Metrics ¶

From the Kubed baseline instrumentation — CPU, memory, disk, etc.

Service Health Metrics ¶

EAP Health Metrics ¶

LLM Call Metrics ¶

Codex & OpenCode Metrics ¶

Emitted by the Codex and OpenCode OTel exporters.

Convention — histogram instruments expose three Prometheus series each (_bucket, _count, _sum), written below as _{bucket,count,sum}.

OpenCode Metrics ¶

opencode_cache_count_total
opencode_cost_usage_USD_total
opencode_message_count_total
opencode_model_usage_total
opencode_retry_count_total
opencode_session_count_total
opencode_session_cost_total_USD_{bucket,count,sum}
opencode_session_duration_milliseconds_{bucket,count,sum}
opencode_session_token_total_tokens_{bucket,count,sum}
opencode_token_usage_tokens_total
opencode_tool_duration_milliseconds_{bucket,count,sum}

Codex Metrics ¶

codex_api_request_duration_ms_milliseconds_{bucket,count,sum}
codex_api_request_total
codex_plugins_startup_sync_final_total
codex_plugins_startup_sync_total
codex_process_start_total
codex_remote_models_load_cache_duration_ms_milliseconds_{bucket,count,sum}
codex_shell_snapshot_duration_ms_milliseconds_{bucket,count,sum}
codex_shell_snapshot_total
codex_sqlite_init_count_total
codex_sqlite_init_duration_ms_milliseconds_{bucket,count,sum}
codex_sse_event_duration_ms_milliseconds_{bucket,count,sum}
codex_startup_phase_duration_ms_milliseconds_{bucket,count,sum}
codex_thread_skills_description_truncated_chars_{bucket,count,sum}
codex_thread_skills_enabled_total_{bucket,count,sum}
codex_thread_skills_kept_total_{bucket,count,sum}
codex_thread_skills_truncated_{bucket,count,sum}
codex_thread_started_total
codex_tool_call_duration_ms_milliseconds_{bucket,count,sum}
codex_tool_call_total
codex_tool_unified_exec_total
codex_turn_e2e_duration_ms_milliseconds_{bucket,count,sum}
codex_turn_memory_total
codex_turn_network_proxy_total
codex_turn_token_usage_{bucket,count,sum}
codex_turn_tool_call_{bucket,count,sum}
codex_turn_ttfm_duration_ms_milliseconds_{bucket,count,sum}
codex_turn_ttft_duration_ms_milliseconds_{bucket,count,sum}

Alerting ¶

EAP gateway alerts: alerting list (service=eap-gateway).

The alert rules are defined in eap-k8s-alert-rules.json and eap-health-status-alert-rules.json. Grafana evaluates both rule groups every 60 seconds. All rules use service=eap-gateway, link to the eap-health-status dashboard, treat a Grafana no-data result as OK, and report rule evaluation errors as Error. Missing telemetry is detected by the explicit health-metrics-missing rules.

Kubernetes Resource Alerts ¶

These rules cover the enterprise-agent-platform (prod) and enterprise-agent-platform-stage namespaces. They retain the cluster, namespace, and affected Kubernetes resource labels so notifications identify the exact container or PVC.

Health Status and LLM Alerts ¶

Unless a row names a specific environment, these rules cover all eap-gateway metric series available to the alert datasource and retain labels such as cluster, namespace, pod, component, endpoint, method, or path for diagnosis.

Logs ¶

Traces ¶

OpenSearch ¶

Portal: OpenSearch Portal.

Tempo + Grafana (LMA) ¶

How to Find Traces ¶

LMA Sample ¶

Step 1 — Get the Conversation ID ¶

Every trace is keyed by a conversation_id. The easiest way to obtain it is directly from the EAP page:

1. Open your browser's developer console.
2. Refresh the page and select the ACP request (see the image below).
3. Copy the conversation_id from the conversation you want to track.

Step 2 — Search for the Trace in Grafana ¶

1. Open Grafana and go to the Explore page.
2. Select the tempo-ccps-poc datasource.

3. Query by the conversation_id you copied, for example:

   {span.eap.conversation_id="conv_6053668dd24644eea2857e3b3670740a"}
   

4. Click the matching trace to inspect its details.

Step 3 — Inspect an MCP Tool Call ¶

To isolate an MCP tool call within a trace, filter by an attribute such as target = codex_core::mcp_tool_call and turn off the Show all spans switch. The tool calls then stand out clearly.

What Conversation Issues Can Be Diagnosed? ¶

• End-to-end tracing across systems — Using eap.conversation_id or codex.thread_id, you can follow the full timeline of a single conversation in one trace—from the gateway receiving the prompt, to forwarding it to Codex, through Codex's internal processing—without manually stitching IDs together from logs.
• Latency and timeout root cause — Compare the duration of the acp.prompt span with its child Codex spans to tell whether the bottleneck is on the gateway side (e.g., authentication, binding/reconnection) or inside the Codex runtime.
• Failing conversation turn — Because each prompt turn maps to a span, the conversation_id pinpoints the user and the exact interaction that failed, while child spans locate the point of failure within Codex.
• Reconnection and binding anomalies — Events from the replace_binding reconnection logic within a prompt appear directly in the span timeline.