The Numbers
The Setup
Frank and Trish were in the middle of a routine task: renewing the IBM Instana NFR license for the Arrow Experience Center lab environment. The old entitlement was expiring that day. Using stanctl license download on instana-node-01, they had just downloaded the new license file when they discovered the correct command was stanctl license update — not stanctl license apply as initially assumed.
At that exact moment, IBM Bob had just been wired to the Instana MCP server — a direct API integration that gives Bob live read access to Instana's infrastructure monitoring, events, and application metrics. The license update ran, Core and Unit reconciled cleanly, and the team confirmed the renewal was successful.
IBM Bob connects to IBM Instana through a Model Context Protocol (MCP) server — a structured API bridge that lets Bob query live observability data directly in conversation. No copy-paste from dashboards. No context switching. Bob can ask Instana "what's wrong right now" the same way an engineer would — except Bob does it in milliseconds and never misses an alert.
With the MCP connection live and the license confirmed, Bob ran a routine environment health check. That's when the alert came in.
The Alert
Bob's first MCP query after the license renewal returned a single open event — but it was critical:
// Instana MCP → get_events (live query)
{
"problem": "You will run out of disk space in about 8 minutes",
"severity": 10,
"state": "open",
"entity": {
"label": "compute-1-ru6.fusion1.ibm.aessatl.arrow.com"
},
"affectedMetrics": ["fs./dev/sda4.free"],
"detail": "Clean up space on '/dev/sda4' or kill processes that fill up the disk."
}8 minutes to full disk on a production OpenShift worker node. That node was running two KubeVirt virtual machines (the cp4ba3 and MAS31 guest clusters), a Prometheus instance, OVN networking, and dozens of Cloud Pak for Data pods. A full disk would trigger cascading pod evictions across all of them.
Bob immediately cross-referenced the AEC knowledge base: compute-1-ru6 is on fusion1 (172.20.21.10), part of the IBM Fusion HCI bare-metal OpenShift cluster. Direct SSH access from Bob's workstation was not available — the only path in was oc.
Root Cause Analysis
Bob queried the OCP API directly using the existing oc session. The first thing that surfaced told the full story before a single file was examined:
$ oc describe node compute-1-ru6.fusion1.ibm.aessatl.arrow.com | grep -E "DiskPressure|Eviction"
DiskPressure True ... KubeletHasDiskPressure kubelet has disk pressure
Taints: node.kubernetes.io/disk-pressure:NoSchedule
Warning EvictionThresholdMet 4m (x1127 over 24d) kubelet Attempting to reclaim ephemeral-storage1,127 eviction events over 24 days. This wasn't a sudden spike — the node had been oscillating at the edge of disk capacity for nearly a month, recovering just enough to avoid sustained impact before filling again.
Bob accessed the node through the machine-config-daemon pod — a privileged daemonset that runs on every RHCOS node with full host filesystem access — and mapped the entire disk layout:
/var/lib/containers/storage/overlay
199 GB — CRI-O container image and layer cache. Dozens of ContainerStatusUnknown pods from clusters-cp4ba3 and clusters-mas31 (84 days old) had left orphaned overlay layers that GC couldn't reclaim while the node oscillated through eviction cycles.
/var/lib/kubelet/pods
153 GB — KubeVirt VM disk images. Two guest cluster VMs are permanently scheduled on this node: virt-launcher-cp4ba3pool and virt-launcher-mas31pool. Each carries a 50–80 GB virtual disk in kubelet pod storage. Expected — but no headroom left once overlay bloat accumulated.
/var/log/pods — 7.2 GB
Top log producers: ovnkube-node (709 MB), rbd-csi-ctrlplugin (261 MB), prometheus-k8s-0 (212 MB), and three replicas of wo-agentic-task-manager (~300 MB combined). Not the primary cause, but accelerating the fill rate.
Spark Runtime Preload Jobs
Scheduled spark-hb-preload-jkg-* jobs pull 6–8 large runtime images on a recurring schedule (miniconda, miniforge, wxd runtimes). Each pull lands in the overlay cache and isn't immediately GC'd — the periodic refill driving the 24-day oscillation.
Remediation
Bob identified the safe, non-disruptive action: prune the orphaned CRI-O container layers and dead container records. No pod evictions, no VM migrations, no node drain required.
Remove 133 exited containers
Bob collected all container IDs in Exited state via crictl ps -a --state exited and bulk-removed them through the machine-config-daemon privileged exec. 133 dead containers cleared, freeing their writable layer references so the image GC could proceed.
crictl rm $(crictl ps -a --state exited -q)
# 133 containers removedCRI-O image prune
With dead container references cleared, Bob ran crictl rmi --prune to remove all unreferenced image layers from the overlay store. CRI-O hit a few DeadlineExceeded timeouts on edge cases under load, but the bulk of orphaned layers were successfully evicted.
crictl rmi --prune
# Pruned orphaned overlay layers
# (some DeadlineExceeded on 4 images under load — non-critical)35 GB freed — DiskPressure cleared
Disk dropped from 85% (378 GB used) to 77% (343 GB used). The kubelet detected sufficient free space within its next heartbeat cycle and cleared the DiskPressure condition and disk-pressure:NoSchedule taint automatically.
$ oc get node compute-1-ru6... -o jsonpath='{.status.conditions[*]}'
DiskPressure False kubelet has no disk pressure
Ready True kubelet is posting ready statusOutcomes
Why This Matters
This wasn't a demo scenario. Frank and Trish were doing completely unrelated work — a license renewal — when a production incident fired. The reason Bob caught it wasn't luck. It was the direct consequence of wiring Bob to Instana's MCP server moments earlier.
Traditional workflow: alert fires → lands in someone's inbox → ticket opened → engineer investigates → diagnosis takes 30–60 minutes → remediation applied. Total time: hours, often next business day.
What happened here: alert fires → Bob sees it in the same conversation → root cause in 4 oc commands → remediated. The entire incident — detection through resolution — happened within a single working session, without interrupting the original task.
The 24-day oscillation pattern has a structural root cause: two KubeVirt VMs permanently consuming 153 GB on a node with a fixed 447 GB partition, leaving insufficient headroom for the Spark runtime preload jobs' image churn. Bob documented the options for the application owner: migrate one KubeVirt VM to another worker node to rebalance pod storage, and lower CRI-O's imageGCHighThresholdPercent to trigger automatic GC before kubelet hits eviction threshold. That conversation is next.
IBM Bob + Instana MCP: The Integration
The Instana MCP server is an open-source Model Context Protocol server that exposes Instana's observability API as structured tools Bob can call natively — infrastructure analysis, application metrics, event queries, SLO data, and more.
In this session, Bob used the MCP server to:
- Query all open events across the environment after the license renewal
- Retrieve detailed event metadata including the EvictionThresholdMet history (1,127 events over 24 days)
- Cross-reference the alert against the AEC infrastructure knowledge base to identify the node's role, IP, and running workloads
- Confirm DiskPressure cleared post-remediation without leaving the conversation
The MCP integration means Bob doesn't just know about your infrastructure — it can see it live. That's the difference between an assistant that answers questions and one that catches problems you didn't know to ask about.