The Numbers
The Problem Bob Solved
Every AI development session on the AEC's on-premises watsonx Orchestrate environment required a manual checklist before a single line of agent code could be written: open Keeper to find the Vault password, authenticate to HashiCorp Vault, load a dozen environment variables from different secret paths, verify GlobalProtect was connected, log into the OpenShift cluster, and activate the wxO ADK environment. Miss one step and the tools wouldn't work. Do them out of order and the session would fail partway through.
This wasn't a technical limitation — it was a developer experience problem. The time-to-first-keystroke on actual work was 5–10 minutes of ritual. Across a sprint, that's real time lost. And for anyone new to the environment, the cognitive load of the setup was enough to derail momentum before the work had even started.
🤖 BOB AS EFFICIENCY EXPERT
Bob's role here wasn't writing application logic — it was acting as a systems efficiency expert. Bob read the existing scripts, understood the full credential architecture spanning four separate systems (Keeper for personal secrets, Vault for enterprise secrets, OCP for cluster access, ADK for AI platform activation), identified the logical dependency chain between them, and designed a single sourced shell script that handles every step with proper error handling, idempotent checks, and zero credentials written to disk. It also caught two silent bugs that would have passed code review but failed developers at 9am — before the implementation was finished. What would have taken a developer a full day to design and test, Bob designed, built, debugged, and iterated in a single session.
The Gate Sequence
Bob designed a five-gate startup protocol. Each gate validates a prerequisite before proceeding. Any failure gives a clear, actionable error — not a wall of cryptic stack traces. The sequence maps exactly to the credential dependency chain: you need Keeper to unlock Vault, you need VPN to reach Vault, you need Vault credentials to log into OCP, and you need secrets loaded before activating the ADK.
Keeper Commander — Personal Credential Vault
Validates the developer's Keeper Security session with keeper whoami. If the session has expired, it automatically launches klog — the dedicated Keeper login script Bob built — which handles SSO + TOTP with a 30-day 2FA duration. This means after the initial setup, this gate passes silently for a month at a time. Keeper is the root of the credential chain: the Vault password, the OCP password, and all service credentials live here.
GlobalProtect VPN — Network Reachability
Tests a raw TCP connection to the AEC Vault endpoint (172.20.28.37:8200) with a 3-second timeout. If this fails, the script stops immediately with instructions to connect GlobalProtect — rather than proceeding to every subsequent gate and producing confusing network errors. Fail fast, fail clearly: this one gate eliminates an entire category of "why isn't this working" debugging.
HashiCorp Vault — Enterprise Secrets, Zero Disk Exposure
Checks for an active Vault token first — if already authenticated the gate passes instantly. If not, it pulls the Vault password from Keeper automatically, performs an LDAP login to the AESSATL domain, and loads all 17 project secrets into the shell session. The password is extracted from Keeper's JSON output in-memory, used once, and cleared — it never appears in a terminal, shell history, or log file.
✅ IBM_WATSON_ORCHESTRATE_API_KEY · IBM_WATSON_ORCHESTRATE_URL
✅ WATSONX_APIKEY · WATSONX_PROJECT_ID · WATSONX_URL · WATSONX_USERNAME
✅ AZURE_TENANT_ID · AZURE_CLIENT_ID · AZURE_CLIENT_SECRET
✅ MILVUS_GRPC_HOST · MILVUS_PASSWORD · MILVUS_DATABASE · +4 more
── 17 secrets loaded to memory — 0 written to diskOpenShift Login — Live Token Validation
Logs into the AEC bare-metal fusion1 OpenShift cluster — where watsonx runs on L40S GPUs — using credentials from Keeper. On success, switches the active namespace to cpd-watsonx-wxo so every oc command is scoped correctly from the first keystroke.
🐛 BUG CAUGHT BY BOB
The original OCP auth check used oc whoami --show-server to detect an active session. Bob identified that this reads the kubeconfig file on disk and returns the server URL even when the token is expired — a silent false positive that skipped re-authentication entirely, causing every downstream oc command to fail with "the server has asked for the client to provide credentials." Bob replaced the check with oc whoami (no flags), which calls the live API and correctly detects an expired token.
watsonx Orchestrate ADK — Activate and Verify
Registers and activates the wxO ADK environment using credentials loaded in Gate 3 — no manual API key entry. Then runs a live HTTP probe against the wxO endpoint and an Azure Graph token test to confirm both AI platforms are reachable before declaring the session ready.
🐛 BUG CAUGHT BY BOB
The original connectivity test attempted from ibm_watsonx_orchestrate import Client — a class that doesn't exist in this SDK version. Bob correctly identified that the ADK is CLI + decorator-based (no Python client object exists), then replaced the broken import with a plain requests.get() HTTP probe against the wxO route using the API key as a Bearer token. Zero SDK version dependency. Same connectivity signal — a real HTTP 200 from the live cluster.
✅ wxO endpoint reachable — HTTP 200
✅ Token obtained — type=Bearer, expires_in=3599sWhat the Developer Sees
The entire sequence is triggered by a single alias. The developer types wxo and watches the gates pass. When it ends, every credential is loaded, the cluster is connected, the ADK is active, and the project directory is set. Time from cold terminal to writing agent code: under 20 seconds on a warm session.
$ wxo
╔══════════════════════════════════════════════════════════════╗
║ 🤖 watsonx Orchestrate — AI Dev Session ║
║ Arrow Experience Center · On-Prem · Project ARIA ║
╚══════════════════════════════════════════════════════════════╝
── Gate 1 — Keeper Commander
✅ Keeper authenticated (frank.welder@arrow.com)
── Gate 2 — VPN / AEC Reachability
✅ AEC is reachable — GlobalProtect VPN confirmed connected
── Gate 3 — Vault Authentication
✅ Vault already authenticated (fwelder)
✅ 17 secrets loaded into shell session (memory only — not on disk)
── Gate 4 — OpenShift Login (fusion1)
✅ OCP authenticated (fwelder @ fusion1)
✅ Active namespace (cpd-watsonx-wxo)
── Gate 5 — watsonx Orchestrate ADK
✅ wxO environment active (aria)
✅ wxO endpoint reachable — HTTP 200
✅ Azure Graph token obtained — type=Bearer
╔══════════════════════════════════════════════════════════════╗
║ 🚀 AI Dev Session Ready ║
╚══════════════════════════════════════════════════════════════╝The Companion Script Family
The wxo command is backed by a family of purpose-built scripts wired into .zshrc as a complete AI development toolkit:
klog
Keeper SSO login — 30-day 2FA duration, idempotent. Passes silently if already authenticated.
vlog
Vault LDAP login + SSH certificate signing. Credentials pulled from Keeper automatically.
wxlogin
Standalone OCP login to fusion1. Uses live API token validation — not stale kubeconfig.
wxcheck
Namespace health beacon — pods, PVCs, and route accessibility across both watsonx namespaces. Self-heals OCP auth before running.
wxexc
Exception-only cluster report — surfaces only failed pods and unbound PVCs for fast triage.
wxo-env
Prints which of the 17 project secrets are currently loaded — green checkmark or red circle for each variable.
Bob as Efficiency Expert — Not Just a Coding Partner
IBM Bob is most often showcased as a coding partner — writing functions, reviewing pull requests, generating tests. This use case demonstrates a different and equally valuable mode: Bob as a developer experience engineer.
The credential architecture here spans four separate systems — Keeper, HashiCorp Vault, Red Hat OpenShift, and the watsonx Orchestrate ADK — each with its own authentication model, session lifetime, and failure mode. Designing a reliable, idempotent connection protocol across all four required understanding not just the APIs, but the dependency relationships between them, the failure modes of each, and the developer experience implications of every design decision.
Bob brought all of that. It read the existing scripts before writing anything. It proposed the gate pattern. It identified the security implication of passwords in shell history and designed the in-memory extraction approach without being asked. And it caught two silent bugs through systematic analysis — not because it was told to look for bugs, but because it understood the system well enough to recognise what would go wrong at runtime.
The result is a developer environment that gets out of the way. The cognitive load of "am I connected, are my secrets loaded, is my cluster session alive" is gone. Developers open a terminal, type wxo, and they're building agents on IBM watsonx Orchestrate — on bare-metal on-premises infrastructure with enterprise-grade secrets management — before their coffee is cold.
That's the Bob difference: not just faster code, but more time doing the work that matters.
Technology Stack
IBM watsonx Orchestrate
On-premises CPD deployment on bare-metal fusion1 — agents, tools, and ADK on L40S GPU infrastructure
HashiCorp Vault
Enterprise secrets management — LDAP auth, KV engine, SSH certificate signing
Keeper Security
Personal credential vault — SSO + TOTP, 30-day 2FA, keeper-commander CLI
Red Hat OpenShift
Bare-metal cluster on IBM Fusion HCI — cpd-watsonx + cpd-watsonx-wxo namespaces
GlobalProtect VPN
Palo Alto Networks — AEC network gateway, validated as Gate 2 before any cluster calls
Microsoft Azure + Graph
Personal tenant app registration for M365 agent tool connections — verified live in Gate 5