Problem
When an OpenShift cluster incident occurs, the time from alert to root cause identification is dominated by manual correlation across metrics, events, logs and configuration state. SREs must manually cross-reference Prometheus alerts, Kubernetes event streams, node conditions and application logs before they can produce a coherent root cause hypothesis.
This pattern structures the RCA process as a sequential reasoning funnel — moving from deterministic signal collection through event correlation to AI-assisted explanation, with a mandatory human approval gate before the analysis is promoted to a formal RFO.
Context and Assumptions
- OpenShift cluster with Prometheus and Alertmanager configured
- Log aggregation via Elasticsearch or OpenShift Logging operator
- Event collection via Kubernetes API or event exporter
- AI explanation layer is LLM-backed with retrieval from historical RCA corpus
- Approval workflow is integrated with existing ITSM (ServiceNow or equivalent)
Reference Architecture
Architecture Diagram
Component Responsibilities
Component Responsibilities
Signal Collector
Gathers Prometheus metrics, Kubernetes events, node conditions and pod lifecycle events at incident time.
Deterministic Checker
Runs fixed rule-based checks: OOMKilled pods, eviction thresholds, CrashLoopBackOff counts, PVC binding failures, node NotReady conditions.
Correlation Engine
Correlates signals across time window. Identifies causal chains using rule-based temporal ordering.
AI Explanation Layer
Generates natural-language hypothesis from correlated signals. Retrieves similar historical incidents from corpus.
Evidence Pack Builder
Assembles structured evidence: signal timeline, correlated events, AI hypothesis, similarity matches.
Human Approval Gateway
SRE reviews evidence pack, accepts or modifies hypothesis, approves for promotion to RFO.
What Is Deterministic vs. AI-Assisted
Deterministic vs. AI-Assisted
Deterministic
- Prometheus alert rule evaluation
- OOMKilled / eviction / CrashLoop detection
- Kubernetes event time-ordering
- Evidence pack assembly
- Human approval workflow routing
✦AI-Assisted
- Natural-language hypothesis generation
- Cross-signal causal narrative
- Historical incident similarity matching
- Impact scope estimation
- Draft RFO text generation
Reference patterns must be adapted to the customer environment, product versions, security policies and vendor architecture.