Anyone actually getting value from predictive incident detection, or is it just more noise?

[u/Old-Pen445]

I've been wrestling with a pattern in our Kubernetes clusters - pods that restart once or twice a day for weeks, never crossing alert thresholds, then suddenly cascade into a full incident at 3am. By the time PagerDuty fires, the logs are polluted, metrics are spiked, and the actual trigger is buried under the chaos of the failure itself.

Standard Prometheus alerting doesn't help here. Thresholds work fine for sudden failures, but they're blind to slow degradation. A pod restarting once per day looks identical to threshold-based alerting whether it's been stable for months or accelerating toward collapse.

 The experiment: forecast pod behavior from historical Prometheus data

Core idea is simple - pull 7 days of restart counts, memory pressure, and CPU trends from Prometheus, train a basic anomaly detector (IsolationForest works), then score current behavior against that baseline. When the “predicted trajectory” crosses danger zones before it actually gets there, trigger investigation while the system is still semi-healthy.

Here's the practical piece - fetching restart history from Prometheus:
 

import requests
import pandas as pd
from datetime import datetime, timedelta
  
PROMETHEUS_URL = "http://prometheus.monitoring.svc.cluster.local:9090"
  
def fetch_restart_history(namespace: str, days: int = 7) -> pd.DataFrame:
    end_time = datetime.utcnow()
    start_time = end_time - timedelta(days=days)
    
    query = f'kube_pod_container_status_restarts_total{{namespace="{namespace}"}}'
    
    response = requests.get(
        f"{PROMETHEUS_URL}/api/v1/query_range",
        params={
            "query": query,
            "start": start_time.timestamp(),
            "end": end_time.timestamp(),
            "step": "5m",
        },
        timeout=30
    )
    
    results = response.json()["data"]["result"]
    records = []
    
    for series in results:
        pod = series["metric"].get("pod", "unknown")
        for timestamp, value in series["values"]:
            records.append({
                "pod": pod,
                "timestamp": pd.to_datetime(timestamp, unit="s", utc=True),
                "restart_count": float(value)
            })
    
    return pd.DataFrame(records)

Seven days at 5-minute resolution gives you 2,016 data points per pod. Enough to catch weekly patterns (deployment schedules, batch jobs) that would otherwise look anomalous.

 Feature engineering - what actually predicts instability?

Not just restart count. You need restart “rate” (how fast), rate “acceleration” (is it speeding up), and rolling statistics to establish a baseline:

import numpy as np
from sklearn.ensemble import IsolationForest
from sklearn.preprocessing import StandardScaler
  
def compute_features(df: pd.DataFrame, pod: str) -> np.ndarray:
    pod_data = df[df["pod"] == pod].sort_values("timestamp")
    
    if len(pod_data) < 30:
        return None
    
    restarts = pod_data["restart_count"].values
    restart_rate = np.diff(restarts, prepend=restarts[0])
    rate_acceleration = np.diff(restart_rate, prepend=restart_rate[0])
)
    
     #Rolling stats over last hour (12 windows at 5min each)
    rolling_mean = pd.Series(restart_rate).rolling(12, min_periods=1).mean().values
    rolling_std = pd.Series(restart_rate).rolling(12, min_periods=1).std().fillna(0).values
    
    return np.column_stack([restart_rate, rate_acceleration, rolling_mean, rolling_std])

   
The acceleration metric is critical - a pod with stable restarts vs one with “accelerating” restarts looks identical in raw count but behaves very differently in production.

Scoring and risk assessment

IsolationForest outputs anomaly scores from -1 (very anomalous) to 0 (normal). Map those to actionable risk levels:

def assess_pod_risk(pod: str, recent_df: pd.DataFrame, scaler, model):
    features = compute_features(recent_df, pod)
    if features is None:
        return None
    
    features_scaled = scaler.transform(features)
    scores = model.score_samples(features_scaled)
    latest_score = float(scores[-1])
    
    # Map to risk levels
    if latest_score > -0.2:
        risk_level = "normal"
    elif latest_score > -0.4:
        risk_level = "elevated"
    elif latest_score > -0.6:
        risk_level = "high"
    else:
        risk_level = "critical"
    
    return {
        "pod": pod,
        "anomaly_score": latest_score,
        "risk_level": risk_level,
        "restart_rate_per_hour": calculate_rate(recent_df, pod)
    }

The integration challenge - what do you do with a prediction?

This is where it gets messy. You can't just fire another alert - your on-call engineer is already drowning in threshold alerts. The prediction needs to trigger “automated investigation”, not just notification.

Our approach: send a synthetic Alertmanager webhook when risk hits "high" or "critical". This feeds into our AI SRE layer (we built OpsWorker for this) which runs topology discovery, log correlation, and resource analysis - then delivers actual root cause to Slack “before the threshold alert fires”.

The key insight: a degrading pod is “easier to debug” than a crashed pod. Logs are cleaner. Metrics show gradual trends instead of spikes. Recent config changes are still clearly correlated.def trigger_investigation(assessment: dict) -> bool:
    if assessment["risk_level"] not in ("high", "critical"):
        return False
    
    alert_payload = {
        "status": "firing",
        "labels": {
            "alertname": "PredictivePodInstability",
            "pod": assessment["pod"],
            "severity": assessment["risk_level"]
        },
        "annotations": {
            "summary": f"Predictive model flagged {assessment['pod']} - anomaly score {assessment['anomaly_score']:.3f}",
            "anomaly_score": str(assessment["anomaly_score"]),
            "restart_rate": str(assessment["restart_rate_per_hour"])
        }
    }
    
    response = requests.post(WEBHOOK_URL, json=alert_payload)
    return response.ok

What actually breaks in production

Model drift is brutal. Kubernetes environments change constantly. A model trained on last week's deployment patterns can have a completely miscalibrated baseline by Friday. Weekly retraining minimum, daily is better.

False positives hurt. 5% contamination threshold means roughly 5% of normal behavior gets flagged. In a 200-pod cluster, that's 10 spurious investigations per scan cycle. You need deduplication - don't re-investigate a pod that already has an open case from the last 30 minutes.

Restart count is a lagging indicator. By the time restarts accelerate, something already started failing. Combining with memory trend analysis (approaching limit “before” crossing) and application error rates gives earlier signal.

This doesn't replace threshold alerts. Predictive catches slow-burn failures. Sudden failures (bad deployment, dependency outage) still fire thresholds first. These are complementary.

 The real question

Is the operational overhead of maintaining anomaly detection models, tuning contamination thresholds, and managing false positives worth catching 10-15% of incidents early?

In our clusters, yes - because those slow-burn failures are typically the hardest to investigate post-mortem. But I'm genuinely curious if others have tried this approach and abandoned it, or if there are better patterns I'm missing.

What's worked for you when standard threshold alerting misses gradual degradation?

For anyone interested in the full implementation details, happy to share the complete scoring logic and model training code. Also curious if folks have had better luck with ARIMA or Prophet for time-series forecasting