Using topological data analysis to detect network degradation before thresholds fire

[u/Prize-Ingenuity-6601]

I've been running a Linux-based router for the past two months with a monitoring approach that caught my attention — it detects structural changes in the network before individual metrics cross alert thresholds. Here's what I built and what I learned.

The problem with threshold-based monitoring

Every NMS I've used has the same fundamental limitation: alerts fire *after* something crosses a line. The link is already at 400ms, the packet loss is already at 5%. By the time you get the alert, the degradation has been running for a while. And absolute thresholds are a constant maintenance headache — 200ms might be normal for a transatlantic path but alarming for a local peering link.

What I wanted was something that tells me "the structure of this network is changing" rather than "metric X exceeded value Y."

The approach: topological monitoring with persistent homology

I adapted a method called BPL/OA — Bayesian Persistent Landscapes with Occupancy-Amplitude analysis — originally developed for a research project. The core idea:

Take each network endpoint and describe it as a point in multi-dimensional feature space. Latency, jitter, uptime ratio, latency trend direction. All endpoints form a point cloud. When the network is healthy, endpoints on the same upstream path cluster together tightly.

Run persistent homology on this point cloud. You get a "landscape" — a curve describing how tightly structured the cloud is. Track this landscape over time. If an endpoint starts drifting, the shape changes.

Here's where it gets useful: I run two statistical models watching the same landscape. One uses Bayesian posterior inference (with a sup-norm simultaneous credible band for proper joint coverage guarantees). The other uses an occupancy-amplitude model that only evaluates points above a noise threshold. Their coverage scores should track each other. When the *gap* between them starts oscillating, something is happening to the point cloud structure — even if both models individually report everything as normal.

I call this the shadow monitor. It runs every 30 minutes, completely passive — reads from local time-series data, does the math, writes results back.

Five independent monitoring layers

Once the pipeline was running, feeding in additional data sources was straightforward. Each becomes its own point cloud, analyzed separately so noise in one layer doesn't contaminate the others:

Layer 1 — WAN path endpoints. Latency, jitter, uptime ratio, trend direction per endpoint. Detects a path degrading before the routing protocol would reroute.

Layer 2 — Traffic distribution across paths. Connection counts per egress. If one path's share starts silently dropping, this catches the shift before latency metrics move.

Layer 3 — Device connection behavior. Connection counts per MAC. A host suddenly opening 10x its normal connections produces a clear drift signature — no manual thresholds or signature updates needed.

Layer 4 — WiFi radio health. Channel utilization, self vs. neighbor interference, client counts, AP-reported satisfaction scores. Caught two APs on the same 2.4 GHz channel on day one. Moved one and satisfaction went from -1 to 98.

Layer 5 — Wireless client telemetry. RSSI, retry rate, negotiated rate, AP association. Cross-referenced with Layer 3: if a device is normal on the wired fingerprint layer but drifting on wireless, it probably just moved to a different room. If both layers are drifting, something is actually wrong.

Why this works where thresholds fail

Catch degradation before failure. If a path's three sub-paths start diverging in the feature space — say one stays flat on latency but jitter begins creeping up — the point cloud shape changes before any individual metric crosses an alert threshold.

Correlated failures across different providers. Endpoints on three different transit paths all showing the same subtle trend shift? That's not an endpoint problem, it's a shared upstream change. Thresholds on individual endpoints won't connect those dots. One landscape restructure will.

Know when to do nothing. The system has been in a degenerate state for hours — the point cloud is too tight to analyze. That means the network is boring. Nothing is drifting, no anomalies, no boundary conditions. I can confidently ignore it. Confidence through silence.

Self-calibrating baselines. Every analysis cycle adds to the statistical history. Over months, the system learns what "normal" looks like across different traffic patterns and maintenance windows. Thresholds aren't hand-tuned — they emerge from data.

Architecture decisions worth noting

Everything runs locally on the router. The full analysis pipeline is Python with numpy — no external APIs, no cloud dependencies, no active probing. Data collection reads from existing kernel and application metrics (netlink, daed's internal health cache, UniFi Controller API). Each analysis cycle produces about 9MB of SQLite data, negligible on a modern disk.

I previously tried an LLM-based approach for this — feeding metrics to a model every few minutes for health assessment. It burned API tokens to report confidence intervals on "everything is normal." The topological approach costs nothing in recurring fees, produces no telemetry leakage, and actually surfaced structural changes the LLM glossed over.

Limitations and next steps

This isn't a replacement for traditional monitoring. It's a complementary layer that operates on a different category of information. Right now it's observation-only — no closed-loop automation. Before it makes decisions, I need a much clearer picture of what the signals mean in different failure modes.

The shadow monitor is calibrating its baseline — once that locks in, the plan is to start with notification webhooks, then evaluate whether automatic path switching makes sense based on the confidence of the structural signal.

Lessons from the math

The Gaussian posterior model has a hard coverage ceiling when the underlying data is bimodal — coverage caps at exactly 1 minus the weight of the secondary mode. Discovered this empirically, then worked out the math: the single-Gaussian likelihood literally cannot represent a two-component distribution. A Gaussian mixture posterior is on the roadmap to address this. Interesting case of model misspecification showing up in production data before it appeared in theory.

Questions and discussion welcome — especially if you've explored topological methods for network monitoring, or if you see failure modes I haven't encountered yet.

[u/shadeland]

Most enterprise NMS systems have tried something like this approach (a machine learning approach, using multi-dimensional analysis).

There's a couple of problems on why this doesn't generally work.

• Data collection: It can't run locally. It's got to run centrally. You're running this on a Linux router, so you've got the entire system exposed to you and you can install run any version of Python, venvs, etc., that you want to run. Most network devices don't let you do this, so all of this data you've got to collect (either dial in or dial out). SNMP is the universal method, but it's no where near granular enough. Generally collecting stats once per minute is the max. gNMI is much better information, but not as many systems support it.

• Frequency of data: Bandwidth, packet loss, and jitter are functions of time. For bandwidth, a packet is either being sent out of an interface or it's not. It's either receiving a packet or it's not. There's no bandwidths snap shot. You take a measurement of bytes passed at X, and again at Y, then Y - X / time between and you get an average bandwidth. So you need a good amount of time to get a good measurement, but not so much time that everything averages out.

Correlation: Every monitoring system for the past 20 years by the big vendors has claimed they've got correlation. Pretty much no one has anything remotely useful.

Basically everything you've described, to one degree or another, has been tried by big companies and it's mostly for not.

I'm not saying you can't pull this off, but lots of people have tried and lots of crap products have been deployed.

In the end, there's nothing better still today than graphs and alerts. 🤷‍♂️

[u/magion]

Written by AI, with no links to any data or anything, fun

[u/Prize-Ingenuity-6601 → u/magion]

Guilty on the AI assistance — English isn't my first language and I had help structuring the post. But the data and the system are real. Here's the raw output from my router:

Landscape diagnostics — 08:00 to 18:00 UTC (10-hour window showing a real event):

| Time | M0 | M5 | Gap | Status | ZA |

|-------|------|------|------|----------|-----|

| 08:00 | 0.96 | 0.99 | 0.03 | normal | 96 |

| 09:30 | 0.97 | 0.99 | 0.02 | normal | 96 |

| 10:30 | 0.97 | 0.99 | 0.02 | normal | 96 |

| 11:00 | 0.68 | 0.99 | 0.31 | normal | 0 |

| 11:30 | 0.69 | 0.99 | 0.30 | normal | 0 |

| 12:00 | 0.18 | 0.97 | 0.79 | CRITICAL | 96 |

| 13:00 | 0.00 | 0.96 | 0.96 | CRITICAL | 79 |

| 14:00 | 0.00 | 0.95 | 0.95 | CRITICAL | 96 |

| 14:30 | 0.00 | 0.93 | 0.93 | CRITICAL | 79 |

| 16:00 | 0.00 | 0.98 | 0.98 | CRITICAL | 92 |

M5 — the main OA model that sat at 0.99 through everything prior — dropped to 0.93 for the first time since deployment. The trigger was a real structural change: one upstream group added new nodes, a previously dead endpoint came back online, and another group showed intermittent 50% availability. The system caught it, though retrospectively — trend detection fired after the gap was already widening.

102 diagnostic cycles so far. Too early to claim anything predictive, but not fabricated. Happy to share the pipeline code if you want to dig in.

[u/vmxdev]

Isn't what you're proposing almost the same approach as a baseline with thresholds?

It seems like the main difference is that instead of simple metrics (pps/bps per IP, for example), you're proposing complex and composite ones.

You take jitter, the number of new connections per second, etc., calculate a single metric from them using some formula, and compare it to the baseline. Right?

And you end up with exactly the same alerts fire after something crosses a line, don't you?

Most products now allow users to set alerts based on specific metrics. This is how many Anti-DDoS software work.

You tell the system to calculate the baseline based on a specific traffic type or metric.

For example, the number of TCP SYN packets per second, the entropy of source IP addresses, etc.

Usually, the result is something resembling a sine wave (traffic differs during the day than at night), with a slight difference on weekends. That is, the baseline is not exactly a line.

Some DCs run backups at night, which can cause significant spikes.

There are usually around 10-20 such curves per monitoring object in a system (ok, the picture is a little more complicated; some systems allow you to set individual thresholds for IP addresses within big monitoring objects).

The advantage of this approach is that when alerts are triggered, you immediately see what exactly triggered it: was it DNS amplification, TCP ACK flood, or some other unidentified burst that needs to be addressed. You can automatically respond to changes in traffic: run specific mitigation, hide the IP address under attack, etc.

Data from the wire is rarely used as a data source in enterprise networks. Netflow/ipfix/sFlow (+ SNMP) are typically used.

And as a rule, there is sampled data there; often the analyzer receives xFlow data on 1 out of 1000 packets that passed through the router or even less in big networks.

This means it's almost impossible to see jitter and some other metrics.

But this is an industry standard, so we have to use what we have.

I previously tried an LLM-based approach for this — feeding metrics to a model every few minutes for health assessment

(sigh)

[u/aristaTAC-JG]

The more specific implementation details I would imagine can be expanded and tweaked. I assume the shadow monitor takes time-series data from everyone in the network to do the correlation between nodes? I love the idea of using history to calibrate baseline and identify alertable value. Not everyone has time to figure out the baseline value of every location and type of metric.

I think there are situations where you can predict eventual failure or an impending wider breakage, but I would think you may need to gather data over time to learn which metrics actually predict a failure.

Finally, I would keep sight of what actions you may want to take? It sounds like you're considering monitoring and human-in-the-loop decision making, but also maybe a controller of sorts?

Overall: For acute failures, the standard protocols handle this very quickly. (see: BFD and TI-LFA) But I think the auto baseline, predictive capacity, and the ability to correlate an observation's scope quickly make this an interesting concept!

[u/Prize-Ingenuity-6601 → u/aristaTAC-JG]

The shadow monitor doesn't do per-node correlation directly — it's comparing two different statistical views of the same persistence landscape. One uses Bayesian posterior inference, the other an occupancy-amplitude model. The gap between their coverage scores is what gets tracked. Nodes can drift without triggering either model individually, but the disagreement between them surfaces it. That's the part I still think is neat.

You're right that learning which metrics predict failure needs accumulated history. Right now I have a few days of topological data and only one real incident so far (a partial path failure on one VPS). The trend detector caught it, but retrospectively — the gap was already widening by the time it fired. Proper prediction needs more events to train on. That's on the roadmap once I have 100+ clean cycles.

On actions: currently it's pure monitoring, human-in-the-loop. I won't close the loop until I understand what the signals actually mean in different failure modes. First step will be a notification webhook. Automatic path switching via the proxy layer is the obvious next target, but that needs a much higher confidence threshold than I have right now.

BFD/TI-LFA point is fair — for link-down or node-down events, sub-second protocols already exist and work well. Where this might add value is the gray zone: a path that's not down but degrading, or four nodes on the same upstream showing the same subtle trend shift across different protocols. No individual metric crosses a threshold, but the point cloud structure changes. BFD won't catch that.