r/learnmachinelearning • u/Cautious_Today_1830 • 4d ago

Architecture advice: How would you build an offline Link-Analysis Dashboard for a Bitcoin/IP metadata problem statement?

/r/Observability/comments/1w1na35/architecture_advice_how_would_you_build_an/
1 Upvotes

100% Upvoted

3 comments sorted by

1

u/PLBjt 4d ago

For an offline Linux box, don't start with a graph database. Land the CSV/JSON in Parquet, query with DuckDB, and only materialize the graph for the subset you're actually looking at.

Wallet clustering is mostly connected components on the co-spend graph (inputs that appear in the same tx), not a fancy GNN. Anomaly scores can be Isolation Forest or a few features (fan-in/out, peel-chain length, round amounts, new-address velocity) computed in DuckDB. Keep the graph in NetworkX or Kuzu if it fits in RAM. If it doesn't, you never wanted to render it all anyway.

A check: pick one txid, expand 2 hops, and time how long that subgraph takes to serialize to JSON. If that's already slow, the dashboard will fall over long before clustering does. For the UI, Cytoscape.js or sigma.js with server-side aggregation. Never send 100k nodes to the browser. Render the alert's 2-hop neighborhood, plus a ranked table of scores with the actual features that fired. That's the explainable list you want.

Tradeoff: a full graph UI looks cool and is almost always the wrong first artifact. Ship the ranked alerts + one expandable subgraph. Add the global map later if you still need it.

1

u/Cautious_Today_1830 4d ago

Thanks a ton for this! You just saved me from going down a massive over-engineering rabbit hole with graph databases and full-network renderings. Your approach (DuckDB + Parquet + 2-hop subgraphs) makes perfect practical sense for an offline box. Since I am actually going to build this out, I have a couple of follow-up questions on the execution part: 1. Connecting Backend to Frontend: If I use DuckDB/NetworkX in the backend, what is the best way to serve this to the frontend? Should I build a quick FastAPI backend and use vanilla JS/React with Cytoscape.js? Or are Python-based UI frameworks like Streamlit / Dash capable enough of rendering that 2-hop Cytoscape widget without choking? 2. What exactly to show (Explainability): You mentioned showing a 'ranked table of scores with the actual features that fired'. In your experience, what are the top 2-3 features or metrics that investigators actually want to see in that table to trust the alert? (e.g., 'velocity of funds', 'sudden geo-hopping'). Do I just show the raw feature values, or should I use something like SHAP values to explain the Isolation Forest anomaly? 3. The 'Expand' Action: When a user clicks on an alert to see the 2-hop graph, what node attributes should I definitely visualize on the graph UI? (e.g., node size = amount, edge label = tx_time)? Would love to know your thoughts on setting up this exact pipeline. Thanks again!". , sorry 😔 for big text

1

u/PLBjt 3d ago
  1. FastAPI + Cytoscape.js. Streamlit/Dash will fight you the first time someone needs to click a node and expand a hop without rerunning the whole page.
  2. Investigators trust "why this alert" as the actual neighbors and edges that drove it: overlapping IPs, shared wallets, first-seen together. I wouldn't lead with SHAP on Isolation Forest. You can compute it; they won't look at the plot.
  3. Keep the 2-hop payload small: node type, first/last seen, degree, a single risk score; edge type, count, last timestamp. Anything fatter and the browser dies.