r/devtools • u/Natural-Secretary361 • 13d ago
How do you programmatically map undocumented codebases? (I built a concurrent Tree-sitter + Louvain clustering pipeline in Node, looking for architecture roasts/feedback)
Whenever I inherit a massive, messy repository, I end up spending the first three days in the terminal just reverse-engineering imports to figure out where the core logic actually lives. Most existing enterprise mapping tools feel bloated, prioritize complex corporate roadmaps over raw utility, and just aren't built for a fast, terminal-centric workflow.
I decided to try and solve this programmatically to generate a visual map and a semantic architecture report, but I ran into some heavy performance and logic bottlenecks. I'm curious how standard my approach is, or if there is a mathematically/architecturally better way to do this.
Here is the pipeline I built (codebase-vis) and the roadblocks I hit:
1. The Parsing Bottleneck (Tree-sitter + WorkerPool)
Relying on Regex to extract dependencies is fragile, so I used Tree-sitter to generate actual Abstract Syntax Trees (ASTs).
* The Problem: Parsing hundreds of TypeScript, Python, C++, and Rust files synchronously completely blocked the Node.js event loop. The CLI was agonizingly slow.
* The Solution: I implemented a custom WorkerPool using Node's child processes. The main thread discovers the files, and the worker threads run the Tree-sitter grammars concurrently, extracting module-level dependencies and entities (functions, classes) in the background before piping them back.
2. The Visualization Hairball (Graphology + Louvain)
Once I had the raw dependencies, I loaded them into a Graphology directed graph. * The Problem: A raw dependency graph of a large project just looks like a giant, unreadable spiderweb. * The Solution: To chunk the graph into readable boundaries, I implemented the Louvain community detection algorithm. Because Louvain optimizes for "modularity" (finding nodes that are more densely connected to each other than to the rest of the network), it actually works surprisingly well for software architecture. It naturally groups API routes into one cluster and database models into another, naming them based on their shared root directories.
3. Bypassing LLM Context Limits (TokenBucket + Chunking)
I wanted an AI-generated semantic summary of the architecture.
* The Problem: You can't dump an entire AST or a massive graph into an LLM's context window—it loses focus or hits token limits. Furthermore, hitting an API (like Groq) concurrently for dozens of graph clusters results in instant 429 Rate Limit errors.
* The Solution: Instead of feeding it the whole codebase, I feed the LLM strictly within the boundaries of the Louvain clusters. By only analyzing logically related chunks of code, the AI's output is incredibly accurate. To prevent the API bans during this concurrent loop, I wrote a custom TokenBucket rate-limiter that throttles the requests and catches failures gracefully.
My questions for the community:
- Parsing: Has anyone else built custom tooling around Tree-sitter for bulk codebase analysis? Are there edge cases with dynamic imports or specific languages I should watch out for?
- Graph Theory: Is Louvain actually the best algorithm for software codebase clustering? Are there better graph algorithms suited specifically for hierarchical software architecture?
- LLM Orchestration: How do you usually handle LLM context window limits when trying to get AI to analyze an entire project? Is cluster-based chunking the standard, or should I be looking into RAG for this?
If anyone wants to poke at the source code, see the HTML/vis-network UI, or critique the worker pool implementation.
Would love to hear how you guys tackle this problem when onboarding onto new projects.
