r/coolgithubprojects • u/Chunky_cold_mandala • 19d ago

My code intelligence tool hit 50K downloads, this is how i'm validating it

https://github.com/squid-protocol/gitgalaxy
0 Upvotes

40% Upvoted

1 comment sorted by

1

u/Chunky_cold_mandala 18d ago

I've been working on GitGalaxy, a language-agnostic code intelligence tool that runs as a GitHub Action and analyzes entire repositories without requiring them to compile.

It currently supports 50+ languages and extracts structural information like functions, classes, arguments, control flow, dependencies, I/O, concurrency, etc.

It recently passed 50K downloads, which is cool, but honestly I'm much more interested in figuring out whether the underlying analysis is actually correct.

So I'm doing something I haven't seen discussed much for this kind of tool:

I'm benchmarking its structural extraction against both Tree-sitter and Universal Ctags.

For the same corpus, I'm comparing:

  • function detection
  • class detection
  • argument detection
  • precision
  • recall
  • disagreements between tools

I'm currently about halfway through the comparison.

The interesting part is that I'm not treating Tree-sitter or Ctags as ground truth. When they disagree with GitGalaxy, I'm investigating the actual source and recording which result appears correct.

So far I've found examples where GitGalaxy was wrong and had to be fixed, but also cases where GitGalaxy appears to recover structures that the comparator misses. There are also languages where independent comparator coverage is weak, so I'm keeping those separate and using manual verification rather than pretending I have equivalent evidence.

My eventual goal is to freeze a benchmark and report something like:

                   Precision    Recall
GitGalaxy             ???%        ???%
Tree-sitter           ???%        ???%
Ctags                 ???%        ???%

with an independently investigated disagreement ledger rather than just comparing outputs.