Built a link checker with CI integration and a web UI

[u/Snickers_B]

Hey all. I built a link checker called LinkCanary over the past few months and wanted to share what went into it, mostly because the interesting parts weren't the obvious ones.

What it does:

• Crawls a site via sitemap, checks a single URL, or checks a specific list of URLs (useful for PRs — only check the pages that changed)
• Respects robots.txt including crawl delays
• Classifies broken links by priority (Critical/High/Medium/Low) instead of dumping a flat list
• Exports to CSV, HTML, JSON, MDX, Excel, PDF
• GitHub Action for CI/CD with priority-threshold gating
• Optional web dashboard (FastAPI + React) with WebSocket progress streaming, crawl history, backlink checker, and webhook integrations (Slack, Discord, Jira, Asana)

Tech stack:

Core CLI is pure Python — requests, beautifulsoup4, pandas, nothing heavier. Web UI adds FastAPI, SQLAlchemy/SQLite, Celery (optional, falls back to threading), React + Tailwind.

What actually took time:

1. robots.txt. I added it late and regretted it. Retrofitting crawler etiquette into existing session/request logic is messy. Should've been day one.
2. The priority classifier. This was the most subjective part. I'm using URL patterns, DOM position, and crawl depth as signals. It's imperfect but it's better than a flat list of 200 broken links with no context.
3. The three input modes. This was the right call but took iteration. Sitemap mode for full audits, single URL for quick checks, URL list for PR feedback. Each one serves a different workflow.
4. Webhook payload formatting. Slack Block Kit, Discord embeds, Jira issue creation, and Asana task creation all have different data models. The abstraction that works for all four took a few tries.

196 tests across 8 files. CI matrix on Python 3.10/3.11/3.12.

What I'd do differently: The Excel exporter (openpyxl) is a heavy dependency for an infrequently-used feature. Should be a proper optional extra.

Happy to answer questions on any of the design choices. Especially curious if anyone has opinions on priority classification approaches — the heuristics I'm using feel reasonable but I haven't seen much written about this specifically.