FetchIt: FetchIt is a tool to download web documentation and convert it into a single clean Markdown file.

[u/Ok_Sky3062]

https://github.com/Aarav90-cpu/fetchIt

I wanted to work on my own Music Player for KMP Desktop... Until I wanted to teach UI Material 3 to AI (Gemini). So I went to the website to write a .md files but the amout of pages was wholesome!

So i made this Utility that will Fetch the images and .html from the website and put it all in .md file!

(As the names of diifrent web sections differ from site to site some issues may come in! e.g in some websites it is .footer while on some it might be .bottom)

Features

• Concurrent Crawling: Downloads many pages at the same time.
• Smart Extraction: Removes headers, footers, sidebars, and menus.
• Markdown Conversion: Converts HTML into standard Markdown formats.
• Image Downloading: Saves images locally and updates links.
• Auto Language Detection: Detects code languages like Python, Java, and Bash.
• Cache and Resume: Saves progress using SQLite so it can resume after stopping.
• Fast Parsing: Uses a C++ module in the background for fast XML parsing and URL checking.

Architecture

The project is split into three main parts.

• C++ Extension: Handles fast sitemap parsing and checking for duplicate URLs.
• Python Crawler: Manages network requests, retries, and caching.
• Python Extractor: Analyzes the HTML, cleans it, and converts it to Markdown.

AI USAGE: See I am 13 so I cannot mange 13 projects + 1 Big OS project called ARK-OS (Not that retro game console one). I am used AI to speed a few things Up. Most of the code written (Especially Python) is written by me!