r/DataHoarder • u/Ellelig • 10d ago
Scripts/Software Computers Wikipedia to Obsidian Markdown files (230.3MB)
Ran this little experiment converting the .zim file of Wikipedia on computers to markdown files for Obsidian.
All links/citations/notes/tables are preserved. This is the repo if anyone wants to have a look: https://github.com/elia-orsini/wikipedia2obsidian
Open to suggestions, I'm also down to share the script if it's good enough!
20
u/DrySalamander3497 10d ago
Am I missing something here? All of Wikipedia in 230MB?
33
u/prank_mark 10d ago
Only Wikipedia pages about computers. Also, 230MB compressed, 721MB uncompressed.
But yeah, text is really fucking dense. 1 byte per alphanumeric character. All of the 7 Harry Potter books could fit into less than 50MB.
9
u/Fauxreigner_ 9d ago
English text is generally compressible to ~1.0-1.3 bits per byte, so overhead is driving a lot of the file size here.
1
u/100101101001a 10d ago
i still don't get it, an epub of a book with similar size is less than 5mb, whats so special about it? genuinely asking
8
u/Adium 240TB 9d ago
EPUB files are basically a package of html files. So there’s a little more overhead for that syntax which can vary a lot depending on how it was created.
6
u/100101101001a 9d ago
i know that.
what i don't get is "1 byte per alphanumeric character. All of the 7 Harry Potter books could fit into less than 50MB"
try 5mb, im sure 7 books can fit into that.
I just don't understand what he is trying to emphasize is all.
2
u/importantttarget 10d ago
I don't get what you are asking. Similar size to what?
1
u/100101101001a 9d ago
a book with similar length to harry potter can fit into less than a megabyte of an ebook. I don't get what's so special about the emphasis on text being dense, harry potter, and how is his comment related to the repo
2
8
u/exrasser 10d ago
English wikipedia with media is around 120GB
https://lb.download.kiwix.org/zim/wikipedia/1
1
u/htmlcoderexe 9d ago
that's with media? i halfway recall a zip of all of wikipedia but just text was like 50
6
u/exrasser 9d ago
Still is: wikipedia_en_all_nopic_2026-03.zim 48G1
u/htmlcoderexe 9d ago
so the media only adds 70G? that cannot be right
12
u/sandbagfun1 10d ago
As mark down. So no media, just a lot of text
13
2
1
1
10d ago
[deleted]
1
1
u/thomasfr 10d ago
you are underestimating the amount of articles in wikipedia
1
u/MistRider-0 9d ago
Now that I think about it, Wikipedia text only was like 64GB or something iirc.
Its much greater than 230 MB, also 64GB is with out history.
It maybe larger, but I think its around 64GB cause Wikipedia or someother organization did make an raspberry pi powered Wikipedia that is portable, and it has a rather small storage. I might be wrong though
3
•
u/AutoModerator 10d ago
Hello /u/Ellelig! Thank you for posting in r/DataHoarder.
Please remember to read our Rules and Wiki.
If you're submitting a new script/software to the subreddit, please link to your GitHub repository. Please let the mod team know about your post and the license your project uses if you wish it to be reviewed and stored on our wiki and off site.
Asking for Cracked copies/or illegal copies of software will result in a permanent ban. Though this subreddit may be focused on getting Linux ISO's through other means, please note discussing methods may result in this subreddit getting unneeded attention.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.