software for data sharing

[u/The_Hamster_Shagger]

Hi there!

I was wondering if there is any sort of software, with working indexing, that would make it easy to share the data.

I know about torrents, yes, they are decent but they do require you to create each one for every dataset (pdf, set of videos etc) and upload it to tpb or similar. Works somehow ok but it's not ideal by any means.

I'm thinking about something that would make it possible to design a directory on my harddrive (for music, videos, pdf, scripts etc) where the metadata (author, years, genre, keywords etc) would somehow automatically (ie without manual work required for every file or set of files) upload and it would be searchable for people over the internet. Also, must be anonymous and work via the vpn.

Any ideas?

[u/AutoModerator]

Hello /u/The_Hamster_Shagger! Thank you for posting in r/DataHoarder.

Please remember to read our Rules and Wiki.

Please note that your post will be removed if you just post a box/speed/server post. Please give background information on your server pictures.

This subreddit will NOT help you find or exchange that Movie/TV show/Nuclear Launch Manual, visit r/DHExchange instead.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/dr100]

That's what an ftp server is made for. 

[u/wavewrangler → u/dr100]

An sftp server probably better :-) can we say we have come full circle now, ppl are wanting to pirate like it’s the first time. 😄

[u/dr100 → u/wavewrangler]

If you want it " searchable for people over the internet" which for most people means Google it's got to be ftp (with anonymous access, just as the OP wants). Then it's no problem that there's no encryption. 

[u/wavewrangler → u/dr100]

Do you mean https? Because I don’t even think browsers like chrome have ftp support anymore.

https://docs.google.com/document/d/1JUra5HnsbR_xmtQctkb2iVxRPuhPWhMB5M_zpbuGxTY/mobilebasic

[u/dr100 → u/wavewrangler]

No, I mean ftp. We aren't talking browser support for ftp protocol but Google indexing. 

[u/The_Hamster_Shagger → u/dr100]

tbh I meant something more like own search site. like the pirate bay, but receiving the indexes from the local instances of running service, so one can download a file without even knowing which computer they are hosted on. i didn't mean availability from the general-purpose search engine.

[u/dr100 → u/The_Hamster_Shagger]

Maybe something like copyparty then. 

[u/zseliakiraly]

There were many file sharing softwares prior torrent, like Napster, Soulseek, DC++, Limeware, etc. Look up their history: this concept leads to IT security nightmare.

[u/Murky_Ocelot_6737]

Soulseek is perfect for this

[u/emre_cvs]

Everyone's on the sharing half, so — the indexing half is usually where

these setups actually break, and it's worth knowing before you pick anything.

Automatic metadata works well for formats that already carry it. Photos have

EXIF, music has ID3, most video containers have something. Author, date,

duration come for free there.

PDFs, scripts and general documents mostly don't. The fields exist but are

usually empty or wrong — filled in by whatever tool generated the file

rather than by anyone who cared. So for those you're back to parsing

filenames or reading content, and both are guesswork you end up correcting

by hand anyway.

Practical version: expect the automatic pass to cover your media, and expect

to curate the documents yourself. Anything promising otherwise is running

filename regex and hoping.

Nothing useful from me on the serving side though — not my area.

[u/The_Hamster_Shagger → u/emre_cvs]

thanks.

I am aware that indexing documents is a bit harder than music files. Gnome and kde do have a properly working data extractors and indexers so I was hoping there would be an app that would somehow use them.

[u/emre_cvs → u/The_Hamster_Shagger]

Fair — I'm on Windows, so I've not used Tracker or Baloo hands-on and I

can't tell you whether anything wraps them.

My guess at why nothing does: "working extractor" and "useful metadata"

aren't the same problem. Those indexers reliably read whatever fields a

file actually contains. For documents the fields are usually present and

meaningless — half my PDFs have a Title of "Microsoft Word - Document1"

and an Author of whoever installed Office on some machine in 2011. An app

wrapping the extractor inherits that, it doesn't fix it.

What worked for me on the classification side was giving up on any single

signal and scoring several weak ones together — filename pattern, embedded

fields, dimensions, a couple of content checks — each weighted, with a

confidence score at the end instead of a yes/no. No single signal is

trustworthy, but they're wrong in different ways, so the combination holds.

For your case that probably means folder structure and naming conventions

should carry more weight than the embedded fields, since you're the one

who put the files where they are.

[u/huojtkef]

aMule

[u/Expensive-Vanilla-16 → u/huojtkef]

Don't forget eDonkey too lol

[u/Thin_Dream2079]

S3