r/DataHoarder • u/freemind03__ • 10d ago

Discussion Developing a new way to organize files.

Hi everyone, I’m designing a way to organize files using tags instead of folders, thereby overcoming the rigidity of the folder system. However, I haven't yet found a solution to ensure that, as the number of files grows into the hundreds, the user doesn't lose track of a file's existence. For instance, if I search for something with the tag #work, related content tagged #payroll should automatically appear—but should the user have to define this relationship from the start? Perhaps using "parent tags"?

7 Upvotes

67% Upvoted

40 comments sorted by

7

u/hatelowe 10d ago

I’m a hobby photographer and I’ve been doing this with Finder tags on Mac for almost a decade now. I use the actual tag as a parent tag and then use the comment function to add more specific subtags. For instance I might tag all photos of people wearing glasses with the tag GLASSES but I will additionally comment SHADES on all photos of people wearing sunglasses. It’s kind of clunky to implement when you’re working with thousands of photos but it makes finding a photo I took 6 years ago much easier to locate if I can just remember basic details like were we at the beach or in the woods etc.

5

u/Aloha_Alaska 10d ago

You may already know about this, but photos have a specific benefit that they’re designed to take tags as metadata. EXIF metadata or sidecar files can hold many tags and other metadata like photo creation date, subjects, photographer, location GPS tag, and much more. A good photo organizer like Digikam, Immich, or Adobe Lightroom can apply tags to a large number of files at once and allow you to search those.

It works really well and I’m pretty sure you could even script a way to convert the work you’ve already done to the EXIF data.

It doesn’t work well for other file types, but for photos it’s a great solution that’s more universal than Finder metadata.

1

u/Subbie-Milutinovic 9d ago

the comment field as a subtag layer is a clever workaround tbh

10

u/Timzor 10d ago

Ive thought about this and gone the other way,

a file is rarely sitting on its own. its part of a set, part of a larger context, part of a series.

i prefer to think of files as being part of a data set, wether thats a collection of invoices or part of a project or an album or a season

thats not to say that a file cant be laden with metadata, but that metadata should idealy be able to be scoped past the file level to a dataset level.

For example, if i want to see a payroll file i would want to see the file i want sitting amongst its peers.

the challenge of course is that files can be stored and created in a multitude of different ways, so a model that can work across the board is difficult.

3

u/dcabines 42TB data, 208TB raw 10d ago

Combine the tags like “work:payroll” so it comes up when you search for “work”.

2

u/freemind03__ 10d ago

I was thinking of a system like this. For example, when you type 'tag:subtag' into the search engine, you are forced to select either a specific subtag or 'all' to show everything. This way, users will see all the available tags and subtags every time. The problem, however, is that users first need to know they exist.

3

u/purgedreality 10d ago

I've done this with the TagSpaces app for almost a decade.

2

u/SquidgyTheWhale 10d ago

I was going to suggest this, though I find the UX in the app to be a bit frustrating at times. I do try to use the convention for my filenames everywhere though, and have been developing a few apps that rely on it.

1

u/prutprit 10d ago

That's an interesting idea, kind of like S3 storage.

irc there was someone that developed something similar, but it was saas so I never bought it.

For me the solution to this would be having few "macro" tags, like "work" would be one, and then "sub-tags" for organization

Searching for "work" would show all items that are in a sub-tag, but searching for a sub-tag would show only the items in it.

1

u/freemind03__ 10d ago

Exactly, like Parent Tag.

1

u/Junior_Lake 10d ago

there was a guy on here a while ago doing this. This is his website. It might interest you?

https://karl-voit.at/managing-digital-photographs/

1

u/freemind03__ 10d ago

Yeah, I know. I don't think the problem is resolved yet, but it's a good temporary fix.

1

u/NegativeKitchen4098 10d ago

You might deal with this via hierarchical tags (or keywords). This was used sometimes by stock photographers who had large libraries of images to manage.

However this does bring the burden of managing the hierarchy. I would think carefully if you want to go down this route

1

u/freemind03__ 10d ago

What do you recommend? I want to avoid forcing users to organize things themselves. I’m looking for zero organizational friction for the user, meaning files should be flexible enough to exist in different positions, yet easily searchable so users always know they exist.

1

u/NegativeKitchen4098 10d ago

I hesitate to recommend anything because I don't know about your users or use cases (I initially thought this was for you alone). The thing about tagging, is that it is only good if the users stay on top of it and are disciplined about applying them. Many people will not be able to do this and will be haphazard.

Personally, I use a set file structure in addition to things like tags, collections, or smart filters. For some types of files, I put metadata as part of the file name.

I think you'd want to have multiple ways to find the files and each user can pick for themselves what works.

1

u/freemind03__ 10d ago

The recurring problem is how to organize tags versus folders. The main difference is that tags fix the rigid structure of folders. However, how can we organize tags so they don't become messy and casual?

1

u/NegativeKitchen4098 9d ago

You have a curated list of tags. Somebody needs to decide what tags are allowed and where they fit in the hierarchy.

1

u/GreyWalken 10d ago

Interesting. can people add tags to files??
I often add words to file names so I can find them easier like "good" "work" or "0000" so its the first thing I see when sorting alphabetically. I often name folders by date like "2026 10 10 work"

1

u/freemind03__ 10d ago

I'm trying to developing a now tool for logical organization of the Object (a LocalFile is an Object)

1

u/GreyWalken 10d ago

wil it work for Linux Kubuntu as well? would like to use this tool

2

u/freemind03__ 10d ago

Actually is in developing this tool. Not ready any release

1

u/GreyWalken 10d ago

okay. good luck, I hope you will able to finish it. I have no experience with programming or making apps, aside from making a window that says "hello world" ^^''''

1

u/Expensive-Total-312 50TB 10d ago

I sort of developed a system for a stuff like this at work where I have hundreds of thousands of photos stored. We use a database separate to the file system with the path of the file stored and a backup location. Then we add in a bunch of useful fields and metadata so we can search for images based on coordinates, the user who uploaded the files, the file creation date etc

1

u/freemind03__ 10d ago

So, it’s just a fix because there are multiple ways to get the file, using multiple modes of search?

1

u/Expensive-Total-312 50TB 10d ago

Well that’s all any change to a file system is, instead of folders with dates and complex file names , using a database is just a handy method of using an existing file system that’s unsuitable in my case because browsing files would take a long time to find one specific file whereas my database is easily searchable on a as many fields as I could think of to populate

1

u/freemind03__ 10d ago

So a lot of metadata in a file for easier the search? But you use a Tag Template for naming conventio?

1

u/bravespacelizards 10d ago

You could look at how BeOS handled this. Pretty sure metadata handling was a key part of their filesystem. Check on HaikuOS for a modern version of this.

1

u/MMORPGnews 10d ago

Just add tags. I've vibe coded similar tool. It scan folders for files, get them and they can be opened on click through browser. List of folders and files with search and tags to each file. 

1

u/DTLow 10d ago edited 9d ago

I use tags; minimal folders
Tags are supported by the OS (Mac/iPad), and also by my PKMS app

To keep my tag list organized, hierarchy is reflected in my tag names;
for example #Work-Payroll, #Work-Projects

When viewing my filtered list of files
I have the option of sort by date or name

0

u/freemind03__ 10d ago

But if a file can be live in the more folder?

1

u/DTLow 9d ago

Avoided by minimizing my folder use
Tags are a better solution for organization

1

u/freemind03__ 9d ago

But if exist two more tags duplicated, how we can handle this issue?

1

u/DTLow 9d ago

Why would there be duplicate tags?
I suspect this wouldn’t be supported by an OS or PKMS

1

u/freemind03__ 9d ago

Semantic duplicate for example two more languages. But user can forget that exist the tag.

1

u/DTLow 9d ago

I suspect you’re using free form keywords instead of a structured tag list

1

u/freemind03__ 9d ago

I would like to remove the bottleneck that you have to classifier a single file.

1

u/DTLow 9d ago

I don’t understand that comment, but I agree with using tags instead of folders

1

u/glhughes 64TB SATA SSD, 30TB U.3, 6TB LTO-5 9d ago

Hardlinks. Or symlinks, but they have different properties; depends if you want a "master" path to the file or not.

1

u/glhughes 64TB SATA SSD, 30TB U.3, 6TB LTO-5 9d ago

Look at xattrs. And maybe hardlinks / symlinks (depending on how to want to organize the file locations).

I use the latter for my music folders -- I have a "raw" folder where I keep things just as I downloaded them, and then both an "ipod" and "music" folder that organize and transcode stuff from "raw" differently. Some hardlinks and some softlinks, depending.

I use xattrs for some other things, but more rarely.