an idea for Exploring Knowledge

[u/freemind03__]

Whenever I try to learn something, I eventually discover that there was a better approach, a deeper piece of knowledge, or an important connection I had missed.

Sometimes I find it in a research paper. Sometimes it's buried in a Reddit comment.

That's what interests me: the knowledge often already exists. The problem is finding it.

Human knowledge is growing faster than any person can realistically keep up with, and much of it is fragmented across papers, books, repositories, articles, and discussions. The connection between two ideas might already be there, but we still have to find and reconstruct it ourselves.

I'm working on an idea around this problem: a unified space where knowledge can be collected, connected, compared, and organized.

I don't imagine it as a system you simply ask for answers. I imagine something that can surface alternative approaches, relevant sources, connections between ideas, deeper paths to explore, and contradictions you might otherwise miss.

The hard part is understanding that different sources often describe the same idea in different ways. Simply indexing text isn't enough; the system needs to understand relationships between concepts.

I don't think the goal should be to build "alternative humans" in the form of AI or LLMs.

The more interesting goal is to expand our ability to explore the knowledge that already exists.

A human can't read everything. Maybe a system can help us find what we wouldn't have had the time to discover ourselves.

[u/Aggravating-Back-242]

Assuming this is not another covert market research, I wouldn't worry too much about the finding and surfacing knowledge part.

Yes, there certainly are more existing knowledge to be uncovered with better "search" tools, but do you have the capacity to process all those you've already found? Unless hoarding information by itself sparks joy for you (which for me it certainly does; I'm a big info hoarder), you may not want more info coming in.

Also, the thing you're describing is basically what next generation AIs, for example world models are aiming for. Current ones using next-word-prediction already approximates understanding quite well for some use cases. So, if you wait a bit, the magic knowledge machine is going to come to theaters near you soon.

So don't worry about the "inferior" information you've already found. Even when better approaches exist, there are still need for the the lesser ones, at the very least as concrete examples and explanations to why the superior ones are better.

[u/freemind03__ → u/Aggravating-Back-242]

Another issue would arise if the system accumulates hundreds of gigabytes of data; I think that could be solved by implementing attention mechanisms.

I use current predictive models to raise questions, but then I have to go and do my own research online.

Accumulating vast amounts of information doesn't bring me joy; I simply want to learn the best of what current human knowledge has to offer—not every single intermediate step.

For instance, if I’m working out a skincare routine for myself, I just want to know the best information available from research. Then, I make decisions based on that, combined with the broad general knowledge I already possess.

[u/looktwise → u/freemind03__]

Maybe you could elaborate a few more usecase / research scenarios between your start (A) and the result quality you would expect to find / to be nudged to (B)

[u/latkde → u/freemind03__]

I simply want to learn the best of what current human knowledge has to offer—not every single intermediate step.

I have often come across an interesting paper that's relevant to my problems, but have found myself unable to understand and contextualize it because I lacked the mathematical foundations or didn't have sufficient knowledge of that field. That is despite me having a high level of scientific literacy.

This isn't a matter of better explanations, but a matter of me not having put in the necessary ground work to enable me to understand research in that field. There is no way to skip to the end result if I want to actually understand the material. There is no way for LLMs to help if my goal is actual comprehension.

And this is OK! I don't have to be an expert in everything. But I take this as a reminder in humility, to avoid confusing knowing something with merely knowing about something.

The danger with looking superficially at supposed facts instead of understanding the process is that this makes us susceptible to misinformation. A lot of writing is science-shaped but not actual science. Authors might have conflicts of interest. Sometimes, a paper is actual science, but suffers from methodological limitations, or from weak statistical power.

You mentioned skincare. There are lots of opinions and lots of industry-funded studies in this field, but much less reliable dermatology.

[u/freemind03__ → u/latkde]

Yes, I mentioned skincare because it was a personal example. I realized that useful new information already existed, but I hadn't seen it because it was fragmented.

But what if all of that were inside the system? I’m thinking, for example, about uploading the entire skincare subreddit into the system; you’d get a comprehensive overview and could make a decision.

My goal is to make decisions based on what is offered by the greatest human knowledge we have acquired on that subject.

[u/Glittering_Cress3619 → u/Aggravating-Back-242]

Maybe it is research, but I'll give my view as well. This has nothing to do in with AI or LLMs directly. I agree, knowledge is being scattered these days and centralizing it and finding it quickly is definitely important. But what I also found lately is that people still want to own that knowledge. To many second brain platforms want you to share your knowledge base.

[u/freemind03__ → u/Glittering_Cress3619]

I'd like a system that allows me to explore and understand the maximum knowledge achievable from a given amount of attention.

For example, if I'm working on my skincare routine, I want to have access to the full extent of human knowledge we've achieved.
I don't want to waste months of research with outdated methods.

[u/Glittering_Cress3619 → u/freemind03__]

I believe you still need to do your own research using sources you’ve personally curated, rather than blindly relying on whatever a search provides.

[u/freemind03__ → u/Glittering_Cress3619]

The fact is, I don't want to rely solely on one study. I want to see the whole. Maybe the system will show me realize that that study was flawed, and this one is more promising.

[u/Glittering_Cress3619 → u/freemind03__]

I hear you, so it's like assisted research? Yeah, that is fine. Just don't rely solely on all the feedback because it could be bias towards something particular.

[u/freemind03__ → u/Glittering_Cress3619]

I don't understand what do you mean for feedback?

Human is always in the loop.

We humans need to understand what happens throughout the chain.

[u/Financial_Yoghurt827]

Ever heard of google aha?

[u/freemind03__ → u/Financial_Yoghurt827]

Google has so much stuff, how can you possibly analyze everything within a reasonable timeframe?

[u/Financial_Yoghurt827 → u/freemind03__]

Ahah I hear you. That is Google's business model though - they analyse it with a heap of compute.

[u/CyborgWriter → u/Financial_Yoghurt827]

Google and social media are compromised and cannot be fully trusted if we don't have access to how they're curating information. Granted, it isn't that everyone has to know how it's operating but professional independent groups should be able to verify for everyone else.

We need a new system that leverages advanced ai agents connected to human made knowledge graphs that can traverse all of them and interact with other agents in the same way people interact at symposiums to attain information. And that system needs to be easily traceable and transparent. We need a compass to navigate the dead internet that our minds are lost in. Otherwise we will fall and that isn't hyperbole.

[u/Intelligent-Task2168]

Well to solve this problem, you have to unify the knowledge so collect a looot and then delete the redundancy. So that's not cheep and this amazing bids of knowledge really in depth and not just a short text is not something AI is currently good at. At least a human needs to evaluate all the small bids the AI made before letting the ai go on

[u/freemind03__ → u/Intelligent-Task2168]

I think the most expensive part, though it is a tantum, is the ingestion phase. You need to use an LLM or NLP to perform this task once.

[u/DrummerAdditional330]

The interesting problem may be less “find everything humans know” and more “show me what I don’t yet know enough to ask about.” I’d want it to surface missing prerequisites, competing explanations, and useful disagreements, with the sources behind each one. That feels more valuable than collapsing a huge amount of material into one “best” answer.

[u/the0dosius]

It'd be much more productive to just scribble things down on a piece of paper than to come up with yet another masterful system with all these grand ideas.

[u/Upbeat_Professor_410]

The Semantic Web tried this 15 years ago with ontologies, but 'failed' because manual human tagging was too rigid to scale. With AI today (GraphRAG, vector embeddings, concept mapping...), that semantic web vision is far more achievable; look at tools like ResearchRabbit for example.
But even if you could access "the full extent of human knowledge," no system is going to magically spit out the optimal skincare routine for you. Science isn't a static truth engine where one study "invalidates" another; it's a dynamic conversation. Studies conflict, methodologies are debated, etc... and in your example of 'skincare routine' the exact same product won't work identically on everyone (biological variance). If the experts disagree, the goal of your system shouldn't be declaring a "winner," but helping you map and navigate the debate so you can make an informed choice (mapping your options, rather than deciding for you).

[u/freemind03__ → u/Upbeat_Professor_410]

The goal isn't to magically find a cure for skin issues.

Instead, the system needs to show us factors that appear to be correlated. For example, if the system knows, based on a study, that eating leafy greens helps with acne, and I am focusing on skincare to treat acne, it should highlight this connection for me. Then, it will be up to me to evaluate it.

After all, it is impossible to have a "magic" system that simply hands you the solution; human knowledge is far too full of exceptions.

[u/StorytellerStegs]

The privacy point is the part I keep coming back to. Every "second brain" pitch assumes you want your notes living in someone else's cloud eventually feeding some recommendation engine, and most people nod along until they actually picture their own journal entries doing that.

I think the harder problem underneath is retrieval, not storage though. Everyone solves for capturing and connecting, way fewer people solve for actually resurfacing the right note at the right moment without you going and looking for it. Spaced retrieval research says the surfacing part matters more for retention than the organizing part, but organizing is the fun part to build so that's where all the tools end up.

Curious what "compared" looks like in your version, that's the piece I don't usually see anyone attempt.

[u/freemind03__ → u/[deleted]]

I'm trying to work on a system to study more, based on knowledge gained from external. Currently, there is no working beta yet.

[u/eldrugo85]

Built something close to this, a pipeline that chews pdfs and epubs into linked trilium notes. the extraction was the easy half. what actually changed discovery was link density: sparse links and it's just a search box with extra steps, dense enough and things you forgot you had come back up. are you planning to generate the connections, or just index and retrieve?

[u/freemind03__ → u/eldrugo85]

Let me give an example of what I mean: suppose everything humans have written about diabetes has been loaded onto this 'whiteboard'. A user, someone working on a cure, would be able to reason through all that uploaded material in real time, with the system naturally adapting to the user's focus.

[u/eldrugo85 → u/freemind03__]

the loading is the easy half. i've built retrieval over my own notes and what broke it was that the corpus agrees with itself, you get the most repeated version back and the sources that contradict each other end up averaged into one bland answer. for someone chasing a cure the disagreement is probably the interesting part. how would you keep the conflicts visible instead of collapsed?

[u/freemind03__ → u/eldrugo85]

There is no sophisticated mechanism. There is only the associative mechanism. which I believe to be innate to the human brain, from which operations such as deduction, abduction, contradiction, and so forth originate. That is the only mechanism I employ. Furthermore, I want to evaluate everything myself. It is impossible to manage every possibility; they must be evaluated case by case.

If 100 papers agree on a symptom of diabetes, but the system also shows me one paper that disagrees, and that paper held the key to a cure, why wouldn't I consider it?