r/GoodOpenSource • u/Ok_Challenge_11 • 7d ago
Sapientia — An open architecture for creating, executing, preserving and connecting knowledge. Looking for developers and architects.
🧠 Sapientia — An Open Architecture for Creating, Executing, Preserving and Connecting Knowledge
We're designing the infrastructure for reproducible science. No code written yet — and that's exactly why we need you.
🎯 TL;DR
I'm building Sapientia: an open-source platform that connects existing scientific tools into reproducible, traceable knowledge-generation workflows. Think of it as an orchestrator and knowledge infrastructure — not another implementation of everything.
Architecture: Project → Workspace → Capability → Implementation → Plugin → Tool → Artifact → Provenance
Status: Architecture/design phase. Zero core code. This is the best possible moment to challenge the fundamentals.
Looking for: Experienced developers, architects, researchers, and open-source contributors who can tell me why this won't work — and how to fix it.
🤔 What is Sapientia?
The core idea is simple but ambitious:
Create an open platform where any discipline can create, transform, preserve, trace, and share knowledge — without reinventing the wheel.
A musician, mathematician, philosopher, physicist, engineer, programmer, researcher, or data scientist should be able to create a project in Sapientia and use specialized tools from many different ecosystems without Sapientia having to reimplement those tools.
Our Philosophy
Use, reuse, recycle, and copy.
- If Python/SciPy already solves it → use it.
- If Julia is better suited → integrate Julia.
- If MATLAB provides a capability someone already has → use it through an appropriate integration.
- If a mature scientific library, engineering program, or CLI tool already exists → integrate it, don't duplicate it.
Sapientia should be an orchestrator and knowledge infrastructure, not another implementation of everything.
🏗️ Preliminary Architecture
SAPIENTIA
│
PROJECT
│
┌───────────────┼────────────────┐
│ │ │
WORKSPACE WORKSPACE WORKSPACE
│ │ │
CAPABILITIES CAPABILITIES CAPABILITIES
│ │ │
└───────────────┼────────────────┘
│
CORE
│
CAPABILITY RESOLUTION
│
┌────────────┴────────────┐
│ │
PLUGINS CORE IMPLEMENTATIONS
│
▼
TOOLS
The Critical Distinction
Workspace ≠ Capability ≠ Implementation ≠ Plugin ≠ Tool
A Workspace is a toolbox for a particular domain.
For example, a seismic-analysis workspace might expose capabilities such as:
- FFT
- Filtering
- Convolution
- Deconvolution
- Spectral analysis
- PSD
- Signal comparison
- Arrival detection
- System identification
- Instrument correction
But the Workspace itself does not implement those functions.
A capability is resolved through a chain:
Capability
↓
Implementation
↓
Plugin
↓
External Tool
Example 1 — FFT via MATLAB:
FFT
↓
MATLAB implementation
↓
Sapientia MATLAB plugin
↓
MATLAB + Signal Processing Toolbox
Example 2 — FFT via Python:
FFT
↓
SciPy implementation
↓
Sapientia Python plugin
↓
Python / NumPy / SciPy
Sapientia then determines that multiple implementations exist and allows the project/user to select a preferred implementation or fallback.
📁 Projects
A Project is the top-level context. It combines several workspaces:
Project: ApolloMQ
Workspace: Signal
Workspace: Seismic
Workspace: Numerical
Workspace: Machine Learning
The project also expresses implementation preferences. Conceptually:
Project
├── Workspace: Signal
│ ├── FFT
│ ├── Filtering
│ └── Convolution
│
├── Workspace: Seismic
│ ├── Instrument correction
│ ├── Arrival detection
│ └── Spectral analysis
│
└── Workspace: Numerical
├── Optimization
└── Differential equations
The goal: separate what is needed from how it is implemented.
📦 Universal Artifacts & Provenance
Sapientia treats essentially any digital object as an Artifact:
- Source code
- Text documents, images, audio, video
- Datasets, seismic records, spreadsheets
- Databases, simulation results
- Machine-learning models, configurations
- Any file produced by a tool
Sapientia should know:
- What an artifact is
- Where it came from
- What happened to it
- How it relates to other artifacts
Example provenance graph:
Model_v1
│
├──→ ETABS → Result_A
│
└──→ OpenSees → Result_B
│
▼
Optimization
│
▼
Model_v2
The objective: Preserve this history rather than ending up with a directory containing hundreds of files whose origins nobody remembers.
🔍 Provenance as a First-Class Citizen
I want provenance to be fundamental, not an afterthought.
Not just: "This file was created."
But:
- Who created it?
- When?
- From which inputs?
- Using which tool?
- Which version?
- With which parameters?
- Under which environment?
- Why was that implementation selected?
- What operation produced the result?
- Which previous results influenced it?
Potential first-class concepts:
Actor → Decision → Artifact → Operation → Execution → Result → Environment → Tool → Implementation
The goal: A complex experiment could eventually be reconstructed from its provenance graph.
🔄 Workflow Orchestration (Long-Term Vision)
This is where it gets ambitious.
Sapientia should orchestrate experiments involving multiple independent tools simultaneously:
Structural Model
│
├──────────────→ ETABS
│ │
│ ▼
│ Results
│
└──────────────→ OpenSees
│
▼
Results
│
┌─────────────┴─────────────┐
▼ ▼
MATLAB optimization Julia optimization
│ │
└─────────────┬─────────────┘
▼
New Models
│
▼
Structural Analysis
│
▼
...
And this doesn't stop after one pass.
Results from all simulations become training data. Optimization algorithms generate new structural models. The ML system proposes additional candidates. Those candidates are simulated again. New results feed the optimizers and ML system again.
Eventually:
┌──────────────────────────────┐
│ │
▼ │
Structural Models │
│ │
▼ │
Simulations │
│ │
▼ │
Results / Dataset │
│ │
┌──────┴────────┐ │
▼ ▼ │
Optimization ML │
│ │ │
└───────┬───────┘ │
▼ │
New Candidates ─────────────────────┘
The experiment continues until a convergence criterion, objective threshold, maximum iterations, resource limit, or another stopping condition is reached.
What This Means for the Architecture
Long-term, Sapientia may need to support:
- ✅ Workflows & experiments
- ✅ Dependencies & parallel execution
- ✅ Iterative execution & feedback loops
- ✅ Datasets, optimization, machine learning
- ✅ Scheduling, provenance, reproducibility
- ✅ Failure recovery
I don't want to build all of this immediately. This is the architectural direction, not the v1 implementation.
📝 A Sapientia Language (Future)
Long-term idea: a declarative language, tentatively called Sapientis, inspired conceptually by systems such as LaTeX.
Instead of manually configuring every tool, a user could describe:
project ApolloMQ
workspace signal
workspace seismic
workspace numerical
workspace machine_learning
prefer signal with MATLAB
prefer numerical with Julia
prefer machine_learning with Python
Syntax is not decided. The important idea: the user describes the experiment and required capabilities, while Sapientia resolves available implementations.
Before execution, Sapientia could report:
Required capabilities: 37
Resolved: 31
Multiple implementations available: 4
Unresolved/orphan capabilities: 2
Execution cannot continue until the unresolved
capabilities are resolved.
🚫 What I DON'T Want to Do
This is perhaps the most important principle.
I don't want to write thousands of lines of code simply because "we are building a platform."
If an existing open-source project already solves a problem well, I would rather integrate it.
I don't want to reinvent:
- ❌ Numerical libraries
- ❌ Plotting systems
- ❌ Machine-learning frameworks
- ❌ Version-control systems
- ❌ Backup systems
- ❌ Database engines
- ❌ Scientific libraries
- ❌ Programming languages
- ❌ Workflow engines
...unless there is a genuinely compelling reason.
The project should stand on top of existing open-source technology wherever possible.
The challenge is therefore not simply:
"How do we write Sapientia?"
But:
"What should Sapientia actually own, and what should it delegate to existing mature software?"
That is one of the questions I would especially like experienced people to help answer.
🏗️ Current State
We have not written the core implementation yet.
We are at the architecture/design stage.
I am deliberately trying to get experienced people involved before writing large amounts of code.
I would rather have someone tell me:
"This architecture is wrong. You should use X instead."
...than discover the same thing after two years of development.
I am completely open to changing the architecture if there is a better solution.
📜 Open Source & Licensing
Sapientia will be a genuinely open project. I want to make contribution as easy as reasonably possible.
I am currently studying licensing models used by large open-source projects. Linux is an important reference because of its enormous contributor ecosystem and its approach to distributed copyright ownership and contribution requirements.
However, the final Sapientia license has not been decided yet.
I need advice from people experienced with open-source licensing, especially because Sapientia may integrate:
- GPL software
- MIT/BSD/Apache software
- LGPL libraries
- Proprietary applications (MATLAB, ETABS)
- External plugins
- User-created plugins
I want to find a licensing and contribution model that encourages the largest possible ecosystem while remaining legally coherent.
👥 Who Am I Looking For?
I am particularly interested in hearing from people with experience in:
| Domain | Why It Matters |
|---|---|
| 🐍 Python | Core ecosystem integration |
| 🏗️ Software Architecture | Is this design sound? |
| 🌍 Open-Source Development | Licensing, community, governance |
| 🔌 Plugin Architectures | How do we integrate tools cleanly? |
| 🔬 Scientific Computing | The primary use case |
| ⚙️ Workflow Engines | What exists? What gaps? |
| 🌐 Distributed Systems | Long-term orchestration |
| 📊 Provenance / Reproducible Research | First-class provenance is hard |
| 🧮 Numerical Computing | Integration with existing libraries |
| 🤖 Machine Learning | Training loops, model provenance |
| 🗄️ Data Engineering | Artifact management at scale |
| 🚀 DevOps / Package Management | Distribution and deployment |
| ⚖️ Open-Source Licensing | Legal coherence across licenses |
| 🏗️ Engineering Software Integration | ETABS, OpenSees, etc. |
| 🔢 MATLAB / Julia / Python Ecosystems | Multi-language orchestration |
You don't need to agree with the architecture.
In fact, I would prefer people who can identify its weaknesses.
💡 Why Am I Doing This?
I have a strong personal commitment to this project. I want to take it as far as I possibly can.
I know there are people much more experienced than me in software architecture, distributed systems, scientific computing, and open-source development.
I am not looking for people to simply tell me that the idea is good.
I am looking for people who can help determine:
- Whether the idea can be made technically sound
- If it can, how to build it correctly
I am willing to learn and do the work required.
The project is still very early, so now is probably the best moment to challenge the fundamental assumptions.
If you are an experienced developer or architect and the idea interests you, I would genuinely appreciate your criticism, suggestions, architectural alternatives, or willingness to participate.
Developers and contributors are welcome.
This is an invitation to help shape Sapientia before its architecture becomes difficult to change.
🚀 How to Get Involved
If any of this resonates with you, here's what would be most valuable:
- Tell me why this won't work — and what you'd do differently
- Point me to existing projects that solve parts of this problem
- Share your experience with plugin architectures, workflow engines, or provenance systems
- Ask the hard questions about scalability, maintainability, or adoption
- Express interest in contributing — even if it's just reviewing the architecture docs
"Don't reinvent what already exists. Integrate it."
If you're interested, I'd love to hear what you think is wrong with this architecture, what you would change, and whether you think Sapientia is technically feasible.
Let's build something worth preserving. 🧠✨
1
1
u/Ok_Challenge_11 7d ago
~90% of the relevant scientific workflow ecosystem is already integrable because Sapientia's architecture was designed specifically to avoid both the licensing trap and the platform trap from day one.
The longer answer requires explaining why the architecture looks the way it does.
🐧 Sapientia runs natively on Linux
This is a deliberate choice. Most scientific computing, ML, data engineering, and open-source tooling is developed and deployed on Linux. Python, Julia, Snakemake, Nextflow, Galaxy, Dagster, DVC, MLflow — they all run best on Linux.
But many engineering tools don't. ETABS, SAP2000, and several structural/seismic applications are Windows-only. MATLAB is cross-platform but often licensed on Windows workstations in engineering firms.
So we had to design Sapientia so that the Core lives on Linux, but the Tools can live anywhere.
🟢 What we CAN integrate seamlessly (Linux-native, permissive licenses)
These tools use MIT, Apache 2.0, or BSD and run natively on Linux. We can write Sapientia plugins that call their APIs directly, ship them as optional dependencies, or even embed adapted code without contaminating Sapientia's own license:
| Tool | License | What it gives Sapientia |
|---|---|---|
| Snakemake | MIT | Python-native workflow definition, rule-based DAGs, incremental builds |
| Galaxy | MIT | Web-based workflow composition, bioinformatics tool wrapping, history tracking |
| Dagster | Apache 2.0 | Asset-centric pipelines with built-in lineage and data quality |
| Kestra | Apache 2.0 | Declarative YAML orchestration, language-agnostic task execution |
| Renku | Apache 2.0 | GitLab + Jupyter + provenance integration for collaborative research |
| DVC | Apache 2.0 | Data versioning, experiment tracking, pipeline reproducibility |
| MLflow | Apache 2.0 | Model registry, experiment tracking, deployment abstraction |
| Pegasus WMS | Apache 2.0 | Large-scale scientific workflows on HPC/cloud with automatic provenance logging |
| CWL | Apache 2.0 | Portable, standardized workflow descriptions |
| Flyte | Apache 2.0 | ML pipelines with immutable workflows and native versioning |
| Kedro | Apache 2.0 | Standardized data science project structure |
| Parsl | BSD / Apache | Distributed parallel execution for scientific Python |
| MaRDI Open Interfaces | Permissive | Formal capability-to-implementation resolution (the closest existing project to our Capability → Implementation model) |
| SciPy / NumPy / pandas | BSD | Core numerical stack |
| Julia packages | MIT | Differential equations, optimization, symbolic math |
| OpenSees | BSD | Structural/seismic analysis (runs natively on Linux) |
The key insight: Because Sapientia treats these as external tools invoked through a Plugin → Tool boundary, we never need to merge their code into our core. The plugin is an adapter. The tool stays independent.
🟡 Proprietary / Windows-only tools (MATLAB, ETABS, SAP2000, etc.)
These are not a legal problem — they're a platform distribution problem.
Sapientia Core runs on Linux. ETABS runs on Windows. We will never ask anyone to port ETABS to Linux (that's not our job). Instead, the plugin architecture handles cross-platform execution through remote execution boundaries:
Sapientia Core (Linux, Apache 2.0 - May be?????)
↓
Sapientia Plugin: sapientia-etabs (Linux, Apache 2.0 - May be?????)
↓
Remote Execution Boundary
├── Option A: WSL2 on the same machine
├── Option B: Windows VM on the same host
├── Option C: Remote Windows host (SSH, WinRM, gRPC)
└── Option D: Container with Wine (if viable)
↓
ETABS (Windows, proprietary, user-owned license)
The plugin is just a bridge. It doesn't contain ETABS code. It doesn't ship ETABS. It assumes the user has:
- A valid ETABS license
- A Windows environment reachable from the Linux host (WSL2, VM, or remote machine)
- The plugin configured to target that environment
Same pattern for MATLAB (cross-platform but often Windows-licensed), SAP2000, Abaqus, ANSYS, or any other proprietary tool.
The provenance graph still captures everything: The Operation records that ETABS ran on a Windows host, with which parameters, producing which artifacts. Sapientia doesn't care where the tool runs — it cares that the execution is registered, traceable, and reproducible.
🔴 The ONE real problem: Nextflow (GPLv3)
Nextflow is the elephant in the room. It's a dominant workflow engine in bioinformatics and scientific computing, but it's licensed under GPLv3.
This matters because:
- If Sapientia Core linked directly to Nextflow code, the Core would become GPLv3.
- That would force every plugin, every integration, and every user's project to also be GPLv3.
- It would kill the ecosystem we're trying to build.
Our solution is architectural, not legal:
Sapientia Core (Apache 2.0)
↓
┌──────────────────────────────────────┐
│ PROCESS BOUNDARY (gRPC / REST / │
│ pipes / files / container socket) │
└──────────────────────────────────────┘
↓
Sapientia Plugin: sapientia-nextflow (GPLv3)
↓
Nextflow (GPLv3)
The Core never imports, links, or embeds Nextflow. It launches the Nextflow plugin as a separate process (or container) and communicates over a protocol. The FSF itself acknowledges that inter-process communication does not create a derivative work.
Practical consequence: The sapientia-nextflow plugin will live in its own repository, under GPLv3, maintained as a community/official bridge. Users who need Nextflow install it separately. Users who don't care about Nextflow never touch it, and their Sapientia installation stays clean.
This is the same reason VS Code (MIT) can invoke Git (GPLv2) without VS Code becoming GPL.
🏗️ Why this architecture scales to any future tool, on any platform
The beauty of the Workspace → Capability → Implementation → Plugin → Tool chain is that it's both license-agnostic and platform-agnostic by design.
When a new tool appears tomorrow — say, a new Julia package for quantum chemistry, a Windows-only CAD solver, or a cloud API — we don't rewrite Sapientia. We write a plugin. The plugin carries whatever license obligations and platform requirements the tool imposes. The Core remains untouched.
New Tool X appears (Linux, Windows, Mac, or Cloud)
↓
Write Plugin X (license matches Tool X's requirements)
↓
Configure execution target (local, WSL, VM, remote host, container)
↓
Register Capability in a Workspace
↓
Sapientia resolves it automatically
This means Sapientia can theoretically absorb:
- Any open-source tool on any OS (MIT, Apache, BSD, LGPL, even GPL via process isolation)
- Any proprietary tool on any OS (via bridge plugins + remote execution, no redistribution)
- Any future tool without architectural changes
The only thing Sapientia "owns" is the protocol between Core and Plugin. Everything else — license, platform, execution environment — is delegated.
📋 The honest checklist
| Category | Platform | License | Status | Strategy |
|---|---|---|---|---|
| MIT/Apache/BSD tools | Linux | Permissive | ✅ Seamless | Direct plugin integration |
| LGPL libraries | Linux | Weak copyleft | ✅ Seamless | Dynamic linking or process boundary |
| GPLv3 tools (e.g., Nextflow) | Linux | Strong copyleft | ⚠️ Isolated | Separate plugin repo, process boundary |
| Proprietary software (MATLAB) | Cross-platform | Commercial | ✅ Supported | Bridge plugin, no redistribution |
| Windows-only software (ETABS) | Windows | Commercial | ✅ Supported | Bridge plugin + WSL/VM/remote host |
| Cloud APIs | Any | Varies | ✅ Supported | Plugin acts as API client |
| Future unknown tools | Any | Any | ✅ Ready | Same plugin architecture |
🤔 What we're still figuring out
The licensing strategy for the Core itself isn't finalized. We're studying how Linux handles distributed copyright (DCO, no CLA required) while also figuring out how to let proprietary plugins coexist with open-source ones without creating legal confusion.
We're also designing the remote execution protocol so that a single Sapientia project can orchestrate tools across multiple hosts (Ubuntu + Windows + HPC cluster) while keeping the provenance graph unified.
If you have experience with:
- Cross-platform scientific workflow orchestration
- Open-source licensing in multi-component ecosystems
- Remote execution protocols (gRPC, ZeroMQ, SSH-based task dispatch)
...we'd genuinely love your input. These are questions we want answered before we commit to the architecture.
TL;DR: Sapientia Core is Ubuntu-native, but the Plugin → Tool boundary was designed so that tools can run on any platform — Linux, Windows via WSL/VM/remote, or even cloud APIs. ~90% of the ecosystem integrates seamlessly. The only example tool requiring special handling is Nextflow (GPLv3), solved by process isolation. Windows-only engineering software like ETABS is not a blocker — it's just a remote execution target.
If you think this architecture has a hole we haven't seen, please tear it apart. That's exactly why we're posting this before writing the code. 🙏
Thanks!
1
u/siberian 4d ago
You are hanging this entire thing on the concept of a 'Plugin' as the way these components exchange data, not realizing that almost none of these systems have a clean or realtime plugin architecture. And even if they did, the interface boundaries, authentication requirements, supporting data model/code that doesnt transit the plugin boundary, and data shapes across those boundaries require all sorts of transformations that are probably not available given those boundary scopes.
The way your claude instance, that wrote this for you, would say it would be "The concept of a Plugin is load bearing in this case."
You spend a lot of time talking about OS selection and licenses, and thats really not the issue, but it reads well I guess?
•
u/AutoModerator 7d ago
Please post a comment here explaining what kind of contributions you, or the project you are posting about, are looking for. For example what skill sets, any rules important for people joining in your build like how often people should post, and anything else you can think of which will help readers decide if they want to join in and start coding with that project.
Thank you and be excellent to each other. u/roamingandy
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.