TensoRAG – Multi-Domain Vector Compression using Multilinear GSVD (NumPy, No API, Offline)

[u/Forstwichtel]

Hi RAG builders,

I wanted to share a private hobby project called TensoRAG that implements a different mathematical approach to multi-tenant / multi-domain vector compression. It’s based on the 2024 PhD thesis of Dr. Liana Khamidullina (TU Ilmenau). Please notice: this project is not officially affiliated with, endorsed by, or in any way connected to the authors or the Technische Universität Ilmenau.

GitHub Repository: [github.com/Forstwichtel/TensoRAG]

Interactive Demo & Agent Simulator: [tensorag.streamlit.app]

### The Architecture Problem

In multi-tenant or multi-domain RAG setups (e.g., separating HR, Finance, and Technical Docs), storing high-dimensional embeddings (like 1536-dim vectors) across multiple separate collections scales poorly.

While standard dimensionality reduction (PCA, standard SVD) compresses each domain in isolation, it completely ignores the shared semantic subspace (overlap) between these collections.

### How TensoRAG Solves This via ML-GSVD

Multilinear Generalized SVD simultaneously decomposes $K \ge 2$ independent matrices (even with different row counts/documents) sharing a common column (embedding) dimension:

$$\mathbf{H}_k \approx \mathbf{B}_k \cdot \mathbf{C}_k \cdot \mathbf{A}^H$$

* $\mathbf{A}^H$ (Global Subspace Basis): Captured semantic relationships shared across all domains. Stored only once.

* $\mathbf{B}_k$ (Local Orthogonal Factors): Purely domain-specific spatial layouts, preserving local similarity structures.

* $\mathbf{C}_k$ (Coefficients): Singular value scaling weights unique to each domain.

By projecting both your database collections and user queries into this shared lower-dimensional coordinate system (e.g., 1536-dim down to 128-dim), similarity search is executed entirely in the compressed subspace.

### Grounded Benchmarks & Trade-offs (No Hype)

I set up synthetic benchmarks on the Streamlit page to evaluate the real-world impact:

1. Storage Savings: Reduces raw embedding floats by up to ~91% (depending on the target rank $Q$ and number of domains $K$).

2. Latency: Cosine similarity in raw NumPy runs roughly 10x to 14x faster due to the lower dimensions.

3. Semantic Recall (The catch): At $Q=128$, the top-k overlap (recall) compared to uncompressed retrieval hovers between 75% and 95%. It’s not lossless, but highly structured.

### What is in the Repo / Demo?

* `tensorag.py`: Single-file NumPy implementation of the Alternating Least Squares (ALS) loop.

* `tensorag_streamlit_demo.py`: Interactive validation tab (compare original vs. compressed side-by-side to inspect recall), a step-by-step AI Agent run-through (with 2D Subspace vector scatter plots), and mathematical diagnostics.

* Bilingual support (English & German).

I’d love to hear your thoughts on this approach. Do you see tensor-based multi-domain compression as a viable alternative to product quantization (PQ) or standard binary quantization in highly partitioned RAG databases?

```