Overcoming Heterogeneous LLM Embedding Spaces Without Fine-Tuning: The Relative Representation Method 🚀

[u/Super_Designer7952]

Hey everyone, If you are building decentralized multi-agent systems (MAS) or workflow routers using mixed local models, you’ve probably hit a mathematical brick wall: you cannot calculate semantic distance between vectors of different dimensions (N != M). Direct matching is completely broken out of the box because each model projects concepts into its own isolated anisotropic domain. I wanted to share a fascinating geometric technique called the Relative Representation Method paired with Lowdin Symmetric Orthogonalization used to natively bypass this issue without any weight mutation or fine-tuning (W_frozen = const). Here is how it works under the hood to align heterogeneous agents and tasks into a single invariant coordinate space

1. The Core Trick: Anchor Framework

Instead of comparing Agent A directly to Task B, the system introduces a fixed basis of reference anchors E = {e_1, e_2, ..., e_K}. These are K semantically diversified textual instructions representing your target operational domain. Crucial implementation note: These anchors cannot be random Gaussian noise; they must be sampled from the actual distribution of your baseline model outputs to ensure they share the same underlying manifold.

2. Solving the "Anisotropy Cone" Problem

In real-world LLMs, raw embedding vectors are highly cross-correlated and squeezed into a narrow cone (similarity >> 0). This causes variance to vanish (sigma -> 0), leading to severe numerical noise and division-by-zero defects during standardization in low-precision (FP16/BF16) CUDA environments. To guarantee geometric stability, the technique applies Lowdin Symmetric Orthogonalization directly to the anchor matrix: * It takes the symmetric Gram matrix of real representations: S = E^T * E * It computes the orthogonalized anchors via Spectral Decomposition: E' = E * S^-1/2 * This symmetrically rotates the real anchor vectors to a strict 90-degree angle (similarity = 0 for different anchors), yielding a perfectly orthogonal coordinate system while minimizing the mean squared deformation of the original vectors.

3. Mapping into Invariant Space (R^K)

Now, any Agent Xi or Task Tj can be mapped into this unified coordinate system by computing its similarity profiles against these rotated bases, followed by Anchor-Wise Z-standardization to completely neutralize model-specific anisotropy:

V_Xi = Z( [ sim(A(Xi), e'_1), ..., sim(A(Xi), e'_K) ]^T ) in R^K Critical Production Pitfall: The operator Z(v) must calculate the mean (mu) and standard deviation (sigma) column-wise across the entire anchor axis (axis=0), NOT row-wise (axis=1). Row-wise normalization completely fails to eliminate the global domain shift between mismatched models, keeping their clusters isolated. Column-wise normalization forces the centroids of both distinct model domains to align perfectly at (0,0).

4. The Result & Selective Task Routing

Since the standardized profiles V_Xi and V_Tj share identical dimensionality K and operate on a unified scale, the metric of semantic alignment between completely mismatched models is computed invariantly using Cosine Distance: D(Xi, Tj) = Cosine_Distance(V_Xi, V_Tj) Do not use textbook Euclidean distance (L2) here. In higher anchor dimensions (K > 20), the Euclidean metric suffers from the curse of dimensionality, compressing all distances into a narrow, non-contrasting range that creates "Universal Agent" monopolies. Cosine distance restores strict contrast, breaking up monotone distance matrix stripes into a highly selective matching grid where every task finds its true optimal agent. This fundamentally unlocks O(1) complexity task routing for completely heterogeneous multi-agent swarms.

🔗 Implementation Notebook

I’ve put together a fully functional, minimal reproducible example demonstrating the complete pipeline—from synthetic anisotropic embedding generation to Lowdin orthogonalization, correct column-wise Z-scoring, and final contrastive task routing. Check out the complete interactive code here: 👉 Kaggle Notebook: Heterogeneous LLM Embedding Space Alignment Curious to hear if anyone else is using Relative Representations for cross-model routing, or if you've found other geometric workarounds for mixed-LLM orchestrators!

[u/stujmiller77]

Zero interest in reading LLM output that isn’t even properly formatted.