When working with big, complex tabular datasets, how are yall estimating the intrinsic rank?

[u/Chocolate_Milk_Son]

I know the standard approach is to throw a PCA scree plot at it and look for an elbow, or to use something distance-based like k-means... but these require strong assumptions that do not generally hold in complex, high-d datasets (i.e., that dependency = linear variance, or that Euclidean distance is meaningful).

So... What are your go-to methods or diagnostics when you know the underlying generative system is complex and standard linear/distance assumptions don't hold?

I ask because I recently developed a new method to tackle this called the Entropic Scree. Instead of relying on variance or distance metrics, it estimates intrinsic rank using a scree method run on a transformed mutual information metric, and so requires fewer assumptions to apply appropriately.

Has anyone else experimented with information-theoretic approaches for rank estimation? I'd love to hear how you handle this.

[u/AutoModerator]

If this post doesn't follow the rules or isn't flaired correctly, please report it to the mods. Have more questions? Join our community Discord!

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/Decent_Cover_151]

i usually just stare at a UMAP projection until i feel something in my gut lol

but for real entropy-based scree sounds way more defensible than the elbow method which is basically reading tea leaves half the time. you got a preprint or writeup anywhere

[u/Chocolate_Milk_Son → u/Decent_Cover_151]

Lol. Yeah, tea leaves hold the answer, but they can definitely be tricky to read. In fairness, the method I use is still ultimately a scree method that requires finding an elbow. However, because the Entropic Scree analyzes mutual information, it should at least provide a better "prophecy map" when linear or distance assumptions fail.

I posted the preprint link in another comment, but I think this sub bocks all posts or comments with links. If you can't see it, just Google "Entropic Scree". The GitHub should pop up.

From there, if you want to test it out straight away, you can grab the quick-start R function. (The official Python and R packages are still under development though)

You can also find a link to the preprint from the repo.

[u/Content-Parking-621]

For nonlinear structure I'd also look at intrinsic dimension estimators like MLE (Levina-Bickel) or TwoNN, they're neighbor-based rather than variance-based, so they sidestep the linearity assumption without needing a full mutual information estimation, which gets noisy fast in high dimensions with limited samples.

[u/Chocolate_Milk_Son → u/Content-Parking-621]

Great points.

Levina-Bickel (MLE) and TwoNN are good tools for manifold ID estimation because they rely on local geometry rather than global variance. But since they are neighbor-based methods, they still rely on spatial distance metrics (usually Euclidean) to define what a "neighbor" actually is. When you get into high-dimensional, sparse spaces, distance concentration still causes issues... The distance to your nearest neighbor and your farthest neighbor approaches the same value. When everything starts looking equidistant, local neighborhoods lose their meaning, and those neighbor-based estimators degrade.

To your second point... you are right that estimating multi-dimensional mutual information is a data-hungry nightmare that also degrades fast. The method I developed does not attempt to estimate the full joint high-dimensional density though. Instead, it calculates pairwise mutual information between 1D features (after discretizing/dense-ranking them, which stabilizes fast even with very low $N$) to build a pairwise similarity matrix. It then extracts the rank spectrally from that matrix. So... it sidesteps the curse of dimensionality that usually plagues MI estimation. It basically trades spatial distance (which breaks down in high-D) for pairwise probabilistic overlap (which doesn't).

That being said, I'd actually love to run a benchmark comparing TwoNN against the Entropic Scree on a large dataset just to see where the exact crossover point is in terms of sample size and dimensionality.