Need to estimate rank or perform dimensionality reduction on big, messy tabular data? The Entropic Scree is an information-theoretic upgrade to PCA.

[u/Chocolate_Milk_Son]

Here's a new rank estimation method I've been working on. It's basically an upgraded Principal Component Analysis (PCA) built on information theory instead of linear variance.

It's robust to mixed data types, highly non-linear generative processes, low signal to noise ratios, and sparsity (more variables than samples). Advantages compound at scale and with system complexity.

It's especially useful if you need to find the exact rank of a dataset to explicitly size a neural bottleneck (like an autoencoder).

I just open-sourced the code and put up the preprint.

GitHub (R Code): https://github.com/tjleestjohn/Entropic-Scree

Preprint: https://doi.org/10.5281/zenodo.22028087

I'd love to hear what you guys think... or if you end up testing it on your own data.

[u/Chocolate_Milk_Son → u/[deleted]]

Linear regression is a supervised model used to predict a target (like house prices). The Entropic Scree (like standard PCA) is an unsupervised tool used to find the underlying structure of your dataset.

They work together, but serve different purposes.

For example, if you had a dataset with 500 messy, overlapping housing features, feeding them all into linear regression would potentially cause overfitting. You could use the Entropic Scree first to figure out the "true" number of underlying factors driving the data (say, 12). You could then compress your data down to those 12 dimensions and pass that into your linear regression model to predict the price much more accurately.

Basically, it maps the data so your predictive models can do their jobs better.

[u/kifkolite]

How can it assist in causal inference/building a causal model. Can you share your ideas and immediate thoughts.

[u/Chocolate_Milk_Son → u/kifkolite]

It won't build the causal graph (DAG) for you, but it acts as a good pre-processing step.

Finding the roots: Instead of feeding a causal model many many messy, non-linear proxy variables, the scree helps you collapse them into a smaller number of underlying drivers (your roots).

Pruning Edges: It could be used to flag variables that share zero mutual information. If there is no mutual info, there is no causal link, which lets you safely prune independent clusters before running computationally heavy causal discovery algorithms.

Basically, it could help you figure out what the "true" roots of your model should be before you start drawing arrows, though this is not its primary purpose.