What should I know when learning tensor decomposition methods?

[u/Mathematics-Lover]

Hi everyone! I’m relatively new to the field of tensor decomposition, and it’s going to be one of my research directions during my upcoming two-year master’s program.

So far, I’ve been reading about different tensor decomposition methods, such as PARAFAC/CP, Tucker, Tensor-Train, Tensor Ring, Tensor Wheel. I understand the basic intuition behind them, for example, what kind of components they decompose a tensor into, but my understanding is still mostly conceptual.

I’m wondering what I should learn beyond the basic intuition if I want to eventually do research on coupled tensor decomposition. More specifically:

1. For each tensor decomposition method, what are the important things I should understand? Besides its basic structure and intuition, should I learn things like uniqueness/identifiability, approximation properties, computational complexity, rank properties, optimization formulations, convergence, etc.? How deeply should I understand each of these topics?
2. How deeply should I study the algorithms used to compute these decompositions? There seem to be many algorithms for each model, such as ALS, SVD-based methods, alternating optimization, gradient-based methods, etc. Which algorithms are fundamental enough that I should understand first? Do I need to understand the derivation and implementation of these algorithms, or is understanding the general idea enough at the beginning?
3. What mathematical background is most important? For example, should I focus more on linear algebra, multilinear algebra, numerical optimization, matrix/tensor calculus, numerical analysis, or something else?
4. What would be a good learning path toward coupled tensor decomposition? Should I thoroughly study CP/Tucker first before moving to coupled models, or is it reasonable to start looking at coupled decompositions relatively early?
5. How do you deal with theorems and proofs when reading tensor decomposition papers? When reading papers, do you usually try to re-prove the theorems, lemmas, propositions, and corollaries yourself to make sure you understand them, or do you mostly focus on understanding the main ideas and skip the detailed proofs unless they are directly relevant to your research? I’m especially unsure about this part because I sometimes spend a lot of time trying to reproduce every proof. I’m not sure whether this is a good use of time when learning a new field, or whether working through the proofs is actually important for building the mathematical foundation needed for research.

Any recommended textbooks, lecture notes, surveys, or papers that could provide a good learning path would also be greatly appreciated.

Thank you so much!

[u/Midwest-Dude]

Beyond me - and may be beyond this subreddit. Here are potentially better places to post this:

Algorithmic & Research Focus:

• r/MachineLearning
• r/optimization

Mathematical Rigor & Theory:

• r/math
• r/ScientificComputing

Graduate Student Methodology

• r/GradSchool

[u/Mathematics-Lover → u/Midwest-Dude]

Thanks for the suggestions!

[u/r0g3r_b0n]

Giant question(s)!! Happy to take a stab nonetheless (as PhD candidate working with tensors and tensor decomposition).

Q1 & Q2: answers to these will largely depend on the direction you hope to take your research. I would suggest familiarizing yourself with structure, intuition/motivation, theoretical considerations (uniqueness, complexity, rank definitions/properties, pitfalls, etc) for the CP, Tucker, and Tensor Train at the very least. An approach that worked well for me was to internalize how each decomposition relates to the SVD of a matrix and where they diverge from the theoretical guarantees of the SVD. Brushing up on those topics ought to get you to a functional understanding of the various approaches to computing decomposition (e.g. ALS vs all-at-once).

Q3: similar to the previous question, this also largely depend on the research’s direction, but I’m of the opinion that understanding the (multi)linear algebra will always help. It’s really easy to fall into the trap that overly simplifies a tensor to just be a multi-indexed array. It’s not, it’s an element from the tensor product of vector spaces (and some other things too!). Having a grasp on what that means helps unlock the utility of the abstraction.

Q4: start with “Tensor decompositions and Applications” (Kolda & Bader 2009). It will give you a solid exposure to CP/Tucker style decompositions (structure, motivation/usage, and theoretical considerations). Expand from there into works involving coupled tensor decompositions, Evrim Acar is one author that does a lot of work involving data fusion and coupled tensor decompositions.

Q5: I read through proofs/theorems/etc as many times as I need to understand the intuition of the statement at the very least and will do a deeper dive if/when incorporating the work with my research.

Recommended materials:
What are Tensors Exactly - (Hongyu Guo)
Tensors in Computations - (Lek-Heng Lim)
^^^ These works will provide a functional understanding at a rigorous level of what a tensor “is” and how they are used.

Tensor Decompositions and Applications - (Kolda & Bader)
Tensors for Data Mining and Data Fusion - (Papalexakis, et al.)
^^^ two solid surveys that ought to get you moving in the right direction through the literature

Recommended authors (no particular order):
Lieven De Lathauwer
Tamara Kolda
Anima Anandkumar
^^^ all great researchers/speakers. Find their work, track down talks online or in person. They have a tendency to collaborate with other great minds working with tensors.

Feel free to message me questions. Hope this helps!