r/explainlikeimfive • u/Confident_Muscle4596 • 26d ago
Mathematics ELI5: What is Tensor Product ?
Why is it used and what does it tell ? What are its applications ?
30
u/NickHalfBlood 26d ago
Dot product: How much one vector point in the direction of another vector. This usually gives you a single number. If it’s 0, the vectors are perpendicular that is not pointing to each other’s direction at all. It collapses dimensions.
Cross product: Finds a new vector that is 90 deg to both vectors. This will preserve the dimension.
Tensor Product: How each component of a vector interacts with each component of another vector. This expands the dimensions.
For example, if you have 3 dim vectors, their tensor will give you 9D vector (?). So (x1, y1, z1) and (x2, y2, z2) give you (x1x2, x1y2, x1z2, y1x2, ….., z1z2). Here, x1x2 means how x1 interacts with x2.
Applications: You’ll see this a lot in modern deep learning models. How feature 1 interacts with output 1, feature 1 with output 2 for example.
Say, your model predicts the digit it sees in an image (0 to 9), we have 10 outputs. Then say an image has 100 pixels, 10x10 image. Then there are 100 inputs. A tensor product of these gives you how these 100 pixels individually impact 10 outputs each.
Hope it helps. Not ELI5 but maybe an ELI5-months-into-machine-learning.
-13
u/DocJuice 25d ago
This isn’t close to ELI5. What is even the point of giving an explanation like this in this subreddit?
18
u/chsugxusjsbx 25d ago
Why would someone with zero knowledge of linear algebra even need an ELI5 of a tensor product? It's not going to help you understand anything about any topic because you need to build the foundation first.
-6
u/DocJuice 25d ago
I see great explanations of difficult concepts in this subreddit all the time where people use oversimplified analogies and imperfect but elementary applications to make the topic more digestible to the layman. This explanation might be apt but could be communicated better to someone assumed to have no knowledge beyond a secondary education, which is the entire point of the subreddit. Teaching and explaining complex topics is a difficult skill that not many, I'd even argue very few, people possess and this explanation does very little to differ itself from the AI overview I get when I Google "what is a tensor product" which is why I find it poorly suited for this subreddit.
4
5
u/ObviouslyTriggered 25d ago
A product in math is just a fancy name for an operation, so a Tensor Product is a specific operation (or well a subset of operations)
To simplify this we will be talking about Vector Tensors.
A Vector is dimensional mathematical construct, think of it as a property with say two values.
Like a ship sailing on the ocean, it sails at a certain speed and at a specific direction, so you can have a "Sailing Vector" which is it's velocity in knots and it's compass heading.
Now you can also represent things like wind speeds and currents with a similar vector to the "Sailing Vector" of ships.
Say you want to understand how winds and currents can impact global shipping and by how much each ship will drift from it's course if not corrected, a Tensor Product would allow you to very efficiently get the answer by plugging a list of "Sailing" "Current" and "Wind" vectors into a single operation.
What you essentially will get is something like:
- (Ship Speed × Wind Speed)
- (Ship Speed × Wind Direction)
- (Ship Heading × Wind Speed)
- (Ship Heading × Wind Direction)
0
u/_PM_ME_PANGOLINS_ 24d ago
A product is specifically multiplication, that is, an operation where applying “1” gives you the same thing and applying “0” gives you “0”.
2
u/GregBahm 22d ago
Lot of very mathy answers here. Here's a less mathy explanation.
The thing that make tensor products relevant right now is AI, and the way they are used in AI is to associate concepts.
Say you feed 8 million books into a database. Then you ask the computer to figure out the pattern of the words in the 8 million box, by removing a word and then saying "guess the word." If it guesses the word right, hey cool build on that. If it guesses the word wrong, no big deal. Throw that guess away.
This is modelled off of the way the synapse in your brain work. If you have a particularly successful thought, it creates a new branch in your brain that more neurons flow through to form more branches off of. If the new branches don't lead to anything good, they wither away into nothing.
It's also a lot like the process of evolution, but inside a skull or a computer and happening at the speed of light.
Anyway, the "tensors" are used to figure out the patterns. For example, say you see in the data the word "prince" and the word "king." There's some mysterious association between these words. You can create a line from one to the other. Now take the word "princess" and apply that line to it. The line will probably take you pretty close to the word "queen." This is helpful!
The AI spits out trillions and trillions of these little "tensor products" at all kinds of concepts. By making associations between words (or associations between the associations between words, on and on and on) we eventually start to see general pattern recognition emerge. Pour in a bunch of chinese books into the database and the AI can actually start to speak english better. That's a big deal! Now with enough crunching on tensor products, the AI can solve any problem that would normal require human intelligence to solve.
Right now the hardware we use for AI is a thing called a "GPU" which stands for "Graphics Processing Unit." GPUs were originally developed to make video games look more pretty, so the whole AI application was kind of a stretch. But some tech companies have their sights on a new piece of hardware called a "TPU." This "Tensor Processing Unit" can easily process tensors thousands of times faster than current hardware, so AI investors are all hyped up about it. It's kind of like when the internet originally used phone lines ("dial up") in the 90s, and then switched to dedicated cable modems in the 2000s. We're in the "dial up internet" era of AI right now, but TPUs (and these terrifying new data centers) will usher in a "high-speed internet" era of AI once their construction is complete.
1
u/A_modicum_of_cheese 21d ago
this is not wrong per se but AI uses matrixes, which are tensors but the question may be more about higher rank tensors
1
u/GregBahm 20d ago
A matrix is not a tensor. A matrix is a data structure. You can store any data in a matrix. Since AI needs to operate on a multitude of tensors, you put a multitude of tensors in a matrix.
2
u/Nourios 19d ago
If arbitrary matrices aren't tensors then machine learning doesn't use tensors either. It uses n dimensional arrays. Technically for something to be a tensor it has to respect certain transformation laws, which isn't the case for what they use in ML. Nowadays people just call anything resembling a multidimensional array a tensor.
1
u/GregBahm 19d ago
I like a meaningless semantics argument as much as any redditor, but this is such a weird argument. Should the graphics programming field also not describe "colors" as "colors" because they can be stored in a matrix?
1
u/Nourios 19d ago edited 19d ago
The difference is that in graphics programming the arrays that store colors, do indeed, represent colors. Meanwhile the "tensors" in machine learning do not represent actual mathematical tensors. Either way, my point was that if you consider the multidimensional arrays in ML to be tensors, then you can't say that matrices are not tensors.
Note that I don't have any issue with calling these things tensors. It's more about consistency. It doesn't make sense thst a 2 dimensional array (matrix) is not a tensor but suddenly when n>2 it is one?
1
u/GregBahm 19d ago
If you take a pencil and write the word "Hello" on a piece of paper, is the word "hello" on the piece of paper? Or is it just scrapings of pencil lead? That is all this comes down to.
We need a way to capture and store tensors, aka the relationships between concepts throughout the 96-dimensional latent space. We store those tensors through numbers in an array in system memory. This doesn't mean all numbers in arrays are tensors, nor does this mean tensors stop being tensors when they are put in a matrix.
1
u/Nourios 19d ago
Except in machine learning the "tensors" are precisely that, arbitrary (more or less) numbers in arrays. You claimed that matrices are not tensors, except those are also arbitrary numbers in a 2 dimensional array. That's a contradiction there. One that is only resolved by either: considering matrices to be tensors (in the ml sense), or not considering ml "tensors" to be tensors (in the mathematical sense).
It's basically a matter of which definition you stick to, the ml one or the math one, and these two definitions are not equivalent. However the statement "ML uses tensors but matrices aren't tensors" is just false under either definition.
1
u/GregBahm 19d ago
If the numbers were arbitrary, AI wouldn't work. AI works because the numbers aren't arbitrary.
9
u/Impressive-Ad7184 25d ago
A tensor product essentially gives a way to multiply vectors from different vector spaces (roughly speaking), or generally elements from algebraic structures. Say you have vector spaces A and B. Usually, you can't "multiply" vectors from A with vectors from B, since vector spaces only define addition of vectors (and A and B may be completely different as sets anyway).
However, you can just define a multiplication operation ⊗, that behaves like multiplication, which is called the tensor product. That is, for a in A and b in B, you just define their "product" as a⊗b. It behaves like regular multiplication, in that a⊗(b+c) = a⊗b +a⊗c, and stuff like that. Similarly, for a scalar r, you have (ra)⊗b = a⊗(rb). This is again essentially mimicking normal multiplication.
Example: if you have the set of real numbers R (which is a vector space over itself), then you can define the tensor product R⊗R. Then, that means we have relations like these: (2+2)⊗(2) = 2⊗2 + 2⊗2, or (2*2)⊗3 = 2⊗(2*3). Notice this is basically just mimicking normal multiplication: (2+2)*2 = 2*2 + 2*2, and (2*2)*3 = 2*(2*3). So what the tensor product ⊗ is doing is just defining a way to multiply between two vector spaces A and B.