Vector inner product


This is basically a middle school math textbook concept, the inner product of two vectors is very simple, let’s just look at the formula to review:


In this case, X and Y are both n-dimensional vectors, and two vectors can compute the inner product only if they have the same dimension. As can be seen from the above formula, the inner product of two vectors is equal to the sum of the products of the components of the two vectors corresponding to each dimension.

To distinguish matrix multiplication from ordinary multiplication, we usually write the inner product of two vectors as:.

Now, there’s a very important property here, for a vector, we can use Euclidean formulas to figure out its length. Further, we can express the inner product of vectors by the length of vectors and the included Angle between vectors, as follows:


One of theIt’s the Angle between the x vector and the y vector, which is pretty straightforward for vectors in three dimensions and below. For higher-dimensional vectors, it’s hard to imagine what it means physically. But it doesn’t matter, we can also say there’s one between the vectorsGeneralized hyperspaceAn Angle inside. In machine learning, we usually use this AngleThe similarity between vectors. The more similar these two vectors are, the smaller the Angle between them, the bigger the cosine cosine. So we can use the cosine between two vectors to reflect how similar they are. That’s where the cosine comes from.


Orthogonal vector


And you can see from the formula above, the inner product of the vectors is equal to the lengths of the two vectors times the Angle between the vectors. For non-zero vectors, they all have to be greater than 0. So the inner product of two vectors depends entirely on the Angle between them.

ifLess than 90 degrees, so, then the inner product is positive. ifGreater than 90 degrees, cosine is negative. So we can tell whether the Angle is acute or obtuse by looking at the cosine. Since speaking of included Angle, naturally can not leave a special case —vertical.

If you’re in two dimensions, and the Angle between these two vectors is 90 degrees, then obviously these two vectors are perpendicular. The same thing is true in higher dimensions, but instead of saying perpendicular, we use a different word, orthogonal. If the inner product of two non-zero vectors is 0, the vectors are orthogonal.


Orthogonal vector set


Now that we know what the orthogonal vectors are, we know what the orthogonal vectors are. An orthogonal vector set is a set of pairwise orthogonal and non-zero vectors.

If the n-dimensional set of vectors:These two are orthogonal, so they must be linearly independent. That is, there is no set of coefficients that are not zeroThat:


And this is easy to prove, because none of the vectors are 0, we just multiply both sides of this equation by any vector, let’s say we multiply by 1, 0. And since it’s orthogonal to everything else, everything else is 0. For this equation to be true, it must be:


Due to theIt’s not zero, soIt must not be zero. In order for this to work, it has to be zeroTo zero.


Normal orthogonal basis


Let’s fuse the idea of orthonormal vectors with the idea of basis, if you have a set of vectorsIt’s a basis for our vector space V. If between themOrthogonal to each other, then they are said to be a normal orthogonal basis.

For vector A, we can easily find its coordinates of each dimension under normal orthogonal basis:


That is to say that the coordinates of some dimension of vector A, with the normal orthogonal basis, are equal to the inner product of that with the orthogonal basis vectors of that dimension.

So let’s say we already know that a basis for our vector space V is 1, 2, 3How do we find the normal orthogonal basis for V?

And we’re going to use an algorithm called Schmidt’s algorithm. With this algorithm, we can find the orthogonal basis of a vector space from a basis.

The algorithm is so simple that we can just write the formula for it:


Let’s just take any two b vectors and multiply them by them and we know that they’re orthogonal. So, we just need to unify the b vector set, and we can figure out our normal orthogonal basis.

That is:


This algorithm is not hard, but it’s important. Many dimensionality reduction algorithms in machine learning are related to Schmidt orthogonalization.


Orthogonal matrix


We said earlier, when we introduced matrices, that we can view a matrix as the structure of a particular set of vectors. Similarly, we can consider a normal orthogonal basis vector set to be a matrix, and then the matrix is called an orthogonal matrix.

It has the following properties:


Where I is the identity matrix, if and only if every column in A is an identity column vector, and it’s orthogonal.

Finally, let’s look at the properties of orthogonal matrices. It has three main properties:

  1. If A is an orthogonal matrix, thenIs also an orthogonal matrix, and.

  2. If A and B are orthogonal matrices, and they have the same order, then AB is also orthogonal.

  3. If A is an orthonormal matrix, the determinant of the vector y remains the same after the transformation of A.

All three of these properties are very simple, and we can basically derive them directly from the properties of orthogonal matrices, or they’re very intuitive, and they match our intuition. It doesn’t matter how you derive it, it’s more important for an algorithm engineer to understand what these concepts mean and relate them to the function of the algorithm model, that’s the most important thing.

Today’s content about orthogonal vector and matrix is over here, if you feel that something has been gained, please feel free to click on the attention or forward, your support is my biggest motivation.