Lengths, Angles and Linear Independence

Some Notation

Sequence of Vectors: $(\textbf{v}_1,\textbf{v}_2,\dots, \textbf{v}_n) = (\textbf{v}_j)^n_{j=1}$
Set of Vectors: $\set{\textbf{v}_1,\textbf{v}_2,\dots,\textbf{v}_n} = \{\textbf{v}_j : j\in [n] \}$
Short way of writing column vectors $\begin{pmatrix} v_1 \\ \vdots \\ v_m \end{pmatrix} = (v_i)^m_{i=1}$

What if your $n$ is 0? The range of integers from $1$ to $0$ is empty and an empty sequence $(\textbf{v}_j)_{j=1}^0$ is $()$ whereas an empty sum of vectors $\sum_{j=1}^0 \lambda_j\textbf{v}_j$ is equal to the vector $\textbf{0}$ in the corresponding dimension.

Lengths and Angles

It is a perfectly natural way of thinking to consider vectors as geometric objects that help us move around the space. Therefore we are curious to know how to describe their properties like length (magnitude), position with respect to each other or angles.

Scalar Product (Dot Product)

Definition 1.7 (Scalar Product)

\textbf{v},\textbf{w} \in \mathbb{R}^m: \textbf{v} \cdot \textbf{w} = \begin{pmatrix} v_1 \\ \vdots \\ v_m \end{pmatrix} \cdot \begin{pmatrix} w_1 \\ \vdots \\ w_m \end{pmatrix} = v_1 \cdot w_1 + \dots + v_m \cdot w_m

Notation: we sometimes write $\langle\textbf{v},\textbf{w}\rangle = \textbf{v}\cdot \textbf{w} = \sum^m_{i=1} v_i w_i$

Attention: $\textbf{v}\cdot \textbf{w} \in \mathbb{R}$ that is, the scalar product of two real vectors is a real number!

Note: for $\textbf{v},\textbf{w}\in \R^0$ we have $\textbf{v}\cdot \textbf{w} = 0$

Properties of the Scalar Product

Observation 1.8

Let $\textbf{u,v,w} \in \R^m$ be vectors and $\lambda \in \R$ a scalar. Then

(i) $\textbf{v}\cdot \textbf{w} = \textbf{w} \cdot \textbf{v} \quad \text{(\textit{symmetry})}$
(ii) $(\lambda\textbf{v}) \cdot \textbf{w} = \lambda(\textbf{v}\cdot \textbf{w}) \quad \text{(\textit{scalars can move freely})}$
(iii) $\textbf{u}\cdot(\textbf{v}+\textbf{w}) = \textbf{u}\cdot\textbf{v} + \textbf{u}\cdot\textbf{w}$ and $(\textbf{u}+\textbf{v})\cdot\textbf{w} = \textbf{u}\cdot\textbf{w} + \textbf{v}\cdot\textbf{w} \quad \text{(\textit{distributivity})}$
(iv) $\textbf{v}\cdot\textbf{v} \ge 0 \quad \text{with equality exactly if }\textbf{v} = \textbf{0} \quad \text{(\textit{positive definiteness})}$

These properties will come in handy especially when we start dealing with matrices, vectors and complex equations.

Euclidean Norm

We want to know how long our vectors are. In 2 or 3 dimensions this might be a relatively easy task but we need something that also makes sense for higher dimensions.

Definition 1.9 (Euclidean Norm)

$\norm{\textbf{v}} := \sqrt{\textbf{v}\cdot\textbf{v}}$

v =

All the vectors on the circle have the same norm as the formula of a circle reads $x^2 + y^2 = r^2$ .

In more dimensions:

\text{For } \textbf{v} \in \mathbb{R}^m: \norm{\textbf{v}} = \norm{\begin{pmatrix} v_1 \\ v_2 \\ \vdots \\ v_m \end{pmatrix}} = \sqrt{v_1^2 + v_2^2 + \dots v_m^2} = \sqrt{\sum_{i=1}^m v_i^2}

Unit Vectors: Vectors $\textbf{u}$ such that $\norm{\textbf{u}} = 1$
Producing unit Vectors that point into the same direction as an arbitrary vector: If you take an arbitrary vector $v$ and divide it with its norm, you get a unit vector that is collinear to $v$ .
$\frac{v}{\norm{v}} = \frac1{\norm{v}}\cdot v$
Standard Unit Vectors: $\mathbb{R}^m : \textbf{e}_i = \begin{bmatrix} 0 \\ \vdots \\ 1 \\ \vdots \\ 0 \end{bmatrix} \leftarrow \text{coordinate i}$

There are various different norms. For interesting and commonly used norms you can check the cab norm (1-norm), $p$ -norm and $\infty$ -norm.

Cauchy-Schwarz Inequality

Cauchy-Schwarz inequality tells us that the absolute value of the scalar product of two vectors can not be greater than the product of their norms.

Lemma 1.10 (Cauchy-Schwarz Inequality)

For any $\textbf{v},\textbf{w} \in \mathbb{R}^n:$

|\textbf{v} \cdot \textbf{w}| \le \norm{\textbf{v}} \norm{\textbf{w}}

The expression holds with exact equality if two vectors are collinear.

Let’s see that Cauchy-Schwarz holds with equality for two collinear vectors. Let $\mathbf{v} \in \R^n$ be a vector and $\lambda \in \R$ a scalar. Now set $\mathbf{w} = \lambda\mathbf{v}$ . We know $\mathbf{w}$ is a scalar multiple of $\mathbf{v}$ hence they are collinear. Now use Cauchy-Schwarz:

\begin{aligned} |\textbf{v} \cdot \textbf{w}| &\le \norm{\textbf{v}} \norm{\textbf{w}} && \text{(substitute } \mathbf{w} = \lambda\mathbf{v} \text{)} \\ |\textbf{v} \cdot \lambda\mathbf{v}| &\le \norm{\textbf{v}} \norm{\lambda\textbf{v}} && \text{(Use Observation 1.8 ii and Def of Norm)} \\ |\lambda|\cdot |\textbf{v} \cdot \mathbf{v}| &\le \sqrt{\textbf{v}\cdot\textbf{v}} \cdot \sqrt{\lambda\textbf{v}\cdot\lambda\textbf{v}} \\ |\lambda|\cdot |\textbf{v} \cdot \mathbf{v}| &\le |\lambda|\cdot \textbf{v} \cdot \mathbf{v} \end{aligned}

Two sides are equal since $\mathbf{v}\cdot \mathbf{v}$ is always non negative as sum of squares (see Euclidean norm). $\square$

Angles

You can get help from the Cauchy-Schwarz inequality to determine the angle between two vectors. Again, to come up with a more general method to measure the angle between two vectors is useful for higher dimensions where it might be relatively more difficult to visualize angles in comparison to 2 dimensions.

Definition 1.12 (Angle)

Let $\textbf{v}, \textbf{w} \in \mathbb{R}^m$ be two nonzero vectors. The angle between them is the unique $\alpha$ between 0 and $\pi$ (180 degrees) such that:

\cos(\alpha) = \frac{\textbf{v}\cdot \textbf{w}}{\norm{\textbf{v}} \cdot \norm{\textbf{w}}} \in [-1, 1]

Note that the absolute value of the fraction can not be greater than 1. (comes from the Cauchy-Schwarz inequality) This is good, since cosine can only have values in the interval $[-1,1]$ . So this is a well defined formula. We also have

\alpha = \arccos\left( \frac{\textbf{v}\cdot \textbf{w}}{\norm{\textbf{v}} \cdot \norm{\textbf{w}}}\right)

v = w =

Angles on Higher Dimensions and Cosine Similarity

Angles on higher dimensions might seem counterintuitive or vague. However we actually use angles in higher dimensions. In essence, the angle between two vectors in any dimension mean how similar the vectors are aligned. If the angle is close to $0$ then the vectors are aligned in a more similar way. On the other hand if the angle is close to $\pi$ then the vectors are almost pointing in the opposite directions. You can read more about the use case of this by searching the cosine similarity.

Perpendicular Vectors and Hyperplanes

Perpendicular vectors are the vectors that have an angle of 90 degrees (cosine equal to 0) between them. A more commonly used term is orthogonal. Orthogonality is an important concept in linear algebra and it will have its own section in the lecture notes. We identify orthogonal vectors as in the following:

Definition 1.13

\textbf{u, v} \in \mathbb{R}^m \text{ are called orthogonal if } \textbf{u} \cdot \textbf{v} = 0

Collecting all the vectors that are orthogonal to a vector gives us an hyperplane:

Definition 1.14 (Hyperplane)

Let $\textbf{d}\in \R^m, \textbf{d} \neq \textbf{0}$ . The set

H_d = \set{\textbf{v}\in \R^m : \textbf{v}\cdot \textbf{d} = 0}

is called a hyperplane through the origin.

Triangle Inequality

Lemma 1.15

Let $\textbf{v}, \textbf{w} \in \mathbb{R}^m$ . Then

\norm{\textbf{v}+\textbf{w}} \le \norm{\textbf{v}} + \norm{\textbf{w}}

which is a generalized version of the triangle inequality we know from two dimensional triangles that also applies for vectors in higher dimensions.

On proofs of these facts:

Especially when you want to prove the Cauchy-Schwarz inequality or the formula for the angle between two vectors it is helpful to prove it only for unit vectors first. This will make things easier. Afterwards you can show that these facts also apply for vectors that are not unit vectors. See lecture notes for the proofs.

Linear Independence

Linear independence is one of the most important concepts in linear algebra and you should definitely develop a good understanding of it. We define linear independence for a collection of vectors.

Definitions

There are three alternative definitions given in the lecture. They are of course all equivalent, but provide us with different points of view. The first one is as follows:

Definition 1.19 (Linear (in)dependence)

Vectors $\textbf{v}_1, \textbf{v}_2, \dots, \textbf{v}_n \in \R^m$ are linearly dependent if at least one of them is a linear combination of the others, i.e. there exists an index $k \in [n]$ and scalars $\lambda_j$ such that

\textbf{v}_k = \sum^{n}_{\substack{j = 1 \\ j \neq k}} \lambda_j \cdot \textbf{v}_j

Otherwise, $\textbf{v}_1, \textbf{v}_2, \dots, \textbf{v}_n$ are linearly independent.

Here are the three alternatives for the definition of linear independence:

Theorem (Equivalent definitions of linear independence)

Let $\textbf{v}_1, \textbf{v}_2, \dots \textbf{v}_n \in \mathbb{R}^m$ . The following statements are equivalent (meaning that they are either all true, or all false).

(i) None of the vectors is a linear combination of the other ones. (This means, the vectors are linearly independent according to Definition above)

(ii) There are no scalars $\lambda_1, \lambda_2, \dots, \lambda_n$ besides $0, 0,\dots,0$ such that we have $\sum_{j=1}^{n} \lambda_j \cdot \textbf{v}_j = \textbf{0}$ . We also say that the vector $\textbf{0}$ can only be written as a trivial linear combination of the vectors.

(iii) None of the vectors is a linear combination of the previous ones.

Be careful, if one of your vectors is the vector $\textbf{0}$ , then the sequence of vectors is linearly dependent. Also the vector $\textbf{0}$ can be written as a linear combination of all sequences of vectors and yes even as a linear combination of the empty sequence of vectors.

Lemma 1.22 (Uniqueness of the linear combination)

Let $\textbf{v}_1, \textbf{v}_2, \ldots, \textbf{v}_n \in \mathbb{R}^m$ be linearly independent, $\textbf{v} \in \mathbb{R}^m$ . Let

\textbf{v} = \sum_{j=1}^n \lambda_j \textbf{v}_j \;=\; \sum_{j=1}^n \mu_j \textbf{v}_j

be two ways of writing $\textbf{v}$ as a linear combination of $\textbf{v}_1, \textbf{v}_2, \ldots, \textbf{v}_n$ . Then $\lambda_j = \mu_j$ for all $j \in [n]$ .

Span

Span of a collection of vectors is the set of all linear combinations possible with these vectors.

Definition 1.23

Let $\textbf{v}_1, \textbf{v}_2, \ldots, \textbf{v}_n \in \R^m$ . Their span is:

\textbf{Span}(\textbf{v}_1, \textbf{v}_2, \dots, \textbf{v}_n) := \{ \sum^n_{j=1} \lambda_j \cdot \textbf{v}_j : \lambda_j \in \R \text{ for all } j \in [n] \}

If you consider your steps as vectors, you could move everywhere with a front step, a right step, and an up step -let’s imagine you could fly-. You could go into the opposite direction by simply multiplying the vector that represents right,up,or forward by -1. Then these vectors would be enough to take you anywhere you want. Their span would be the whole 3 dimensional space. In contrast if you only had the ability to go right-left and front-back, you would be stuck at your altitude forever and could not go anywhere you want (hiking would be out of the window for sure). The span is a two dimensional plane in 3 dimensional space in this example. And lastly, if you only had one of those abilities, say being able to go only to right and left, you would move in a line all the time.

This analogy makes it clear that the span of vectors in $\R^3$ is either a line, a plane, or the whole space. We have one more important case: having no ability to move at all. This corresponds to the case where all your vectors are equal to the vector $\textbf{0}$ . Then the span consists of the origin.

Lemma 1.24

Let $\textbf{v}_1, \textbf{v}_2, \ldots, \textbf{v}_n \in \mathbb{R}^m$ , and let $v \in \mathbb{R}^m$ be a linear combination of $\textbf{v}_1, \textbf{v}_2, \ldots, \textbf{v}_n$ . Then

\underbrace{\textbf{\textit{Span}}(\textbf{v}_1, \textbf{v}_2, \ldots, \textbf{v}_n)}_{S} \;=\; \underbrace{\textbf{\textit{Span}}(\textbf{v}_1, \textbf{v}_2, \ldots, \textbf{v}_n, \textbf{v})}_{T}.

You can see this in the example above. While you can move in 2 directions front-back and right-left, the ability to move diagonally would not take you to somewhere new. You could already produce the behavior of the newly added vector with other vectors. In other words, you can take the vector out of the set, which can be written as a linear combination of the others, and the span will stay the same (Corollary 1.25).

Lemma 1.26

let $\textbf{v}_1, \textbf{v}_2, \ldots, \textbf{v}_m \in \R^m$ be linearly independent. Then

\textbf{\textit{Span}}(\textbf{v}_1, \textbf{v}_2, \ldots, \textbf{v}_m) = \R^m

This lemma lays the ground for the concept of a basis of a vector space. Even though we have not yet developed enough linear algebra to understand this in terms of a basis of a vector space, we can think of it as follows: Remember the analogy of moving from above. If you have $m$ linearly independent vectors, this means you have $m$ distinct abilities, each of which enable you to go into a new dimension you could not go before. Now we can wrap up our intuition by saying there are only $m$ “directions” to go in $\R^m$ .

I strongly recommend you to read the proof of this fact from the lecture notes.

Hints

Hints for this week’s assignments:

Will be solved in class.
For b and c, use different definitions of the linear independence out of the 3 we have.
Think of the vectors $\textbf{1} = \begin{bmatrix} 1 \\ 1 \\ \vdots \\ 1 \end{bmatrix}$ for a and $\begin{bmatrix}\sqrt{1} \\ \sqrt{2} \\ \vdots \\ \sqrt{n} \end{bmatrix}$ for b.
For a remember the definition of linear (in)dependence - With which $\lambda_i$ ‘s can you have $\sum_{i=1}^m \lambda_i \cdot v_i = 0$ ? For b try to do the same thing and show in that case every coefficient must be equal to $0$ .
We have a formula for this. Use that. Simplify the fraction in a clever way. Do not forget to use the fact that $x+y+z = 0$ .
See the proof of the Fact 1.5 from lecture notes. Make sure that in your calculations the denominator never gets to be 0.