Reconstruction and Principal Component Analysis (continued)

ECE 57000 — September 16, 2026

David I. Inouye

Monday reduced PCA to choosing one reconstruction subspace

• Choose an orthonormal decoder \(D=W\) without losing reconstruction possibilities.
• For fixed \(W\), the best encoder is \(E=W^T\): reconstruct by perpendicular projection.
• Choose \(W^TW=I_k\) to minimize \(\|X-XWW^T\|_F^2\) for centered data \(X\).
• Perpendicularity splits fixed total squared length into retained length plus reconstruction error.

Next: review the fixed-total decomposition, then use the SVD to find the best \(k\) directions.

Minimizing reconstruction error means keeping the most squared length

Let \(\widehat{\boldsymbol{x}}_i=WW^T\boldsymbol{x}_i\). The reconstruction and residual are perpendicular.

\(\displaystyle \|X\|_F^2\)

\(=\)

\(\displaystyle \sum_i\|\boldsymbol{x}_i\|_2^2\)

(Sum squared row lengths)

\(\,\)

\(=\)

\(\displaystyle \sum_i\bigl(\|\widehat{\boldsymbol{x}}_i\|_2^2+\|\boldsymbol{x}_i-\widehat{\boldsymbol{x}}_i\|_2^2\bigr)\)

(Pythagoras for each observation)

\(\,\)

\(=\)

\(\displaystyle \sum_i\bigl(\|W^T\boldsymbol{x}_i\|_2^2+\|\boldsymbol{x}_i-\widehat{\boldsymbol{x}}_i\|_2^2\bigr)\)

(\(W^TW=I_k\): decoding preserves length)

\(\,\)

\(=\)

\(\displaystyle \|XW\|_F^2+\|X-XWW^T\|_F^2\)

(Collect the rows into matrices)

\[ \underbrace{\|X-XWW^T\|_F^2}_{\text{error to minimize}} =\underbrace{\|X\|_F^2}_{\text{fixed total}} -\underbrace{\|XW\|_F^2}_{\text{retained amount to maximize}}. \]

Only \(W\) changes. The same centered data \(X\) give the same total squared length for every choice.

The decomposition comes from perpendicularity, not from treating the data as fixed. For each observation x_i=hat(x_i)+(x_i-hat(x_i)), the two terms are perpendicular. For z_i=W^T x_i, norm(Wz_i)^2=z_iT W^T Wz_i=norm(z_i)^2. The rows of XW are (W^T x_i)^T, so their squared norms sum to norm(XW)_F^2. Only after proving the identity do we use that X is fixed in this optimization: we select the basis W, while the observations and their centering stay unchanged. Hence norm(X)_F^{2=sum_{i,j}X_ij}2 has the same value for every candidate W. For example, if the total is 100, retaining 80 leaves error 20; retaining 90 leaves error 10. Maximizing the subtracted quantity minimizes error over the same feasible set W^T W=I_k. There is no extra assumption on the data here.

Three steps connect the PCA problem to its solution

The total squared length is fixed. We only need to maximize what the coordinates retain.

1. Rewrite the objective — Established

Least reconstruction error means most retained squared length.

2. Find the optimal directions — Next

Use the SVD to prove that the leading right singular vectors are optimal.

3. Interpret the solution

Read reconstruction error and coordinate variance from the singular values.

The SVD expresses a data matrix as a sum of simple patterns

Now decompose the centered data itself: \(X=U\Sigma V^T\).

\[ X=\sum_{j=1}^{r}\sigma_j\boldsymbol{u}_j\boldsymbol{v}_j^T, \qquad r=\operatorname{rank}(X). \]

Part	Meaning for the Data Matrix
\(\boldsymbol{v}_j\in\mathbb{R}^d\)	One pattern across measured features.
\(\sigma_j\boldsymbol{u}_j\in\mathbb{R}^n\)	How strongly that pattern appears in each observation.
\(\sigma_j\boldsymbol{u}_j\boldsymbol{v}_j^T\)	A matrix of rank one: one shared pattern with observation-specific weights.

The matrix being decomposed has changed: X has n rows and d columns. U now indexes observations, rather than the output features of the earlier arbitrary operator. Define the outer product entrywise: its (i,l) entry is sigma_j u_ij v_lj. Terms with positive singular values have rank one. r = 0 gives an empty sum.

Truncation keeps some patterns and drops the rest

For \(1\le k\le r\), keeping the leading \(k\) terms gives:

\[ X_k=\sum_{j=1}^{k}\sigma_j\boldsymbol{u}_j\boldsymbol{v}_j^T =U_k\Sigma_kV_k^T. \]

• \(U_k\): first \(k\) columns of \(U\).
• \(\Sigma_k\): the \(k\times k\) diagonal matrix of retained singular values.
• \(V_k\): first \(k\) columns of \(V\).

Does this choice retain the most squared length among all \(k\)-dimensional subspaces?

This introduces truncated factors explicitly, after the full-SVD shape diagrams. The approximation remains n by d even though it is represented by smaller factors. The objective is already established: maximize retained squared length. We have defined a candidate; next prove it is optimal even among rotated subspaces.

The SVD measures how much of each direction we keep

Write \(W=[\boldsymbol{w}_1\ \cdots\ \boldsymbol{w}_k]\) and \(X=U\Sigma V^T\), with \(\sigma_j=0\) for \(j>\operatorname{rank}(X)\).

\(\displaystyle \|XW\|_F^2\)

\(=\)

\(\displaystyle \sum_{\ell=1}^k\|X\boldsymbol{w}_\ell\|_2^2\)

(Sum squared column lengths)

\(\,\)

\(=\)

\(\displaystyle \sum_{\ell=1}^k\sum_{j=1}^d\sigma_j^2(\boldsymbol{v}_j^T\boldsymbol{w}_\ell)^2\)

(\(V^T\) reads coordinates; \(\Sigma\) scales them; \(U\) preserves length)

\(\,\)

\(=\)

\(\displaystyle \sum_{j=1}^d\sigma_j^2\Bigl[\sum_{\ell=1}^k(\boldsymbol{v}_j^T\boldsymbol{w}_\ell)^2\Bigr]\)

(Swap sums; factor out \(\sigma_j^2\))

\(\,\)

\(=\)

\(\displaystyle \sum_{j=1}^d\sigma_j^2\alpha_j\)

(The bracketed sum is \(\alpha_j\))

• \(\alpha_j=\|W^T\boldsymbol{v}_j\|_2^2\): amount of direction \(\boldsymbol{v}_j\) kept; \(0\le\alpha_j\le1\).
• \(\sum_j\alpha_j=\sum_{\ell=1}^k\|\boldsymbol{w}_\ell\|_2^2=k\): \(k\) units in total.

Use the full orthonormal right basis V. Define sigma_j=0 for j>rank(X), including extra right directions when d>n. The nested sum follows directly from the norm of Sigma V^T w, with no covariance or trace prerequisites. Alpha_j is the squared length of the projection of the unit vector v_j; Pythagoras bounds it by one. For the total, interchange finite sums and use sum_j(v_j^T w_l)^2=norm(VT w_l)^2=norm(w_l)2=1. Emphasize that alpha can be fractional: the argument includes every rotated candidate subspace, not just selections of singular vectors.

Directions beyond \(k\) contribute at most the cutoff weight

Order \(\sigma_1^2\ge\cdots\ge\sigma_d^2\). Recall \(\alpha_j\ge0\) and \(\sum_j\alpha_j=k\).

\(\displaystyle \sum_{j=1}^d\sigma_j^2\alpha_j\)

\(=\)

\(\displaystyle \sum_{j=1}^k\sigma_j^2\alpha_j+\sum_{j>k}\sigma_j^2\alpha_j\)

(Separate the first \(k\) directions)

\(\displaystyle \,\)

\(\le\)

\(\displaystyle \sum_{j=1}^k\sigma_j^2\alpha_j+\sigma_k^2\sum_{j>k}\alpha_j\)

(For \(j>k\): \(\sigma_j^2\le\sigma_k^2\), \(\alpha_j\ge0\))

\(\displaystyle \,\)

\(=\)

\(\displaystyle \sum_{j=1}^k\sigma_j^2\alpha_j+\sigma_k^2\Bigl(k-\sum_{j=1}^k\alpha_j\Bigr)\)

(All \(\alpha_j\) sum to \(k\))

Why the inequality? Replace each smaller weight by \(\sigma_k^2\). Multiplying by a nonnegative \(\alpha_j\) preserves the inequality.

The cutoff is the kth largest squared singular value. Only the tail weights are increased; the first k terms stay unchanged. The tail has total allocation k minus the allocation in the first k directions. This applies to fractional alpha values too. If k=d, the tail is empty and this inequality is equality.

Each leading direction contributes at most one unit

Continue from the previous bound. For \(j\le k\), \(\sigma_j^2-\sigma_k^2\ge0\) and \(\alpha_j\le1\).

\(\displaystyle \|XW\|_F^2\)

\(\le\)

\(\displaystyle \sum_{j=1}^k\sigma_j^2\alpha_j+\sigma_k^2\Bigl(k-\sum_{j=1}^k\alpha_j\Bigr)\)

(Previous slide’s bound)

\(\displaystyle \,\)

\(=\)

\(\displaystyle k\sigma_k^2+\sum_{j=1}^k(\sigma_j^2-\sigma_k^2)\alpha_j\)

(Expand and regroup)

\(\displaystyle \,\)

\(\le\)

\(\displaystyle k\sigma_k^2+\sum_{j=1}^k(\sigma_j^2-\sigma_k^2)\cdot1\)

(Nonnegative coefficient; \(\alpha_j\le1\))

\(\displaystyle \,\)

\(=\)

\(\displaystyle \sum_{j=1}^k\sigma_j^2\)

(The \(k\) copies of \(\sigma_k^2\) cancel)

Thus every feasible \(W\) retains at most \(\sum_{j=1}^k\sigma_j^2\).

Regrouping subtracts sigma_k squared from each of the leading weights and adds k sigma_k squared. Each difference is nonnegative because the singular values are ordered. The second inequality replaces alpha_j by its upper bound one. Expand the final sum: sum sigma_j squared minus k sigma_k squared, then cancel. A universal upper bound alone does not yet prove a solution; next we must exhibit a feasible W that attains it.

Choosing \(W=V_k\) attains the bound

Choose \(W=V_k=[\boldsymbol{v}_1\ \cdots\ \boldsymbol{v}_k]\). These columns are orthonormal, so \(W^TW=I_k\).

\(\displaystyle \alpha_j\)

\(=\)

\(\displaystyle \sum_{\ell=1}^k(\boldsymbol{v}_j^T\boldsymbol{v}_\ell)^2\)

(Substitute \(\boldsymbol{w}_\ell=\boldsymbol{v}_\ell\))

\(\displaystyle \,\)

\(=\)

\(\displaystyle \begin{cases}1,&j\le k,\\0,&j>k.\end{cases}\)

(Inner products are \(1\) for \(j=\ell\) and \(0\) otherwise)

For \(j\le k\), exactly one term is \(1\). For \(j>k\), every term is \(0\).

\[ \|XV_k\|_F^2=\sum_{j=1}^d\sigma_j^2\alpha_j=\sum_{j=1}^k\sigma_j^2. \]

The upper bound is achieved: \(V_k\) maximizes retained squared length and therefore minimizes reconstruction error.

Both inequalities are equalities for this choice: the tail alpha values are zero, and every leading alpha value is one. Thus this is a feasible global optimum, not merely a plausible choice. The encoder is V_k transpose and the decoder is V_k: Z=XV_k=U_k Sigma_k and Xhat=XV_k V_k transpose=X_k. For k above rank, right null directions complete the basis and error is zero. Ties may permit other optimal subspaces. This is a mathematical solution; we have not yet discussed a numerical algorithm.

Three steps connect the PCA problem to its solution

The leading right singular vectors attain the optimum. Now read what this solution keeps and loses.

1. Rewrite the objective — Established

Least reconstruction error means most retained squared length.

2. Find the optimal directions — Established

Use the SVD to prove that the leading right singular vectors are optimal.

3. Interpret the solution — Next

Read reconstruction error and coordinate variance from the singular values.

Discarded singular values quantify what reconstruction loses

Subtract the optimal retained squared length from the fixed total:

\[ \|X-X_k\|_F^2=\underbrace{\sum_{j=1}^{r}\sigma_j^2}_{\text{total}}-\underbrace{\sum_{j=1}^{k}\sigma_j^2}_{\text{retained}}=\sum_{j>k}\sigma_j^2. \]

For singular values \(6,3,1\):

Retained Features	Squared Reconstruction Error
\(k=1\)	\(3^2+1^2=10\)
\(k=2\)	\(1^2=1\)
\(k=3\)	\(0\)

Apply the proved total-minus-retained identity with W=V_k. Orthogonal SVD factors preserve Frobenius norm, giving total squared length sum_j sigma_j^2. No new matrix inner product or separate approximation theorem is needed. The example shows the error-versus-dimension tradeoff. Use k<=r here; retaining additional null directions beyond rank leaves error zero.

Squared singular values give the variance of each PCA coordinate

The \(j\)th coordinate across observations is \(Z_{:j}=X\boldsymbol{v}_j=\sigma_j\boldsymbol{u}_j\). Since \(X\) is centered, this coordinate also has mean zero.

\(\displaystyle \operatorname{Var}_{\mathrm{sample}}(Z_{:j})\)

\(=\)

\(\displaystyle \frac{1}{n-1}\sum_{i=1}^n Z_{ij}^2\)

(Variance of a centered coordinate; \(n>1\))

\(\,\)

\(=\)

\(\displaystyle \frac{\sigma_j^2}{n-1}\|\boldsymbol{u}_j\|_2^2\)

(Substitute \(Z_{:j}=\sigma_j\boldsymbol{u}_j\))

\(\,\)

\(=\)

\(\displaystyle \frac{\sigma_j^2}{n-1}\)

(\(\boldsymbol{u}_j\) is a unit vector)

The leading PCA coordinates therefore have the largest variances. This interprets the solution we already proved.

Define Z_{:j} as column j of Z, containing one scalar coordinate per observation. It is centered because sum_i x_i=0 implies sum_i v_j^T x_i=0. Sample variance divides the sum of squared centered entries by n-1. Work with j<=k<=r as in the truncated factor slide. We are naming the statistical meaning of squared length, not proving the reconstruction/retained-energy equivalence again.