Reconstruction and Principal Component Analysis: Formulation and Fixed-Total Decomposition

ECE 57000 — September 14, 2026

David I. Inouye

Friday connected reconstruction to the decoder’s subspace

• Center the data: \(X\) contains observations as rows.
• Encode and decode: \(\boldsymbol{z}=E\boldsymbol{x}\), \(\widehat{\boldsymbol{x}}=D\boldsymbol{z}\).
• Every reconstruction lies in the span of the columns of \(D\).

We want to minimize \(\|X-XE^TD^T\|_F^2\). Changing the basis changes coordinates, not the subspace.

Next: prove that an orthonormal decoder loses no reconstruction possibilities; then find its best encoder.

September 11 ended on the QR slide (22/47), with its initial equality visible. The instructor explicitly deferred the derivation at 55:26 and again near 56:09. Repeat that exact slide next and complete the proof before advancing. The remaining topic material is materialized below from the canonical PCA partial; meeting-specific recap belongs only here.

QR lets us choose an orthonormal decoder without changing reconstruction

Start with any encoder \(E_0\in\mathbb{R}^{k\times d}\) and decoder \(D_0\in\mathbb{R}^{d\times k}\), where \(k\le d\). Suppose \(D_0\) does not have orthonormal columns.

The thin QR decomposition writes \(D_0=WR\): \(W^TW=I_k\) and \(R\) is upper triangular. Here \(W\in\mathbb{R}^{d\times k}\) and \(R\in\mathbb{R}^{k\times k}\).

Can we choose \(D=W\) and a new \(E\) to keep every reconstruction the same?

\(\displaystyle D_0E_0\boldsymbol{x}\)

\(=\)

\(\displaystyle (WR)E_0\boldsymbol{x}\)

(QR of the original decoder)

\(\,\)

\(=\)

\(\displaystyle W(RE_0)\boldsymbol{x}\)

(Regroup the factors)

\(\,\)

\(=\)

\(\displaystyle DE\boldsymbol{x}\)

(Choose \(D=W\), \(E=RE_0\))

Same reconstruction for every input: an orthonormal decoder is without loss of generality. Next, fix \(D=W\) and find its best encoder.

Introduce both original operators before factoring the decoder. QR is another standard matrix decomposition, introduced here because it solves a specific problem. The factor usually named Q is called W to match our subspace notation. In thin QR for d by k with k<=d, W has k orthonormal columns and R is k by k upper triangular (entries below the diagonal are zero). If D0 has full column rank, W is a basis of exactly its column span and R is invertible. Thin QR also exists for deficient rank: W spans a k-dimensional space containing the decoder span, and R is singular. We never invert R, so the reconstruction identity holds in either case. For a general rectangular matrix, full QR uses a square orthogonal Q and an upper-trapezoidal R; avoid saying every rectangular R is square triangular. We do not need to teach the QR algorithm here. Triangularity is a property of this standard factorization, not needed for the regrouping. The new encoder RE0 remains arbitrary; its optimal choice W^T is derived next. With row-stacked inputs the identity is X E0^T D0^T = X E^T D^T.

Four steps lead to the PCA formulation

We may choose an orthonormal decoder without changing what we can reconstruct.

1. Define what a good reconstruction means — Established

Choose a representation size and measure reconstruction error.

2. Choose a convenient decoder basis — Established

An orthonormal basis can represent the same reconstruction subspace.

3. Find the best encoder for that decoder — Next

The closest point in the subspace determines the coordinates.

4. Formulate the subspace choice.

Combine these results into the PCA optimization problem.

In 2D, the perpendicular foot is the best reconstruction

Any unit \(\boldsymbol{u}\) works. Illustrate with \(\boldsymbol{u}=[1,0]^T\), so \(\boldsymbol{q}=z\boldsymbol{u}=[z,0]^T\).

\(\displaystyle \|\boldsymbol{x}-\boldsymbol{q}\|_2^2\)

\(=\)

\(\displaystyle (x_1-q_1)^2+(x_2-q_2)^2\)

(Squared distance)

\(\,\)

\(=\)

\(\displaystyle (2-z)^2+(2-0)^2\)

(Substitute coordinates)

\(\,\)

\(=\)

\(\displaystyle (2-z)^2+4\)

(Simplify)

Choose \(z=2\): the horizontal error is zero; the height remains.

For any unit u, the optimal coordinate is z=u^T x and the reconstruction is u u^T x. Rotating the line and input together preserves distances, so the horizontal example illustrates the general result. The coordinate calculation uses u=[1,0]^T only to keep its entries simple. The orange arrow is the decoder column u, with unit length measured from the explicitly marked origin. All q candidates are points, not basis arrows. The labels q_a,q_b,q_c distinguish candidate points; q_1,q_2 in the norm definition are the scalar coordinates of an arbitrary q. For candidate coordinates z=1,2,3, squared errors are 5,4,5 respectively. Both moving before and after the perpendicular foot add horizontal squared error. The norm formula is the sum of squares of coordinate differences: no geometric error decomposition is assumed here. The fixed second coordinate of every candidate is zero, so no choice of z can remove the vertical difference 2. The plane example next generalizes this coordinate calculation via Pythagoras.

In 3D, moving within the plane cannot remove the height

Fix \(D=W\) with two orthonormal columns spanning this plane. Which point should we reconstruct?

\[ \|\boldsymbol{x}-\boldsymbol{q}\|_2^2 =\underbrace{\|\boldsymbol{x}-\boldsymbol{p}\|_2^2}_{\text{fixed perpendicular error}} +\underbrace{\|\boldsymbol{p}-\boldsymbol{q}\|_2^2}_{\text{extra error within the plane}}. \]

Choose \(\boldsymbol{q}=\boldsymbol{p}\). Moving in either plane direction only adds error.

This is a 3D plane viewed obliquely; the right-angle marker is constructed in 3D and projected with the plane. The residual is perpendicular to every in-plane direction, not only the displayed orange leg. Any candidate q gives a right triangle in the slice through x,p,q. This is why the line intuition generalizes.

A dot product gives the length along a unit direction

Recall \(\boldsymbol{u}^T\boldsymbol{x}=\|\boldsymbol{u}\|_2\|\boldsymbol{x}\|_2\cos\theta\). Here \(\|\boldsymbol{u}\|_2=1\).

\(\displaystyle a\)

\(=\)

\(\displaystyle \text{hypotenuse}\,\frac{\text{adjacent}}{\text{hypotenuse}}\)

(\(a\) = adjacent length)

\(\,\)

\(=\)

\(\displaystyle \|\boldsymbol{x}\|_2\cos\theta\)

(Definition of cosine)

\(\,\)

\(=\)

\(\displaystyle \boldsymbol{u}^T\boldsymbol{x}\)

(Dot product; \(\|\boldsymbol{u}\|_2=1\))

The foot is \((\boldsymbol{u}^T\boldsymbol{x})\boldsymbol{u}\). This works in any dimension.

Use the standard dot-product identity as a recalled fact, not a new proof. The adjacent length is the hypotenuse times the adjacent/hypotenuse ratio; that ratio is cos(theta). The recalled identity then gives u^T x for unit u. Here x=[2,2]^T, u=[1,0]^T, norm(x)=sqrt(8), theta=pi/4, and a=2. The geometry applies to any two vectors u and x in any dimension: their span has dimension at most two. Normalize a nonzero u first; without normalization, the signed length is u^T x / norm(u), and the foot is u(u^T x)/(u^T u). In general a is signed: an obtuse angle places the foot on the opposite side of the origin. If x=0, the coefficient is zero without defining theta.

Coordinates along the basis directions locate the point in the plane

For a plane with orthonormal basis \(\boldsymbol{w}_1,\boldsymbol{w}_2\), read the two signed lengths:

\[ a_1=\boldsymbol{w}_1^T\boldsymbol{x},\qquad a_2=\boldsymbol{w}_2^T\boldsymbol{x}. \]

Combine the corresponding moves within the plane:

\[ \boldsymbol{p}=a_1\boldsymbol{w}_1+a_2\boldsymbol{w}_2 =\begin{bmatrix}\boldsymbol{w}_1&\boldsymbol{w}_2\end{bmatrix} \begin{bmatrix}a_1\\a_2\end{bmatrix}=W\boldsymbol{a}. \]

• Perpendicular basis directions let us read each coordinate independently.
• For \(k\) orthonormal directions, the same construction gives \(\boldsymbol{a}=W^T\boldsymbol{x}\) and \(\boldsymbol{p}=W\boldsymbol{a}\).

Next, verify the geometry algebraically: is \(\boldsymbol{x}-\boldsymbol{p}\) perpendicular to every decoder direction?

Do not sum distances; sum the scaled basis vectors. Each scalar a_j measures a signed displacement along w_j. A move along w2 leaves the w1 coordinate unchanged, and vice versa, because w1^T w2=0. Thus p has the same two in-plane components as x. For x=[2,1,2]^T and the xy plane, a=[2,1]^T and p=[2,1,0]^T. Distinguish p, the foot on the full plane, from a1 w1, the foot on just one line. The next slide verifies the perpendicular residual rather than assuming that stacking these coordinates automatically proves optimality. The later error split proves p is the unique closest reconstruction.

The residual is perpendicular to every decoder direction

Fix \(D=W\), with \(W^TW=I_k\). For one centered input, define:

\[ \boldsymbol{a}=W^T\boldsymbol{x},\qquad \boldsymbol{p}=W\boldsymbol{a},\qquad \boldsymbol{r}=\boldsymbol{x}-\boldsymbol{p}. \]

The point \(\boldsymbol{p}\) lies in the decoder’s subspace. Check its residual:

\(\displaystyle W^T\boldsymbol{r}\)

\(=\)

\(\displaystyle W^T(\boldsymbol{x}-W\boldsymbol{a})\)

(Substitute the residual)

\(\displaystyle \,\)

\(=\)

\(\displaystyle W^T\boldsymbol{x}-W^TW\boldsymbol{a}\)

(Distribute)

\(\displaystyle \,\)

\(=\)

\(\displaystyle \boldsymbol{a}-\boldsymbol{a}=\boldsymbol{0}\)

(\(\boldsymbol{a}=W^T\boldsymbol{x}\) and \(W^TW=I_k\))

Each entry is \(\boldsymbol{w}_j^T\boldsymbol{r}=0\): the residual is perpendicular to every basis direction.

a denotes candidate optimal coordinates, not assumed optimal yet. We construct p and verify it is the perpendicular foot before claiming it minimizes error. The residual r depends on x and the chosen subspace, not on the later arbitrary coordinate z. The equalities generalize the fixed height in both illustrations.

Every reconstruction error has two perpendicular parts

For any coordinates \(\boldsymbol{z}\), the decoder returns \(W\boldsymbol{z}\). Insert the perpendicular foot \(\boldsymbol{p}=W\boldsymbol{a}\):

\(\displaystyle \boldsymbol{x}-W\boldsymbol{z}\)

\(=\)

\(\displaystyle (\boldsymbol{x}-\boldsymbol{p})+(\boldsymbol{p}-W\boldsymbol{z})\)

(Add and subtract the same point)

\(\displaystyle \,\)

\(=\)

\(\displaystyle \boldsymbol{r}+(W\boldsymbol{a}-W\boldsymbol{z})\)

(Substitute the two parts)

\(\displaystyle \,\)

\(=\)

\(\displaystyle \boldsymbol{r}+W(\boldsymbol{a}-\boldsymbol{z})\)

(Factor out the decoder)

• \(\boldsymbol{r}\) is the fixed perpendicular part, outside the subspace.
• \(W(\boldsymbol{a}-\boldsymbol{z})\) is the adjustable part, inside the subspace.

Changing \(\boldsymbol{z}\) moves only within the subspace, so it cannot cancel \(\boldsymbol{r}\).

Connect the two terms explicitly to the teal and orange legs in both figures. Their inner product is r^T W(a-z)=(W^T r)^T(a-z)=0. Thus changing coordinates cannot remove the residual: the adjustable vector is perpendicular to it. Unavoidable means for this fixed subspace; choosing a different subspace can change the residual. That is the optimization left for PCA.

The best coordinates remove all adjustable error

Orthonormal columns preserve coordinate lengths: \(\|W\boldsymbol{b}\|_2^2=\boldsymbol{b}^TW^TW\boldsymbol{b}=\|\boldsymbol{b}\|_2^2\).

\(\displaystyle \|\boldsymbol{x}-W\boldsymbol{z}\|_2^2\)

\(=\)

\(\displaystyle \|\boldsymbol{r}+W(\boldsymbol{a}-\boldsymbol{z})\|_2^2\)

(Use the error split)

\(\displaystyle \,\)

\(=\)

\(\displaystyle \|\boldsymbol{r}\|_2^2+\|W(\boldsymbol{a}-\boldsymbol{z})\|_2^2\)

(Pythagoras: the parts are perpendicular)

\(\displaystyle \,\)

\(=\)

\(\displaystyle \|\boldsymbol{r}\|_2^2+\|\boldsymbol{a}-\boldsymbol{z}\|_2^2\)

(Orthonormal columns)

The first term is fixed. The second is nonnegative and becomes zero only at \(\boldsymbol{z}=\boldsymbol{a}=W^T\boldsymbol{x}\).

Therefore \(\widehat{\boldsymbol{x}}=WW^T\boldsymbol{x}\) is the unique closest reconstruction. Write \(P=WW^T\).

No calculus is needed. The objective is a constant plus a squared distance in coordinate space. Its minimum is attained exactly at z=a. Substituting r=(I-WW^T)x gives the full decomposition from the discussion. For fixed W, these are the unique optimal coordinates for every input. A batch encoder matrix need not be unique on directions absent from the observed dataset, but E=W transpose realizes the pointwise optimum on every possible input.

One orthonormal basis supplies both encoder and decoder

We chose the decoder \(D=W\) without loss of generality; minimizing error gives \(E=W^T\).

For the batch of centered observations:

\[ Z=XW,\qquad \widehat X=ZW^T=XWW^T. \]

Role	Choice	Property
Encoder	\(E=W^T\in\mathbb{R}^{k\times d}\)	Orthonormal rows: \(EE^T=I_k\).
Decoder	\(D=W\in\mathbb{R}^{d\times k}\)	Orthonormal columns: \(D^TD=I_k\).

Together, these choices attain the best reconstruction error allowed by general \(E,D\).

What remains to choose is the subspace: which \(k\) directions should \(W\) contain?

E and D are rectangular when k<d; neither is a square orthogonal matrix. The properties E E transpose=I and D transpose D=I alone do not ensure the encoder/decoder are matched: here D=E transpose is essential. We justify their shared basis from the closest reconstruction, rather than impose unrelated orthogonality constraints. DE=WW transpose is a projection, not I_d. This loses no optimality for the stated squared-error reconstruction problem with 1<=k<=d; it does not claim every optimal parameterization must have this form.

Four steps lead to the PCA formulation

For a fixed orthonormal decoder, its transpose gives the best encoder. Which subspace should we choose?

1. Define what a good reconstruction means — Established

Choose a representation size and measure reconstruction error.

2. Choose a convenient decoder basis — Established

An orthonormal basis can represent the same reconstruction subspace.

3. Find the best encoder for that decoder — Established

The closest point in the subspace determines the coordinates.

4. Formulate the subspace choice — Next

Combine these results into the PCA optimization problem.

Build the objective from requirements and choices

We want \(k\) linear features that reconstruct centered measurements well. Which parts are requirements, and which are convenient parameter choices?

Explain the roles of \(k\), squared error, and \(W^TW=I_k\). Would low reconstruction error alone establish biological usefulness?

\(k\) limits representation size; squared error defines “well.” An orthonormal basis is sufficient for an optimum, but biological usefulness needs more evidence.

Pairs, 4 minutes. Students should distinguish a chosen proxy objective from a claim about stakeholder value. Strong answers explain why orthogonality is justified by closest-point projection rather than a new biological assumption.

PCA chooses the best linear reconstruction subspace

For centered data \(X\) and a chosen dimension \(1\le k<d\), PCA solves:

\[ \min_{W\in\mathbb{R}^{d\times k}}\|X-XWW^T\|_F^2 \quad\text{subject to}\quad W^TW=I_k. \]

• Objective: choose the subspace that minimizes total squared reconstruction error.
• Constraint: \(W^TW=I_k\) means the columns of \(W\) are orthonormal.
• One matrix to choose: we may use decoder \(D=W\) without loss of generality; its optimal encoder is \(E=W^T\).

\[ \underbrace{W^TW=I_k}_{\text{identity in coordinate space}}, \qquad \underbrace{WW^T\ne I_d}_{\text{projection in the original space, since }k<d}. \]

The decoder basis is a standardized parameterization, not a unique basis. The closest-point proof gives the encoder as its transpose, so E and D need not be optimized independently. X has centered observations as rows, hence X W W^T is their batch reconstruction. W is rectangular d by k; it is not a square orthogonal matrix. W^T W=I_k preserves coordinates through decode then encode, while W W^T is a rank-k orthogonal projector in R^d. For k<d it cannot be I_d. If k=d, both products are identities and reconstruction is exact. We have defined the optimization problem, but have not yet solved it. The next slides use the SVD of the data matrix to identify an optimal subspace.

Three steps connect the PCA problem to its solution

We have formulated PCA using one orthonormal basis \(W\). Now we will solve that problem.

1. Rewrite the objective — Next

Least reconstruction error means most retained squared length.

2. Find the optimal directions

Use the SVD to prove that the leading right singular vectors are optimal.

3. Interpret the solution

Read reconstruction error and coordinate variance from the singular values.

Minimizing reconstruction error means keeping the most squared length

Let \(\widehat{\boldsymbol{x}}_i=WW^T\boldsymbol{x}_i\). The reconstruction and residual are perpendicular.

\(\displaystyle \|X\|_F^2\)

\(=\)

\(\displaystyle \sum_i\|\boldsymbol{x}_i\|_2^2\)

(Sum squared row lengths)

\(\,\)

\(=\)

\(\displaystyle \sum_i\bigl(\|\widehat{\boldsymbol{x}}_i\|_2^2+\|\boldsymbol{x}_i-\widehat{\boldsymbol{x}}_i\|_2^2\bigr)\)

(Pythagoras for each observation)

\(\,\)

\(=\)

\(\displaystyle \sum_i\bigl(\|W^T\boldsymbol{x}_i\|_2^2+\|\boldsymbol{x}_i-\widehat{\boldsymbol{x}}_i\|_2^2\bigr)\)

(\(W^TW=I_k\): decoding preserves length)

\(\,\)

\(=\)

\(\displaystyle \|XW\|_F^2+\|X-XWW^T\|_F^2\)

(Collect the rows into matrices)

\[ \underbrace{\|X-XWW^T\|_F^2}_{\text{error to minimize}} =\underbrace{\|X\|_F^2}_{\text{fixed total}} -\underbrace{\|XW\|_F^2}_{\text{retained amount to maximize}}. \]

Only \(W\) changes. The same centered data \(X\) give the same total squared length for every choice.

The decomposition comes from perpendicularity, not from treating the data as fixed. For each observation x_i=hat(x_i)+(x_i-hat(x_i)), the two terms are perpendicular. For z_i=W^T x_i, norm(Wz_i)^2=z_iT W^T Wz_i=norm(z_i)^2. The rows of XW are (W^T x_i)^T, so their squared norms sum to norm(XW)_F^2. Only after proving the identity do we use that X is fixed in this optimization: we select the basis W, while the observations and their centering stay unchanged. Hence norm(X)_F^{2=sum_{i,j}X_ij}2 has the same value for every candidate W. For example, if the total is 100, retaining 80 leaves error 20; retaining 90 leaves error 10. Maximizing the subtracted quantity minimizes error over the same feasible set W^T W=I_k. There is no extra assumption on the data here.