Why the Shape of the Objective Matters

David I. Inouye

Which loss would be easiest to optimize?

Which shape gives us the clearest route to a global minimum?

A. The jumpy loss.

B. The smooth loss with several minima.

C. The smooth bowl.

C. The smooth bowl gives useful downhill directions without competing valleys.

Two minutes, including the reveal. These are invented one-parameter objectives, not measured model losses. A has three low plateaus: derivatives are zero on flat intervals and undefined at jumps, so a local gradient is not useful. The point is uninformative derivatives, not extreme complexity. B is a quadratic plus sinusoid with several local minima: smoothness alone does not rule out worse local minima. C is a quadratic. The picture motivates a structural guarantee; it is not a universal runtime comparison.

A useful objective lets us reason about the solution

• Jumpy / flat: the derivative is zero on plateaus and undefined at jumps.
• Smooth, several minima: downhill progress can end at a worse local minimum.
• Smooth bowl: useful downhill directions, without competing valleys.

This suggests a “nice” objective shape. We will call it convex and define it precisely next.

About one minute. An arbitrary objective can hide a better value far from the current point. Without additional structure, local information cannot certify global optimality. Convexity supplies that structure. Do not equate smoothness with convexity or promise that every convex problem is easy to compute. Convex functions can be nonsmooth and can fail to attain a minimum. source:boyd-vandenberghe-convex-optimization.

Objective shape answers three linked questions

1. Define: what does convexity mean geometrically?
2. Check: are our OLS and logistic objectives convex?
3. Interpret: what does that guarantee—and what else do we need?

30 seconds. Connect the next two proofs: the secant inequality defines convexity, and the curvature test gives a practical way to verify it. Then apply the test to our objectives and separate global optimality from uniqueness, existence, and speed.

A “nice” objective can have one bottom—or a flat bottom

The definition of convexity will formalize this shape for one or many parameters.

source:boyd-vandenberghe-convex-optimization. Geometric views of the same definition: Left: one parameter, 2D plot. Center: two parameters and height,3D plot. Right: contour view of a flat direction. These are mathematical illustrations, not empirical loss fits.

A convex domain contains every segment between its points

A set \(\mathcal D\subseteq\mathbb R^d\) is convex if \[\forall\boldsymbol u,\boldsymbol v\in\mathcal D,\quad\forall t\in[0,1],\quad (1-t)\boldsymbol u+t\boldsymbol v\in\mathcal D.\]

Visual segment test: the entire segment must stay inside the domain for every pair.

source:boyd-vandenberghe-convex-optimization,section2.1. These pictures show allowed input coordinates, not function heights. Filled ellipse and rectangle are convex; the annulus fails because a segment crosses its hole. One failed segment disproves convexity; a few successful segments do not prove it. This input-space requirement makes all the function values in the next chord inequality well-defined. About one minute.

Definition: a convex function lies below every chord

\(J\) is convex on a convex domain if, for every \(\boldsymbol{u},\boldsymbol{v}\) in the domain and \(0\le t\le1\),

\[{\color{#27847f}{J\big(}}{\color{#b8872f}{(1-t)}}{\color{#2f6b8a}{\boldsymbol{u}}}+{\color{#b8872f}{t}}{\color{#756486}{\boldsymbol{v}}}{\color{#27847f}{\big)}} \le {\color{#b8872f}{(1-t)}}{\color{#2f6b8a}{J(\boldsymbol{u})}}+{\color{#b8872f}{t}}{\color{#756486}{J(\boldsymbol{v})}}.\]

Loss at the averaged input ≤ average of the endpoint losses.

source:boyd-vandenberghe-convex-optimization,section3.1.1. Definition, not a claim derived from one drawing. A convex domain contains every weighted average (1-t)u+tv of its points. Plot is the scalar case; bold vectors in the inequality allow multiple coefficients. Blue matches u and J(u), purple matches v and J(v), gold matches both convex combinations. At the same averaged input, the teal curve point is below the gold chord point. The two averages have the same weights but are generally different output values. Point to each labeled axis value before reading the inequality.

Convexity makes every local minimum global

Proof by contradiction. Let \(J\) be convex on a convex domain \(\mathcal D\).

Local minimality at \(\boldsymbol u\) means: for some \(r>0\), \[J(\boldsymbol x)\ge J(\boldsymbol u)\quad\text{whenever }\boldsymbol x\in\mathcal D,\ \|\boldsymbol x-\boldsymbol u\|<r.\]

Suppose \(\boldsymbol u\) is not global: some \(\boldsymbol v\in\mathcal D\) has \(J(\boldsymbol v)<J(\boldsymbol u)\).

Choose \(0<t<\min\{1,r/\|\boldsymbol v-\boldsymbol u\|\}\) and set \(\boldsymbol x_t=(1-t)\boldsymbol u+t\boldsymbol v\).

The segment stays in \(\mathcal D\), and \(\|\boldsymbol x_t-\boldsymbol u\|=t\|\boldsymbol v-\boldsymbol u\|<r\).

\[J(\boldsymbol x_t)\le(1-t)J(\boldsymbol u)+tJ(\boldsymbol v)\quad\text{(convexity)}\]

\[\phantom{J(\boldsymbol x_t)}< (1-t)J(\boldsymbol u)+tJ(\boldsymbol u)=J(\boldsymbol u)\quad\text{(strictly lower loss, }t>0\text{)}.\]

Contradiction: a lower value lies inside the local-minimum neighborhood. Thus \(\boldsymbol u\) is global.

source:boyd-vandenberghe-convex-optimization,section4.2.2. Local minimum is relative to the domain, so boundary points are included. The better value implies v differs from u and the norm denominator is positive. Any t in (0,1) gives strict improvement, but only sufficiently small t guarantees the neighborhood condition. The displayed interval is nonempty. Convexity of the domain keeps the segment feasible. Strictness comes from J(v)<J(u) and t>0, not strict convexity or a strict local minimum. No differentiability, open domain, or uniqueness is required. About three minutes.

For a twice differentiable function, convexity means nonnegative curvature

Let \(J\) be twice continuously differentiable on an open convex domain.

Parameters	Equivalent criterion for convexity
One scalar \(\theta\)	\(J''(\theta)\ge0\) everywhere
A vector \(\boldsymbol{\theta}\)	\(H(\boldsymbol{\theta})=\nabla^2J(\boldsymbol{\theta})\succeq0\) everywhere

The Hessian is the matrix of second partial derivatives: \(H_{jk}=\frac{\partial^2J}{\partial\theta_j\partial\theta_k}\).

Positive semidefinite (PSD) means \(\boldsymbol{v}^{\mathsf T}H\boldsymbol{v}\ge0\) for every direction \(\boldsymbol{v}\). Equivalently, all eigenvalues of \(H\) are \(\ge0\): no eigenvector direction curves downward; zero allows a flat direction at that point.

source:boyd-vandenberghe-convex-optimization,section3.1.4. State the equivalence without deriving it. C2 makes the Hessian symmetric, so its eigenvalues are real. PSD means no direction has negative second-order curvature. Require the criterion everywhere, not only at one candidate minimum. A zero eigenvalue indicates zero second-order curvature at that point, not necessarily a globally flat direction.

Local quadratic approximations reveal the sign of curvature

Near any point \(\theta_0\), with \(h=\theta-\theta_0\), \[J(\theta_0+h)=\underbrace{J(\theta_0)+J'(\theta_0)h+\tfrac12J''(\theta_0)h^2}_{\text{local quadratic approximation}}+o(h^2).\]

• Logistic loss: \(J''(\theta)=\sigma(\theta)(1-\sigma(\theta))>0\) at every finite score.
• Cubic: \(J''(\theta)=6\theta<0\) when \(\theta<0\); therefore it is not convex on \(\mathbb R\).

source:boyd-vandenberghe-convex-optimization,section3.1.4. Left: one positive-label logistic loss log(1+exp(-theta)). Right: theta cubed. Solid curves are the functions; short dashed segments are second-order Taylor polynomials at the marked points. They agree in value, slope, and curvature locally, not globally. The formula applies at every point for C2 functions; little-o remainder divided by h squared tends to zero. Upward bending does not mean positive slope or positive function value. The cubic bends downward on the negative half-line and upward on the positive half-line; its zero curvature at zero alone cannot certify convexity. About two minutes.

OLS and logistic log loss have nonnegative directional curvature

Append the intercept to \(\widetilde X\) and the coefficients to \(\boldsymbol{\theta}\).

Objective	Hessian \(H=\nabla^2J\)
OLS	\(\frac1n\widetilde X^{\mathsf T}\widetilde X\)
Logistic log loss	\(\frac1n\widetilde X^{\mathsf T}W\widetilde X\), with \(W_{ii}=a_i(1-a_i)\ge0\)

For logistic loss, \(\ell(z,y)=\log(1+e^z)-yz\) gives \(\ell'=\sigma(z)-y\) and \(\ell''=a(1-a)\).

\[\boldsymbol{v}^{\mathsf T}H\boldsymbol{v} =\frac1n\sum_i a_i(1-a_i)(\widetilde{\boldsymbol{x}}_i^{\mathsf T}\boldsymbol{v})^2\ge0.\]

For OLS, replace the weights by 1. Both objectives are convex in the coefficients.

source:boyd-vandenberghe-convex-optimization. The intercept-augmented observation is a column vector; its transpose is a row of Xtilde. Derive scalar logistic derivatives directly from log(1+exp(z)); sigmoid prime=a(1-a). Outer-product chain rule yields Hessian. Adding lambda/2||w||² adds a PSD diagonal block, preserving convexity.

Convexity makes a local optimum globally meaningful

• Local minimum → global minimum. Zero gradient also certifies a global minimum for a differentiable convex objective.
• Uniqueness needs more: strict convexity excludes multiple minimizers, if a minimizer exists.
• Existence needs more: a convex loss can approach its infimum without attaining it.

Next: a useful shape does not itself tell us how to compute the solution.

source:boyd-vandenberghe-convex-optimization,chapters3,4,9. First-order tangent bound J(v)>=J(u)+gradJ(u)^T(v-u); zero gradient lower-bounds every other value by J(u). Derive tangent bound by taking t->0 in secant inequality if asked. Do not cover smoothness constants or convergence-rate guarantees. Existence and uniqueness are distinct from global optimality.