Further properties of the forward–backward envelope with applications to difference-of-convex programming

Abstract

In this paper, we further study the forward–backward envelope first introduced in Patrinos and Bemporad (Proceedings of the IEEE Conference on Decision and Control, pp 2358–2363, 2013) and Stella et al. (Comput Optim Appl, doi:10.1007/s10589-017-9912-y, 2017) for problems whose objective is the sum of a proper closed convex function and a twice continuously differentiable possibly nonconvex function with Lipschitz continuous gradient. We derive sufficient conditions on the original problem for the corresponding forward–backward envelope to be a level-bounded and Kurdyka–Łojasiewicz function with an exponent of $\frac{1}{2}$; these results are important for the efficient minimization of the forward–backward envelope by classical optimization algorithms. In addition, we demonstrate how to minimize some difference-of-convex regularized least squares problems by minimizing a suitably constructed forward–backward envelope. Our preliminary numerical results on randomly generated instances of large-scale $\ell _{1-2}$ regularized least squares problems (Yin et al. in SIAM J Sci Comput 37:A536–A563, 2015) illustrate that an implementation of this approach with a limited-memory BFGS scheme usually outperforms standard first-order methods such as the nonmonotone proximal gradient method in Wright et al. (IEEE Trans Signal Process 57:2479–2493, 2009).

Appendix: closed form formula for NPG subproblems

In this appendix, we derive a closed form formula for the following problem,

$$\begin{aligned} \min _{x} \ \frac{1}{2}\Vert x - y\Vert ^2 + \mu _1 \Vert x\Vert _1 - \mu _2 \Vert x\Vert , \end{aligned}$$

(40)

where $\mu _1 \ge \mu _2 > 0$ and $y\in \mathbb {R}^n$ is given. It is easy to see that (40) covers (37) as a special case and hence we will obtain a closed form formula for these NPG subproblems. To proceed with our derivation, we first establish the following lemma.

Lemma 7.1

Let $v\in \mathbb {R}^n$ and define ${{\mathcal {I}}}:= \{i:\; v_i < 0\}$. Consider the optimization problem

$$\begin{aligned} \min _{\Vert x\Vert = 1, x\ge 0} v^Tx \end{aligned}$$

(41)

• (i) Suppose that ${{\mathcal {I}}}\ne \emptyset$. Then an optimal solution $x^*$ of (41) is given by

  $$\begin{aligned} x^*_i = {\left\{ \begin{array}{ll} -\frac{v_i}{\Vert v_{{{\mathcal {I}}}}\Vert } &{} \mathrm{if}\ i \in {{\mathcal {I}}},\\ 0 &{} \mathrm{otherwise}, \end{array}\right. } \end{aligned}$$

  (42)

  where $v_{{\mathcal {I}}}$ is the subvector of v indexed by ${{\mathcal {I}}}$.

• (ii) Suppose that ${{\mathcal {I}}} = \emptyset$ and take an $i_*\in \{i:\; v_i = \min _k v_k\}$. Then an optimal solution $x^*$ of (41) is given by

  $$\begin{aligned} x^*_i = {\left\{ \begin{array}{ll} 1 &{} \mathrm{if}\ i = i_*,\\ 0 &{} \mathrm{otherwise}. \end{array}\right. } \end{aligned}$$

  (43)

Proof

It is clear that an optimal solution of (41) exists.

Suppose first that ${{\mathcal {I}}} \ne \emptyset$. In this case, we consider the following relaxation of (41):

$$\begin{aligned} \min _{\Vert x\Vert \le 1, x\ge 0} v^Tx \end{aligned}$$

(44)

Let $x^*$ be an optimal solution of (44). Then it is easy to see that for any i that corresponds to $v_i > 0$, we must have $x^*_i = 0$; because otherwise, one can zero out these $x_i^*$ to obtain a feasible solution with a strictly smaller objective value. Next, since ${{\mathcal {I}}}$ is nonempty, it is not hard to see that one must have $x^*_i = 0$ for all i corresponding to $v_i = 0$, because otherwise, one can further decrease the objective value by setting these entries to zero while increasing some $x_i^*$ with $i\in {{\mathcal {I}}}$. Thus, $x^*_i = 0$ for all i that correspond to $v_i \ge 0$. Finally, it must hold that $\Vert x^*\Vert = 1$ because otherwise one can further increase $x_i^*$ for $i\in {{\mathcal {I}}}$ to decrease the objective value. Thus, we conclude that $x^*$ must take the form of (42). Since this $x^*$ is optimal for (44) and is also feasible for (41), it must also be optimal for (41).

Next, suppose that ${{\mathcal {I}}}$ is empty. This means that v is a nonnegative vector. Observe that for any x feasible for (41) so that $\Vert x\Vert = 1$, we have

$$\begin{aligned} 1 = \Vert x\Vert ^2 \le (e^Tx)^2 \le n \Vert x\Vert ^2 = n, \end{aligned}$$

showing that the following optimization problem is a relaxation of (41):

$$\begin{aligned} \min _{1\le e^Tx \le \sqrt{n}, x\ge 0} v^Tx. \end{aligned}$$

(45)

Since v is nonnegative, a simple scaling argument indicates that any optimal solution $x^*$ of (45) has to satisfy $e^Tx^* = 1$. In particular, this shows that the optimal value of (45) is given by $\min _i v_i$ and the $x^*$ given by (43) is an optimal solution. Since this $x^*$ is clearly feasible for (41), it is also optimal for (41). This completes the proof. $\square$

The next proposition gives an explicit formula for a minimizer of (40).

Proposition 7.1

Let ${{\mathcal {I}}} = \{i:\; \mu _1 < |y_i|\}$.

• (i) Suppose that ${{\mathcal {I}}}$ is nonempty. Then a solution $x^*$ of (40) is given by

  $$\begin{aligned} x^*_i = {\left\{ \begin{array}{ll} \mathrm{sgn}(y_i)(\mu _2 + \Vert |y_{{\mathcal {I}}}|-\mu _1 e_{{\mathcal {I}}}\Vert )\frac{|y_i| - \mu _1}{\Vert |y_{{\mathcal {I}}}|-\mu _1 e_{{\mathcal {I}}}\Vert } &{} \mathrm{if}\ i \in {{\mathcal {I}}},\\ 0 &{} \mathrm{otherwise}, \end{array}\right. } \end{aligned}$$

  where $y_{{\mathcal {I}}}$ is the subvector of y indexed by ${{\mathcal {I}}}$, the absolute value $|y_{{\mathcal {I}}}|$ is taken componentwise, and $e_{{\mathcal {I}}}$ is the vector of all ones of dimension $|{{\mathcal {I}}}|$.

• (ii) Suppose that ${{\mathcal {I}}}$ is empty and take an $i_*\in \{i:\; \mu _1 - |y_i| = \min _k\{\mu _1 - |y_k|\}\}$. Then a solution $x^*$ of (40) is given by

  $$\begin{aligned} x^*_i = {\left\{ \begin{array}{ll} \mathrm{sgn}(y_{i_*})\max \{\mu _2 - (\mu _1-|y_{i_*}|),0\} &{} \mathrm{if}\ i = i_*,\\ 0 &{} \mathrm{otherwise}. \end{array}\right. } \end{aligned}$$

Proof

Using a substitution $x = \alpha \circ w$, where $\alpha \in \{-1,1\}^n$, $w\ge 0$ and $\circ$ denotes entrywise product, and expanding the quadratic term, we see that problem (40) can be equivalently written as

$$\begin{aligned} \min _{\alpha \in \{-1,1\}^n,w\ge 0} \frac{1}{2}\Vert w\Vert ^2 - (\alpha \circ y)^Tw + \mu _1 e^Tw - \mu _2 \Vert w\Vert . \end{aligned}$$

Applying a further substitution $w = ru$ with a number $r\ge 0$ and a nonnegative vector $\Vert u\Vert = 1$, the above problem can be further reformulated as

It is easy to check that the inner minimization is attained at $r = \max \{\mu _2 - (\mu _1e-\alpha \circ y)^Tu,0\}$. Plugging this back, the optimization problem now becomes

Since the function $t\mapsto -\frac{1}{2}(\max \{\mu _2 - t,0\})^2$ is nondecreasing, to obtain an optimal solution of the above optimization problem, one only needs to consider the problem

For this problem, since $u\ge 0$, we must have $\alpha ^* = \mathrm{sgn}(y)$ at optimality. This further reduces the above problem to

$$\begin{aligned} \min _{u\ge 0,\Vert u\Vert =1}(\mu _1e-|y|)^Tu. \end{aligned}$$

The conclusion of this proposition now follows from this observation, Lemma 7.1, the facts that $x = \alpha \circ (ru)$ with $r = \max \{\mu _2 - (\mu _1e-\alpha \circ y)^Tu,0\}$ and $\alpha ^* = \mathrm{sgn}(y)$. $\square$