両方とも前のリビジョン 前のリビジョン 次のリビジョン | 前のリビジョン 次のリビジョン両方とも次のリビジョン |
en:intro:researches:optimization [2019/09/05 11:10] – [Decentralized Optimization Algorithms] Hideaki IIDUKA | en:intro:researches:optimization [2020/02/21 11:30] – [Decentralized Optimization Algorithms] Hideaki IIDUKA |
---|
In particular, it is assumed that $C^{(i)}$ can be expressed as the **fixed point set** of a **nonexpansive mapping**. This implies the metric projection onto $C^{(i)}$ cannot be efficiently computed (e.g., $C^{(i)}$ is the intersection of many closed convex sets or the set of minimizers of a convex function).((The metric projection onto $C^{(i)}$, denoted by $P_{C^{(i)}}$, is defined for all $x\in H$ by $P_{C^{(i)}}(x) \in C^{(i)}$ and $\|x - P_{C^{(i)}}(x)\| = \inf_{y\in C^{(i)}} \|x-y\|$.)) Please see the [[en:intro:researches:fixedpoint|Fixed Point Algorithms]] page for the details of fixed point sets.\\ | In particular, it is assumed that $C^{(i)}$ can be expressed as the **fixed point set** of a **nonexpansive mapping**. This implies the metric projection onto $C^{(i)}$ cannot be efficiently computed (e.g., $C^{(i)}$ is the intersection of many closed convex sets or the set of minimizers of a convex function).((The metric projection onto $C^{(i)}$, denoted by $P_{C^{(i)}}$, is defined for all $x\in H$ by $P_{C^{(i)}}(x) \in C^{(i)}$ and $\|x - P_{C^{(i)}}(x)\| = \inf_{y\in C^{(i)}} \|x-y\|$.)) Please see the [[en:intro:researches:fixedpoint|Fixed Point Algorithms]] page for the details of fixed point sets.\\ |
| |
Here, we divide the problem into the three categories. | Here, we divide the problem into the four categories. |
- **Smooth Convex Optimization Problem**:\\ It assumes that $f^{(i)}$ $(i\in \mathcal{I})$ is smooth and convex. The problem includes [[http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=1206687&url=http%3A%2F%2Fieeexplore.ieee.org%2Fiel5%2F78%2F27152%2F01206687.pdf%3Farnumber%3D1206687|signal recovery problem]], [[http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=4291862&url=http%3A%2F%2Fieeexplore.ieee.org%2Fiel5%2F78%2F4291841%2F04291862.pdf%3Farnumber%3D4291862|beamforming problem]], [[http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=4604754&url=http%3A%2F%2Fieeexplore.ieee.org%2Fiel5%2F49%2F4604726%2F04604754.pdf%3Farnumber%3D4604754|storage allocation problem]], [[http://epubs.siam.org/doi/abs/10.1137/110850542|optimal control problem]], and [[http://epubs.siam.org/doi/abs/10.1137/120866877|bandwidth allocation problem]]. | - **Smooth Convex Optimization Problem**:\\ It assumes that $f^{(i)}$ $(i\in \mathcal{I})$ is smooth and convex. The problem includes [[http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=1206687&url=http%3A%2F%2Fieeexplore.ieee.org%2Fiel5%2F78%2F27152%2F01206687.pdf%3Farnumber%3D1206687|signal recovery problem]], [[http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=4291862&url=http%3A%2F%2Fieeexplore.ieee.org%2Fiel5%2F78%2F4291841%2F04291862.pdf%3Farnumber%3D4291862|beamforming problem]], [[http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=4604754&url=http%3A%2F%2Fieeexplore.ieee.org%2Fiel5%2F49%2F4604726%2F04604754.pdf%3Farnumber%3D4604754|storage allocation problem]], [[http://epubs.siam.org/doi/abs/10.1137/110850542|optimal control problem]], and [[http://epubs.siam.org/doi/abs/10.1137/120866877|bandwidth allocation problem]]. |
- **Nonsmooth Convex Optimization Problem**:\\ It assumes that $f^{(i)}$ $(i\in \mathcal{I})$ is nonsmooth and convex. The problem includes [[http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=4407760&filter%3DAND%28p_IS_Number%3A4407756%29|signal recovery problem]],[[http://www.sciencedirect.com/science/article/pii/S0925231214000964|ensemble learning]], [[http://link.springer.com/chapter/10.1007/978-1-4419-9569-8_17|minimal antenna-subset selection problem]], and [[https://ieeexplore.ieee.org/document/8584116|bandwidth allocation problem]]. | - **Nonsmooth Convex Optimization Problem**:\\ It assumes that $f^{(i)}$ $(i\in \mathcal{I})$ is nonsmooth and convex. The problem includes [[http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=4407760&filter%3DAND%28p_IS_Number%3A4407756%29|signal recovery problem]],[[https://ieeexplore.ieee.org/document/8744480|ensemble learning]], [[http://link.springer.com/chapter/10.1007/978-1-4419-9569-8_17|minimal antenna-subset selection problem]], and [[https://ieeexplore.ieee.org/document/8584116|bandwidth allocation problem]]. |
- **Smooth Nonconvex Optimization Problem**\\ It assumes that $f^{(i)}$ $(i\in \mathcal{I})$ is smooth and nonconvex. The problem has practical problems such as [[http://link.springer.com/article/10.1007%2Fs10107-010-0427-x#|power control]] and [[http://epubs.siam.org/doi/abs/10.1137/110849456|bandwidth allocation]]. | - **Smooth Nonconvex Optimization Problem**\\ It assumes that $f^{(i)}$ $(i\in \mathcal{I})$ is smooth and nonconvex. The problem has practical problems such as [[http://link.springer.com/article/10.1007%2Fs10107-010-0427-x#|power control]] and [[http://epubs.siam.org/doi/abs/10.1137/110849456|bandwidth allocation]]. |
| - **Nonsmooth Nonconvex Optimization Problem**\\ It assumes that $f^{(i)}$ $(i\in \mathcal{I})$ is nonsmooth and nonconvex. The problem has practical problems such as [[https://www.sciencedirect.com/science/article/abs/pii/S037722171400424X|fractional programming]]. |
| |
We focus on the following algorithms for solving the above problems. | We focus on the following algorithms for solving the above problems. |
In 2001, [[http://www.sp.ss.titech.ac.jp/index.php?Isao%20Yamada|Yamada]] proposed [[http://www.sciencedirect.com/science/article/pii/S1570579X01800288|the hybrid steepest descent method]] for //smooth convex optimization over the fixed point sets of nonexpansive mappings//. We have presented algorithms based on the hybrid steepest descent method and their convergence analyses. | In 2001, [[http://www.sp.ss.titech.ac.jp/index.php?Isao%20Yamada|Yamada]] proposed [[http://www.sciencedirect.com/science/article/pii/S1570579X01800288|the hybrid steepest descent method]] for //smooth convex optimization over the fixed point sets of nonexpansive mappings//. We have presented algorithms based on the hybrid steepest descent method and their convergence analyses. |
The following are our previously reported results: | The following are our previously reported results: |
* [[:en:iiduka:|H. Iiduka]]: [[http://link.springer.com/article/10.1007/s10107-013-0741-1|Acceleration Method for Convex Optimization over the Fixed Point Set of a Nonexpansive Mapping]], //Mathematical Programming, Vol. 149, No. 1, pp. 131-165, 2015.// | * [[:en:iiduka:|H. Iiduka]]: [[http://link.springer.com/article/10.1007/s10107-013-0741-1|Acceleration Method for Convex Optimization over the Fixed Point Set of a Nonexpansive Mapping]], Mathematical Programming, Vol. 149, No. 1, pp. 131-165, 2015. |
* S. Iemoto, K. Hishinuma, and [[:iiduka:|H. Iiduka]]: [[http://www.fixedpointtheoryandapplications.com/content/2014/1/51|Approximate Solutions to Variational Inequality over the Fixed Point Set of a Strongly Nonexpansive Mapping]], //Fixed Point Theory and Applications, 51, Vol. 2014, No. 1, 2014.// | * S. Iemoto, K. Hishinuma, and [[:iiduka:|H. Iiduka]]: [[http://www.fixedpointtheoryandapplications.com/content/2014/1/51|Approximate Solutions to Variational Inequality over the Fixed Point Set of a Strongly Nonexpansive Mapping]], Fixed Point Theory and Applications, 51, Vol. 2014, No. 1, 2014. |
* [[:en:iiduka:|H. Iiduka]] and I. Yamada: [[http://epubs.siam.org/doi/abs/10.1137/110850542|Computational Method for Solving a Stochastic Linear-Quadratic Control Problem Given an Unsolvable Stochastic Algebraic Riccati Equation]], //SIAM Journal on Control and Optimization, Vol. 50, No. 4, pp. 2173-2192, 2012.// | * [[:en:iiduka:|H. Iiduka]] and I. Yamada: [[http://epubs.siam.org/doi/abs/10.1137/110850542|Computational Method for Solving a Stochastic Linear-Quadratic Control Problem Given an Unsolvable Stochastic Algebraic Riccati Equation]], SIAM Journal on Control and Optimization, Vol. 50, No. 4, pp. 2173-2192, 2012. |
* [[:en:iiduka:|H. Iiduka]]: [[http://www.sciencedirect.com/science/article/pii/S0377042711005358|Fixed Point Optimization Algorithm and Its Application to Network Bandwidth Allocation]], //Journal of Computational and Applied Mathematics, Vol. 236, No. 7, pp. 1733-1742, 2012.// | * [[:en:iiduka:|H. Iiduka]]: [[http://www.sciencedirect.com/science/article/pii/S0377042711005358|Fixed Point Optimization Algorithm and Its Application to Network Bandwidth Allocation]], Journal of Computational and Applied Mathematics, Vol. 236, No. 7, pp. 1733-1742, 2012. |
* [[:en:iiduka:|H. Iiduka]]: [[http://link.springer.com/article/10.1007%2Fs10957-010-9769-z#|Iterative Algorithm for Solving Triple-hierarchical Constrained Optimization Problem]], //Journal of Optimization Theory and Applications, Vol. 148, No. 3, pp. 580-592, 2011.// | * [[:en:iiduka:|H. Iiduka]]: [[http://link.springer.com/article/10.1007%2Fs10957-010-9769-z#|Iterative Algorithm for Solving Triple-hierarchical Constrained Optimization Problem]], Journal of Optimization Theory and Applications, Vol. 148, No. 3, pp. 580-592, 2011. |
* [[:en:iiduka:|H. Iiduka]]: [[http://www.sciencedirect.com/science/article/pii/S0096300311000099|Three-term Conjugate Gradient Method for the Convex Optimization Problem over the Fixed Point Set of a Nonexpansive Mapping]], //Applied Mathematics and Computation, Vol. 217, No. 13, pp. 6315-6327, 2011.// | * [[:en:iiduka:|H. Iiduka]]: [[http://www.sciencedirect.com/science/article/pii/S0096300311000099|Three-term Conjugate Gradient Method for the Convex Optimization Problem over the Fixed Point Set of a Nonexpansive Mapping]], Applied Mathematics and Computation, Vol. 217, No. 13, pp. 6315-6327, 2011. |
* [[:en:iiduka:|H. Iiduka]] and [[http://www.sp.ss.titech.ac.jp/index.php?Isao%20Yamada|I. Yamada]]: [[http://epubs.siam.org/action/showAbstract?page=1881&volume=19&issue=4&journalCode=sjope8|A Use of Conjugate Gradient Direction for the Convex Optimization Problem over the Fixed Point Set of a Nonexpansive Mapping]], //SIAM Journal on Optimization, Vol. 19, No. 4, pp. 1881-1893, 2009.// | * [[:en:iiduka:|H. Iiduka]] and [[http://www.sp.ss.titech.ac.jp/index.php?Isao%20Yamada|I. Yamada]]: [[http://epubs.siam.org/action/showAbstract?page=1881&volume=19&issue=4&journalCode=sjope8|A Use of Conjugate Gradient Direction for the Convex Optimization Problem over the Fixed Point Set of a Nonexpansive Mapping]], SIAM Journal on Optimization, Vol. 19, No. 4, pp. 1881-1893, 2009. |
| |
==== Decentralized Optimization Algorithms ==== | ==== Decentralized Optimization Algorithms ==== |
[[http://epubs.siam.org/doi/abs/10.1137/S1052623499362111|The incremental subgradient method]] and [[http://iopscience.iop.org/0266-5611/24/6/065014|the broadcast optimization method]] are useful for decentralized convex optimization. However, they can be applied to only the case where $C^{(i)} = C$ $(i\in \mathcal{I})$ and $C$ is simple in the sense that the projection onto $C$ can be easily computed (e.g., $C$ is a half-space). We have proposed decentralized optimization algorithms for solving convex optimization problems with more complicated $C^{(i)}$ (e.g., $C^{(i)}$ is the intersection of simple, closed convex sets), which include [[http://epubs.siam.org/doi/abs/10.1137/120866877|bandwidth allocation problem]] and [[http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=4604754&url=http%3A%2F%2Fieeexplore.ieee.org%2Fiel5%2F49%2F4604726%2F04604754.pdf%3Farnumber%3D4604754|storage allocation problem]]. | [[http://epubs.siam.org/doi/abs/10.1137/S1052623499362111|The incremental subgradient method]] and [[http://iopscience.iop.org/0266-5611/24/6/065014|the broadcast optimization method]] are useful for decentralized convex optimization. However, they can be applied to only the case where $C^{(i)} = C$ $(i\in \mathcal{I})$ and $C$ is simple in the sense that the projection onto $C$ can be easily computed (e.g., $C$ is a half-space). We have proposed decentralized optimization algorithms for solving convex optimization problems with more complicated $C^{(i)}$ (e.g., $C^{(i)}$ is the intersection of simple, closed convex sets), which include [[http://epubs.siam.org/doi/abs/10.1137/120866877|bandwidth allocation problem]], [[http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=4604754&url=http%3A%2F%2Fieeexplore.ieee.org%2Fiel5%2F49%2F4604726%2F04604754.pdf%3Farnumber%3D4604754|storage allocation problem]], and [[https://ieeexplore.ieee.org/document/8744480|ensemble learning]]. |
* [[:en:iiduka:|H. Iiduka]]: [[https://ieeexplore.ieee.org/document/8744480|Stochastic Fixed Point Optimization Algorithm for Classifier Ensemble]], IEEE Transactions on Cybernetics (accepted) | * [[:en:iiduka:|H. Iiduka]]: [[https://ieeexplore.ieee.org/document/8744480|Stochastic Fixed Point Optimization Algorithm for Classifier Ensemble]], IEEE Transactions on Cybernetics (accepted) |
* [[:en:iiduka:|H. Iiduka]]: [[https://link.springer.com/article/10.1007/s11081-019-09440-7|Decentralized Hierarchical Constrained Convex Optimization]], Optimization and Engineering (accepted) | * [[:en:iiduka:|H. Iiduka]]: [[https://link.springer.com/article/10.1007/s11081-019-09440-7|Decentralized Hierarchical Constrained Convex Optimization]], Optimization and Engineering, Vol. 21, No. 1, pp. 181–213, 2020. |
* [[:en:iiduka:|H. Iiduka]]: [[https://www.jstage.jst.go.jp/article/jorsj/58/4/58_330/_pdf|Parallel Optimization Algorithm for Smooth Convex Optimization over Fixed Point Sets of Quasi-nonexpansive Mappings]], Journal of the Operations Research of Japan, //Vol. 58, No. 4, pp. 330-352, 2015//. | * [[:en:iiduka:|H. Iiduka]]: [[https://www.jstage.jst.go.jp/article/jorsj/58/4/58_330/_pdf|Parallel Optimization Algorithm for Smooth Convex Optimization over Fixed Point Sets of Quasi-nonexpansive Mappings]], Journal of the Operations Research of Japan, Vol. 58, No. 4, pp. 330-352, 2015. |
* [[:en:iiduka:|H. Iiduka]]: [[http://www.ybook.co.jp/online-p/JNCA/Open/16/jncav16n11p2159-oa/FLASH/index.html|Distributed Convex Optimization Algorithms and Their Application to Distributed Control in Peer-to-Peer Data Storage System]], Journal of Nonlinear and Convex Analysis: : Special Issue-Dedicated to Wataru Takahashi on the occasion of his 70th birth day, //Vol. 16, No. 11, pp. 2159-2179, 2015//. | * [[:en:iiduka:|H. Iiduka]]: [[http://www.ybook.co.jp/online-p/JNCA/Open/16/jncav16n11p2159-oa/FLASH/index.html|Distributed Convex Optimization Algorithms and Their Application to Distributed Control in Peer-to-Peer Data Storage System]], Journal of Nonlinear and Convex Analysis: : Special Issue-Dedicated to Wataru Takahashi on the occasion of his 70th birth day, Vol. 16, No. 11, pp. 2159-2179, 2015. |
* [[:en:iiduka:|H. Iiduka]] and K. Hishinuma: [[http://epubs.siam.org/doi/abs/10.1137/130939560|Acceleration Method Combining Broadcast and Incremental Distributed Optimization Algorithms]], //SIAM Journal on Optimization, Vol. 24, No. 4, pp. 1840-1863, 2014.// | * [[:en:iiduka:|H. Iiduka]] and K. Hishinuma: [[http://epubs.siam.org/doi/abs/10.1137/130939560|Acceleration Method Combining Broadcast and Incremental Distributed Optimization Algorithms]], SIAM Journal on Optimization, Vol. 24, No. 4, pp. 1840-1863, 2014. |
* [[:en:iiduka:|H. Iiduka]]: [[http://epubs.siam.org/doi/abs/10.1137/120866877|Fixed Point Optimization Algorithms for Distributed Optimization in Networked Systems]], //SIAM Journal on Optimization, Vol. 23, No. 1, pp. 1-26, 2013.// | * [[:en:iiduka:|H. Iiduka]]: [[http://epubs.siam.org/doi/abs/10.1137/120866877|Fixed Point Optimization Algorithms for Distributed Optimization in Networked Systems]], SIAM Journal on Optimization, Vol. 23, No. 1, pp. 1-26, 2013. |
| |
===== Optimization Algorithms for Nonsmooth Convex Optimization ===== | ===== Optimization Algorithms for Nonsmooth Convex Optimization ===== |
The following are the results of the algorithms based on the above methods. | The following are the results of the algorithms based on the above methods. |
==== Decentralized Optimization Algorithms ==== | ==== Decentralized Optimization Algorithms ==== |
* K. Hishinuma and [[en:iiduka:|H. Iiduka]]: Convergence Analysis of Incremental and Parallel Line Search Subgradient Methods in Hilbert Space, Journal of Nonlinear and Convex Analysis: Special Issue-Dedicated to Wataru Takahashi on the occasion of his 75th birth day (accepted) | * K. Shimizu, K. Hishinuma, [[:en:iiduka:|H. Iiduka]]: Parallel Computing Proximal Method for Nonsmooth Convex Optimization with Fixed Point Constraints of Quasi-nonexpansive Mappings, submitted |
* [[:en:iiduka:|H. Iiduka]]: [[https://ieeexplore.ieee.org/document/8584116|Distributed Optimization for Network Resource Allocation with Nonsmooth Utility Functions]], IEEE Transactions on Control of Network Systems (accepted) | * H. Oishi, Y. Kobayashi, [[:en:iiduka:|H. Iiduka]]: [[http://www.ybook.co.jp/online-p/LNA/Open/vol5/lnav5n3p477-oa/index.html|Incremental Proximal Method for Nonsmooth Convex Optimization With Fixed Point Constraints of Quasi-nonexpansive Mappings]], Linear and Nonlinear Analysis, Vol. 5, No. 3, pp. 477-493, 2019. |
| * [[:en:iiduka:|H. Iiduka]]: [[https://ieeexplore.ieee.org/document/8584116|Distributed Optimization for Network Resource Allocation with Nonsmooth Utility Functions]], IEEE Transactions on Control of Network Systems, Vol. 6, No. 4, pp. 1354-1365, 2019. |
| * K. Hishinuma and [[en:iiduka:|H. Iiduka]]: Convergence Analysis of Incremental and Parallel Line Search Subgradient Methods in Hilbert Space, Journal of Nonlinear and Convex Analysis: Special Issue-Dedicated to Wataru Takahashi on the occasion of his 75th birth day, Vol. 20, No. 9, pp.1937-1947, 2019. |
* K. Hishinuma and [[:en:iiduka:|H. Iiduka]]: [[https://www.frontiersin.org/articles/10.3389/frobt.2019.00077/full|Incremental and Parallel Machine Learning Algorithms with Automated Learning Rate Adjustments]], Frontiers in Robotics and AI: Resolution of Limitations of Deep Learning to Develop New AI Paradigms, Vol. 6, Article 77, 2019. | * K. Hishinuma and [[:en:iiduka:|H. Iiduka]]: [[https://www.frontiersin.org/articles/10.3389/frobt.2019.00077/full|Incremental and Parallel Machine Learning Algorithms with Automated Learning Rate Adjustments]], Frontiers in Robotics and AI: Resolution of Limitations of Deep Learning to Develop New AI Paradigms, Vol. 6, Article 77, 2019. |
* [[:en:iiduka:|H. Iiduka]]: [[http://www.tandfonline.com/doi/full/10.1080/10556788.2018.1425860|Two Stochastic Optimization Algorithms for Convex Optimization with Fixed Point Constraints]], Optimization Methods and Software, Vol. 34, No. 4, pp.731-757, 2019. | * [[:en:iiduka:|H. Iiduka]]: [[http://www.tandfonline.com/doi/full/10.1080/10556788.2018.1425860|Two Stochastic Optimization Algorithms for Convex Optimization with Fixed Point Constraints]], Optimization Methods and Software, Vol. 34, No. 4, pp.731-757, 2019. |
| |
==== Decentralized Optimization Algorithms ==== | ==== Decentralized Optimization Algorithms ==== |
| * K. Hishinuma and [[en:iiduka:|H. Iiduka]]: [[https://doi.org/10.1016/j.ejor.2019.09.037|Fixed Point Quasiconvex Subgradient Method]], European Journal of Operational Research, Vol. 282, No. 2, 428–437, 2020 |
* [[en:iiduka:|H. Iiduka]]: [[http://www.ybook.co.jp/|Distributed Iterative Methods for Solving Nonmonotone Variational Inequality over the Intersection of Fixed Point Sets of Nonexpansive Mappings]], Pacific Journal of Optimization, Vol. 10, No. 4, pp. 691-713, 2014. | * [[en:iiduka:|H. Iiduka]]: [[http://www.ybook.co.jp/|Distributed Iterative Methods for Solving Nonmonotone Variational Inequality over the Intersection of Fixed Point Sets of Nonexpansive Mappings]], Pacific Journal of Optimization, Vol. 10, No. 4, pp. 691-713, 2014. |
| |
| |
| |