A Covariance Matrix Adaptation Evolution Strategy for Direct Policy Search in Reproducing Kernel Hilbert Space

Ngo Anh Vien, Viet-Hung Dang, TaeChoong Chung

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)
255 Downloads (Pure)

Abstract

The covariance matrix adaptation evolution strategy (CMA-ES) is an efficient derivative-free optimization algorithm. It optimizes a black-box objective function over a well defined parameter space. In some problems, such parameter spaces are defined using function approximation in which feature functions are manually defined. Therefore, the performance of those techniques strongly depends on the quality of chosen features. Hence, enabling CMA-ES to optimize on a more complex and general function class of the objective has long been desired. Specifically, we consider modeling the input space for black-box optimization in reproducing kernel Hilbert spaces (RKHS). This modeling leads to a functional optimization problem whose domain is a function space that enables us to optimize in a very rich function class. In addition, we propose CMA-ES-RKHS, a generalized CMA-ES framework, that performs black-box functional optimization in the RKHS. A search distribution, represented as a Gaussian process, is adapted by updating both its mean function and covariance operator. Adaptive representation of the function and covariance operator is achieved with sparsification techniques. We evaluate CMA-ES-RKHS on a simple functional optimization problem and bench-mark reinforcement learning (RL) domains. For an application in RL, we model policies for MDPs in RKHS and transform a cumulative return objective as a functional of RKHS policies, which can be optimized via CMA-ES-RKHS. This formulation results in a black-box functional policy search framework.
Original languageEnglish
Title of host publicationThe 9th Asian Conference on Machine Learning
Subtitle of host publicationACML
Pages606-621
Number of pages16
Volume77
Publication statusPublished - 17 Nov 2017
Event9th Asian Conference on Machine Learning - Seoul, Korea, Republic of
Duration: 15 Nov 201717 Nov 2017
http://www.acml-conf.org/2017/

Publication series

NameProceedings of Machine Learning Research
ISSN (Electronic)1938-7228

Conference

Conference9th Asian Conference on Machine Learning
Abbreviated titleACML 2017
CountryKorea, Republic of
CitySeoul
Period15/11/201717/11/2017
Internet address

Fingerprint Dive into the research topics of 'A Covariance Matrix Adaptation Evolution Strategy for Direct Policy Search in Reproducing Kernel Hilbert Space'. Together they form a unique fingerprint.

Cite this