TY - GEN

T1 - A new natural policy gradient by stationary distribution metric

AU - Morimura, Tetsuro

AU - Uchibe, Eiji

AU - Yoshimoto, Junichiro

AU - Doya, Kenji

PY - 2008

Y1 - 2008

N2 - The parameter space of a statistical learning machine has a Riemannian metric structure in terms of its objective function. [1] Amari proposed the concept of "natural gradient" that takes the Riemannian metric of the parameter space into account. Kakade [2] applied it to policy gradient reinforcement learning, called a natural policy gradient (NPG). Although NPGs evidently depend on the underlying Riemannian metrics, careful attention was not paid to the alternative choice of the metric in previous studies. In this paper, we propose a Riemannian metric for the joint distribution of the state-action, which is directly linked with the average reward, and derive a new NPG named "Natural State-action Gradient" (NSG). Then, we prove that NSG can be computed by fitting a certain linear model into the immediate reward function. In numerical experiments, we verify that the NSG learning can handle MDPs with a large number of states, for which the performances of the existing (N)PG methods degrade.

AB - The parameter space of a statistical learning machine has a Riemannian metric structure in terms of its objective function. [1] Amari proposed the concept of "natural gradient" that takes the Riemannian metric of the parameter space into account. Kakade [2] applied it to policy gradient reinforcement learning, called a natural policy gradient (NPG). Although NPGs evidently depend on the underlying Riemannian metrics, careful attention was not paid to the alternative choice of the metric in previous studies. In this paper, we propose a Riemannian metric for the joint distribution of the state-action, which is directly linked with the average reward, and derive a new NPG named "Natural State-action Gradient" (NSG). Then, we prove that NSG can be computed by fitting a certain linear model into the immediate reward function. In numerical experiments, we verify that the NSG learning can handle MDPs with a large number of states, for which the performances of the existing (N)PG methods degrade.

UR - http://www.scopus.com/inward/record.url?scp=56049126020&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=56049126020&partnerID=8YFLogxK

U2 - 10.1007/978-3-540-87481-2_6

DO - 10.1007/978-3-540-87481-2_6

M3 - Conference contribution

AN - SCOPUS:56049126020

SN - 3540874801

SN - 9783540874805

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 82

EP - 97

BT - Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2008, Proceedings

T2 - European Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2008

Y2 - 15 September 2008 through 19 September 2008

ER -