TY - GEN
T1 - A new natural policy gradient by stationary distribution metric
AU - Morimura, Tetsuro
AU - Uchibe, Eiji
AU - Yoshimoto, Junichiro
AU - Doya, Kenji
PY - 2008
Y1 - 2008
N2 - The parameter space of a statistical learning machine has a Riemannian metric structure in terms of its objective function. [1] Amari proposed the concept of "natural gradient" that takes the Riemannian metric of the parameter space into account. Kakade [2] applied it to policy gradient reinforcement learning, called a natural policy gradient (NPG). Although NPGs evidently depend on the underlying Riemannian metrics, careful attention was not paid to the alternative choice of the metric in previous studies. In this paper, we propose a Riemannian metric for the joint distribution of the state-action, which is directly linked with the average reward, and derive a new NPG named "Natural State-action Gradient" (NSG). Then, we prove that NSG can be computed by fitting a certain linear model into the immediate reward function. In numerical experiments, we verify that the NSG learning can handle MDPs with a large number of states, for which the performances of the existing (N)PG methods degrade.
AB - The parameter space of a statistical learning machine has a Riemannian metric structure in terms of its objective function. [1] Amari proposed the concept of "natural gradient" that takes the Riemannian metric of the parameter space into account. Kakade [2] applied it to policy gradient reinforcement learning, called a natural policy gradient (NPG). Although NPGs evidently depend on the underlying Riemannian metrics, careful attention was not paid to the alternative choice of the metric in previous studies. In this paper, we propose a Riemannian metric for the joint distribution of the state-action, which is directly linked with the average reward, and derive a new NPG named "Natural State-action Gradient" (NSG). Then, we prove that NSG can be computed by fitting a certain linear model into the immediate reward function. In numerical experiments, we verify that the NSG learning can handle MDPs with a large number of states, for which the performances of the existing (N)PG methods degrade.
UR - http://www.scopus.com/inward/record.url?scp=56049126020&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=56049126020&partnerID=8YFLogxK
U2 - 10.1007/978-3-540-87481-2_6
DO - 10.1007/978-3-540-87481-2_6
M3 - Conference contribution
AN - SCOPUS:56049126020
SN - 3540874801
SN - 9783540874805
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 82
EP - 97
BT - Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2008, Proceedings
T2 - European Conference on Machine Learning and Knowledge Discovery in Databases, ECML PKDD 2008
Y2 - 15 September 2008 through 19 September 2008
ER -