Free-energy-based reinforcement learning in a partially observable environment

Makoto Otsuka; Junichiro Yoshimoto; Kenji Doya

Free-energy-based reinforcement learning in a partially observable environment

Makoto Otsuka, Junichiro Yoshimoto, Kenji Doya

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Free-energy-based reinforcement learning (FERL) can handle Markov decision processes (MDPs) with high-dimensional state spaces by approximating the state-action value function with the negative equilibrium free energy of a restricted Boltzmann machine (RBM). In this study, we extend the FERL framework to handle partially observable MDPs (POMDPs) by incorporating a recurrent neural network that learns a memory representation sufficient for predicting future observations and rewards. We demonstrate that the proposed method successfully solves POMDPs with high-dimensional observations without any prior knowledge of the environmental hidden states and dynamics. After learning, task structures are implicitly represented in the distributed activation patterns of hidden nodes of the RBM.

Original language	English
Title of host publication	Proceedings of the 18th European Symposium on Artificial Neural Networks - Computational Intelligence and Machine Learning, ESANN 2010
Pages	541-546
Number of pages	6
Publication status	Published - 2010
Externally published	Yes
Event	18th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, ESANN 2010 - Bruges, Belgium Duration: 28-04-2010 → 30-04-2010

Publication series

Name	Proceedings of the 18th European Symposium on Artificial Neural Networks - Computational Intelligence and Machine Learning, ESANN 2010

Conference

Conference	18th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, ESANN 2010
Country/Territory	Belgium
City	Bruges
Period	28-04-10 → 30-04-10

All Science Journal Classification (ASJC) codes

Artificial Intelligence
Information Systems

Cite this

Otsuka, M., Yoshimoto, J., & Doya, K. (2010). Free-energy-based reinforcement learning in a partially observable environment. In Proceedings of the 18th European Symposium on Artificial Neural Networks - Computational Intelligence and Machine Learning, ESANN 2010 (pp. 541-546). (Proceedings of the 18th European Symposium on Artificial Neural Networks - Computational Intelligence and Machine Learning, ESANN 2010).

Otsuka, Makoto ; Yoshimoto, Junichiro ; Doya, Kenji. / Free-energy-based reinforcement learning in a partially observable environment. Proceedings of the 18th European Symposium on Artificial Neural Networks - Computational Intelligence and Machine Learning, ESANN 2010. 2010. pp. 541-546 (Proceedings of the 18th European Symposium on Artificial Neural Networks - Computational Intelligence and Machine Learning, ESANN 2010).

@inproceedings{3a0a75019ac242ec9365b43e4aaeed62,

title = "Free-energy-based reinforcement learning in a partially observable environment",

abstract = "Free-energy-based reinforcement learning (FERL) can handle Markov decision processes (MDPs) with high-dimensional state spaces by approximating the state-action value function with the negative equilibrium free energy of a restricted Boltzmann machine (RBM). In this study, we extend the FERL framework to handle partially observable MDPs (POMDPs) by incorporating a recurrent neural network that learns a memory representation sufficient for predicting future observations and rewards. We demonstrate that the proposed method successfully solves POMDPs with high-dimensional observations without any prior knowledge of the environmental hidden states and dynamics. After learning, task structures are implicitly represented in the distributed activation patterns of hidden nodes of the RBM.",

author = "Makoto Otsuka and Junichiro Yoshimoto and Kenji Doya",

year = "2010",

language = "English",

isbn = "2930307102",

series = "Proceedings of the 18th European Symposium on Artificial Neural Networks - Computational Intelligence and Machine Learning, ESANN 2010",

pages = "541--546",

booktitle = "Proceedings of the 18th European Symposium on Artificial Neural Networks - Computational Intelligence and Machine Learning, ESANN 2010",

note = "18th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, ESANN 2010 ; Conference date: 28-04-2010 Through 30-04-2010",

}

Otsuka, M, Yoshimoto, J & Doya, K 2010, Free-energy-based reinforcement learning in a partially observable environment. in Proceedings of the 18th European Symposium on Artificial Neural Networks - Computational Intelligence and Machine Learning, ESANN 2010. Proceedings of the 18th European Symposium on Artificial Neural Networks - Computational Intelligence and Machine Learning, ESANN 2010, pp. 541-546, 18th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, ESANN 2010, Bruges, Belgium, 28-04-10.

Free-energy-based reinforcement learning in a partially observable environment. / Otsuka, Makoto; Yoshimoto, Junichiro; Doya, Kenji.
Proceedings of the 18th European Symposium on Artificial Neural Networks - Computational Intelligence and Machine Learning, ESANN 2010. 2010. p. 541-546 (Proceedings of the 18th European Symposium on Artificial Neural Networks - Computational Intelligence and Machine Learning, ESANN 2010).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Free-energy-based reinforcement learning in a partially observable environment

AU - Otsuka, Makoto

AU - Yoshimoto, Junichiro

AU - Doya, Kenji

PY - 2010

Y1 - 2010

N2 - Free-energy-based reinforcement learning (FERL) can handle Markov decision processes (MDPs) with high-dimensional state spaces by approximating the state-action value function with the negative equilibrium free energy of a restricted Boltzmann machine (RBM). In this study, we extend the FERL framework to handle partially observable MDPs (POMDPs) by incorporating a recurrent neural network that learns a memory representation sufficient for predicting future observations and rewards. We demonstrate that the proposed method successfully solves POMDPs with high-dimensional observations without any prior knowledge of the environmental hidden states and dynamics. After learning, task structures are implicitly represented in the distributed activation patterns of hidden nodes of the RBM.

AB - Free-energy-based reinforcement learning (FERL) can handle Markov decision processes (MDPs) with high-dimensional state spaces by approximating the state-action value function with the negative equilibrium free energy of a restricted Boltzmann machine (RBM). In this study, we extend the FERL framework to handle partially observable MDPs (POMDPs) by incorporating a recurrent neural network that learns a memory representation sufficient for predicting future observations and rewards. We demonstrate that the proposed method successfully solves POMDPs with high-dimensional observations without any prior knowledge of the environmental hidden states and dynamics. After learning, task structures are implicitly represented in the distributed activation patterns of hidden nodes of the RBM.

UR - http://www.scopus.com/inward/record.url?scp=84887013392&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84887013392&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:84887013392

SN - 2930307102

SN - 9782930307107

T3 - Proceedings of the 18th European Symposium on Artificial Neural Networks - Computational Intelligence and Machine Learning, ESANN 2010

SP - 541

EP - 546

BT - Proceedings of the 18th European Symposium on Artificial Neural Networks - Computational Intelligence and Machine Learning, ESANN 2010

T2 - 18th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, ESANN 2010

Y2 - 28 April 2010 through 30 April 2010

ER -

Otsuka M, Yoshimoto J, Doya K. Free-energy-based reinforcement learning in a partially observable environment. In Proceedings of the 18th European Symposium on Artificial Neural Networks - Computational Intelligence and Machine Learning, ESANN 2010. 2010. p. 541-546. (Proceedings of the 18th European Symposium on Artificial Neural Networks - Computational Intelligence and Machine Learning, ESANN 2010).

Free-energy-based reinforcement learning in a partially observable environment

Abstract

Publication series

Conference

All Science Journal Classification (ASJC) codes

Other files and links

Fingerprint

Cite this