In recent years, much research in reinforcement learning has focused on learning, planning, and representing knowledge at multiple levels of temporal abstraction. If reinforcement learning is to scale to solving larger, more real-world-like problems, it is essential to consider a hierarchical approach in which a complex learning task is decomposed into subtasks. It has been shown in recent and past work that a hierarchical approach substantially increases the efficiency and abilities of RL systems.
Early work in reinforcement learning showed faster learning resulted when tasks were decomposed into behaviors (Lin, 1993; Mahadevan and Connell, 1992; Singh et al 1994). However, these approaches were mostly based on modular, but not hierarchical decompositions. More recently, researchers have proposed various models, the most widely recognized being Hierarchies of Abstract Machines (HAMs) (Parr, 1998), options (Sutton, Precup and Singh, to appear), and MAXQ value function decomposition (Dietterich, 1998). A key technical breakthrough that enabled these approaches is the use of reinforcement learning over semi-Markov decision processes (Bradtke and Duff, Mahadevan et al, 1997, Parr, 1998). Although these approaches speed up learning considerably, they still assume the underlying environment is accessible (i.e. not perceptually aliased).
A major direction of research into scaling up hierarchical methods is extend them to domains where the underlying states are hidden (i.e., partially observable Markov decision processes, or POMDPs). Over the past year, there has been a surge of interest in applying hierarchical reinforcement learning (HRL) methods to such partially observable domains. Researchers are investigating techniques such as memory, state abstraction, offline decomposition, action abstraction, and many others to simplify the problem of learning near-optimal behaviors as they attack increasingly complex environments.
This workshop will be an opportunity for the researchers in this growing field to share knowledge and expertise on the topic, open lines of communication for collaboration, prevent redundant research, and possibly agree on standard problems and techniques.
The format of the workshop is designed to encourage discussion and debate. There will be approximately four invited talks by senior researchers. Ample time between talks will be provided for discussion. Additionally, there will be a a NIPS-like poster session (complete with plenary poster previews). We will also be providing a partially observable problem domain that participants can optionally use to test their techniques, which may provide for a underlying common experience for comparing and contrasting techniques. The discussion will focus not only on the various techniques and theories for using memory in reinforcement learning, but also on higher order questions, such as ``Must hierarchy and memory be combined for practical RL systems?'' and ``What forms of structured memory are useful for RL?''.
We are thus seeking attendees with the following interests:
Submissions: To participate in the workshop, please send a email message to one of the two organizers, giving your name, address, email address, and a brief description of your reasons for wanting to attend. We encourage all participants to give poster presentations since we believe that it will stimulate fruitful discussion and participation. If you wish to present a poster, please send a short (< 5 pages in ICML format) paper or abstract in postscript or PDF format to the organizers. If you have questions, please feel free to contact us.
Return to the main
page
Email the
organizers