Learning Compositional Neural Programs for Continuous Control - ANR - Agence nationale de la recherche Accéder directement au contenu
Pré-Publication, Document De Travail Année : 2020

Learning Compositional Neural Programs for Continuous Control

Thomas Pierrot
  • Fonction : Auteur
Nicolas Perrin
Alexandre Laterre
  • Fonction : Auteur
Olivier Sigaud
Karim Beguir
  • Fonction : Auteur

Résumé

We propose a novel solution to challenging sparse-reward, continuous control problems that require hierarchical planning at multiple levels of abstraction. Our solution, dubbed AlphaNPI-X, involves three separate stages of learning. First, we use off-policy reinforcement learning algorithms with experience replay to learn a set of atomic goal-conditioned policies, which can be easily repurposed for many tasks. Second, we learn self-models describing the effect of the atomic policies on the environment. Third, the self-models are harnessed to learn recursive compositional programs with multiple levels of abstraction. The key insight is that the self-models enable planning by imagination, obviating the need for interaction with the world when learning higher-level compositional programs. To accomplish the third stage of learning, we extend the AlphaNPI algorithm, which applies AlphaZero to learn recursive neural programmer-interpreters. We empirically show that AlphaNPI-X can effectively learn to tackle challenging sparse manipulation tasks, such as stacking multiple blocks, where powerful model-free baselines fail.

Dates et versions

hal-03083161 , version 1 (18-12-2020)

Identifiants

Citer

Thomas Pierrot, Nicolas Perrin, Feryal Behbahani, Alexandre Laterre, Olivier Sigaud, et al.. Learning Compositional Neural Programs for Continuous Control. 2020. ⟨hal-03083161⟩

Collections

ANR
35 Consultations
0 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More