QD-RL: Efficient Mixing of Quality and Diversity in Reinforcement Learning - ANR - Agence nationale de la recherche Accéder directement au contenu
Pré-Publication, Document De Travail Année : 2020

QD-RL: Efficient Mixing of Quality and Diversity in Reinforcement Learning

Geoffrey Cideron
  • Fonction : Auteur
Thomas Pierrot
  • Fonction : Auteur
Nicolas Perrin
Karim Beguir
  • Fonction : Auteur
Olivier Sigaud

Résumé

We propose a novel reinforcement learning algorithm,QD-RL, that incorporates the strengths of off-policy RL algorithms into Quality Diversity (QD) approaches. Quality-Diversity methods contribute structural biases by decoupling the search for diversity from the search for high return, resulting in efficient management of the exploration-exploitation trade-off. However, these approaches generally suffer from sample inefficiency as they call upon evolutionary techniques. QD-RL removes this limitation by relying on off-policy RL algorithms. More precisely, we train a population of off-policy deep RL agents to simultaneously maximize diversity inside the population and the return of the agents. QD-RL selects agents from the diversity-return Pareto Front, resulting in stable and efficient population updates. Our experiments on the Ant-Maze environment show that QD-RL can solve challenging exploration and control problems with deceptive rewards while being more than 15 times more sample efficient than its evolutionary counterparts.

Dates et versions

hal-03083159 , version 1 (18-12-2020)

Identifiants

Citer

Geoffrey Cideron, Thomas Pierrot, Nicolas Perrin, Karim Beguir, Olivier Sigaud. QD-RL: Efficient Mixing of Quality and Diversity in Reinforcement Learning. 2020. ⟨hal-03083159⟩

Collections

ANR
197 Consultations
0 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More