Transferred $Q$-Learning

We consider $Q$-learning with knowledge transfer, using samples from a target reinforcement learning (RL) task as well as source samples from different but related RL tasks. We propose transfer learning algorithms for both batch and online $Q$-learning with offline source studies. The proposed transferred $Q$-learning algorithm contains a novel {\em re-targeting} step that enables {\em vertical information-cascading} along multiple steps in an RL task, besides the usual horizontal information-gathering as transfer learning (TL) for supervised learning. We establish the first theoretical justifications of TL in RL tasks by showing a faster rate of convergence of the $Q$-function estimation in the offline RL transfer, and a lower regret bound in the offline-to-online RL transfer under certain similarity assumptions. Empirical evidences from both synthetic and real datasets are presented to backup the proposed algorithm and our theoretical results.

You can access the most recent version of our paper at the following link: https://arxiv.org/abs/2202.04709. We are currently working on several related projects and would be happy to discuss them with you. If you are interested, please don’t hesitate to reach out to us.