Inverse Reinforcement Learning for portfolio allocation

| October 1, 2022

Project overview

In the context of work on automated trading systems, we rely on reinforcement learning algorithms to create portfolio allocation strategies. These algorithms require large computational resources to obtain a strategy that is not guaranteed to converge. The current approach aims at learning an action policy from interactions between an agent and the trading environment. Thus, the transition and reward functions are learned either implicitly or by a critic in parallel to the action policy.

Inverse reinforcement learning (IRL), related to imitation learning, consists in inferring an expert’s reward function from observed behaviour and using it to learn an action policy. This approach can be doubly interesting in finance where these estimates are complex from interactions, and where an adapted reward function can be difficult to create, in the case of risk mitigation.

The project aims at comparing the different existing IRL solutions, as well as quantifying their advantages over classical RL solutions (DDPG, SAC, PPO). Works [1,2,3,4] gather a number of approaches, as well as the founding theories of the field. We wish to apply this method to portfolio management (OLPS), we can already find first publications on this application, such as [5].

During this project, you will:

get to know the existing IRL techniques,
implement algorithms, with libraries (SB3, torch),
benchmark your methods.

The experiments use end-of-day prices of financial assets, including stocks, commodities and foreign exchange. Expected deliverables include a literature review, comparative implementations and a report in the form of a scientific article.