| October 1, 2021
Project overview
The project investigates the strategies learned by reinforcement learning (RL) agents for portfolio management. In an automated trading system, an agent reallocates funds in response to its environment. Understanding which market characteristics trigger these decisions, and where the resulting strategies reach their limits, can help address constraints in the trading domain and guide improvements to agent performance.
The main objective is to analyse the environmental changes that lead an RL agent to act. Students will review explainability methods specific to reinforcement learning, including work on visualising agent behaviour [1] and examining unexpected behaviour through worst-case analysis [2]. Pre-trained models may be supplied so that the investigation can concentrate on interpreting existing strategies.
A second objective is to relate the agents’ decisions to indicators used by human traders. Traditional portfolio management draws on fundamental and technical analysis; this project focuses the comparison on the types of market movements and their effects, using the technical analysis functions available in TA-Lib [3].
The planned work combines a literature review, implementation and comparison of analysis techniques, and a testbench built on a supplied toolkit for data processing, metrics and backtesting. Development uses Python and machine learning libraries such as PyTorch or TensorFlow, NumPy and pandas. A simplified trading environment is planned, initially with synthetic data and subsequently with real data. The expected deliverable is a detailed report in the form of a scientific article suitable for submission to arXiv.
References:
[1] Visualizing and Understanding Atari Agents, Greydanus, 2018.
[2] Uncovering Surprising Behaviors In Reinforcement Learning Via Worst-case Analysis, Ruderman, 2019.
[3] TA-Lib, python wrapper to the ta-lib library for technical analysis.