Robust Risk-Aware Reinforcement Learning : An approach for robust optimisation of risk-aware performance criteria
We present a reinforcement learning (RL) approach for robust optimisation of risk-aware performance criteria.
To allow agents to express a wide variety of risk-reward profiles, we assess the value of a policy using rank dependent expected utility (RDEU). RDEU allows the agent to seek gains, while simultaneously protecting themselves against downside events.
To robustify optimal policies againstmodel uncertainty, we assess a policy not by its distribution, but rather, by the worst possible distribution that lies within a Wasserstein ball around it.
Thus, our problem formulation may be viewed as an actor choosing a policy (the outer problem), and the adversary then acting to worsen the performance of that strategy (the inner problem).
We develop explicit policy gradient formulae for the inner and outer problems, and show its efficacy on three prototypical financial problems:
Robust Portfolio Allocation, Optimising a Benchmark, and Statistical Arbitrage.
Many problems in financial mathematics, economics, and engineering may be cast in the form of a stochastic optimisation problem, and typically the agent’s optimal control depends on the underlying (dynamic) model assumptions.
Models, however, are approximations of the world, whether they are purely data driven (i.e., empirical) models, parametric models estimated from data, or models that are posited to reflect the given stochastic dynamics.
As models are approximations, understanding how to protect ones decisions from uncertainty inherent in a model is of paramount importance.
Thus, here we consider robust stochastic optimisation problems where the agent chooses the action that is optimal under the worst case within an uncertainty set.
