How Quantitative Finance can benefit from insights from Reinforcement Learning and Quantum Physics

Science

Preamble: two questions for Quantitative Finance

Q1: Where are non-linearities in market models?  

Q2. Why equilibrium models?

How Quantitative Finance can benefit from insights from Reinforcement Learning and Quantum Physics

This presentation will attempt to address these questions by combining  insights and methods from reinforcement learning and quantum  physics. 

The key role in mechanisms to be presented here is played by money  flows in the market, and their price impact. 

 “It is better to be roughly right than precisely wrong”   John Maynard Keynes 

 Market flows impact returns  

• Money flows into markets strongly impact the market dynamics. • Money flows are largely driven by the US employees’ contributions to their retirement plans, as well as retail investors.

Did Finance Oversleep a Century of Development in Physics?

Cumulative flows into equity+bonds+hybrid funds are about $325bn a year. This is about 1% of the S&P 500 market portfolio – is it a ‘small’ number to neglect? 

• Asset management perspective on how flows impact returns: X. Gabaix and R.S.J.  Koijen, “In Search of the Origins of Financial Fluctuations: The Inelastic Markets Hypothesis”, Swiss Finance Institute Research Paper Series (2020), https:// papers.ssrn.com/sol3/papers.cfm?abstract_id=3686935# 

• Microstructural finance’s perspective: J.P. Bouchaud, “The Inelastic Market  Hypothesis: A Microstructural Interpretation” (2021), https://papers.ssrn.com/sol3/ papers.cfm?abstract_id=3896981  

Optimal investment as a control problem: sequential (multi-step) decision-making by choosing optimal flows for each time step. This involves involves planning and forecasting the future.  

Feedback loop (price impact): Flows make impact on returns. 

Optimal investment as RL: The optimal investment problem for the aggregated investor can be formulated as a Reinforcement Learning (RL) problem (a data-driven approach to optimal control problems)  

The problem for the aggregated investor is similar to the problem of optimal investment for an individual investor (the Merton problem) in the presence of market impact.  

 A discrete-time portfolio model with flows and frictions  

• A general discrete-time portfolio model with money flows and price impact:  

x_t+Δt= (1+ r_t+ΔtΔt) ! (x_t + a_t! x_tΔt) 

r_t+Δt= r + wz_t + f(a_{t) +}¹_Δtε_t 

x_t 

a_t! x_tΔt 

• Here is a vector of asset values (in dollars), and is a vector of  ^{[t,t + Δt]}z_t 

money flows at the start of the step .  

• The second equation specifies returns as a linear function of signals, market f(a_t) 

ε _t 

impact (a negative feedback effect – a critical point!), and noise  

• The same two equations can be used to specify the dynamics for either an  individual investor’s portfolio, or a market portfolio (the focus of this talk)  • To complete the model specification, we need to define how the money flows  

a^tf(a_t)

x_t rates depend on the asset prices (or returns), as well as define the price impact function  

• The parallelism between the Merton problem for an individual investor and the market modeling  problem with outside investors was proposed in I. Halperin and I. Feldshteyn, “Market Self Learning of Signals, Impact and Optimal Trading: Invisible Hand Inference with Free Energy”  (2018) https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3174498 

 Money flows as a feedback control  

• A 1D single stock/market discrete-time model with flows and price impact:  X_t+Δt= (1+ r_t+ΔtΔt)(X_t + a_t X_tΔt) 

r_t+Δt= r + w^T z_t + f (a_{t) +}¹_Δtε_t 

a_t X_tΔt 

• In general, the money flow should be (partially) driven by the asset  a_t= a_t(X_t) a_t= a_t(ΔX_t⁾a_t(X_t)

value or alternatively by it change: or .  • We can consider a low-order Taylor expansion to specify function 

a_t= a(X_t) = a +φX_t + λ X_t² +… 

a,φ,λ 

• Here are parameters that can be slowly-varying functions of market  variables. For simplicity, we can treat them as constants  

• In addition, we need to specify the price impact function. A simplest choice  (generalizations/improvements will come later!) is a linear impact function  f (a_t) = −∝a_t 

• Using these equations together and taking the continuous-time limit, we obtain  the ‘Quantum Equilibrium-Disequilibrium’ (QED) model (see the next slide) 

 The QED model  

• Combining the previous four equations and taking the continuous time limit:  _κ − X_{t −} ^g_κX_t2 ^⎛_⎝⎜^⎞_⎠⎟ dt + X_tσ dW_t 

dX_t=κ X_tθ + w^T z_t 

W_t 

θ,g,κ 

• Here is the standard Browning motion, and parameters are  θ = r_f + a, κ = (∝ −1)φ, g = (∝ −1)λ

• This is the QED (“Quantum Equilibrium-Disequilibrium”) model for a market as an open system with money flows and frictions. (I. Halperin and M. Dixon, “Quantum Equilibrium-Disequilibrium: Asset Price Dynamics, Symmetry Breaking, and Defaults as Dissipative Instantons”, https://papers.ssrn.com/sol3/ papers.cfm?abstract_id=3223243.)  

g,κ → 0 

• If we take the limit , we formally recover the Geometric Brownian  Motion (GBM) model (the same as the lognormal model) with signals 

dX_t= θ X_tdt + X_t σ dW_{t +}^!_w^!z_t ( dt) 

 Dumb money and quadratic impact model  f (a_t) = −∝a_t ∝ > 0 

• The linear impact model with may be overly simplistic: it  only captures the negative long-term impact but misses the positive short-term  impact of money flows (positive inflows should initially increase returns) • A better model is provided by a quadratic impact model: 

f (a_t) = ∝(ξu_t^τ + a_t)(u_c − ξu_t^τ − a_t) = ∝uˆ_t(u_c − uˆ_t), uˆ_t := ξu_t^τ + a_t u_tτ 

u_c ,ξ a_t 

• Here are additional parameters, and is a running mean of  T 

u_t^τ _:=¹_Ta_t−t′ 

∑ 

t′=1 

• Such quadratic impact function could implement the ‘dumb money’ effect when the impact is positive for small flows but becomes negative for large flows (see  A. Frazzini and O.A. Lamont, “Dumb Money: Mutual Fund Flows and the Cross Section of Stock Returns”, Journal of Financial Economics, Elsevier, vol. 88(2),  

pages 299-322 (2008))  

a_t= ηu_t^τ +φX_t 

• With quadratic impact, we can take a simpler control and still  get the same QED dynamics with different (and time-dependent) parameters  

θ = r_f − c +ηu_t^τ + ∝(η + ξ)u_t^τ (u_c − (η + ξ)u_t^τ ) 

( ⁾), g = ∝φ²

κ = −φ 1+ ∝(u_c − (η + ξ)u_tτ 

 The drift term is robustly non-linear  

• We observe that the non-linear (cubic) drift of the QED model is robust:  _κ − X_{t −} ^g_κX_t2 ^⎛_⎝⎜^⎞_⎠⎟ dt + X_tσ dW_t 

dX_t=κ X_tθ + w^T z_t 

θ,g,κ

• The same cubic drift (with different parameters) is obtained with either linear impact and quadratic control, or with quadratic impact and linear control  

• This suggests that a non-linear drift with at least cubic non-linearity in the state variable should arise in any model that incorporates both money flows as investors’ decision variables, and their impact on asset prices. The QED model is a ‘minimal’ model of such sort. More complex control or price impact function would produce non-linearities higher than a cubic one. 

• The QED dynamics provides a specific cubic non-linearity in the drift for a general class of Ito’s diffusion (with the GBM model being a special case) 

dX_t= ∝ X( _t)dt +σ X( _t)dW_t 

• Alternatively, the QED dynamics is a special case of Langevin dynamics 

 The Langevin equation  

• Paul Langevin (1908) generalized Einstein’s theory of a free Brownian motion to  a motion of Brownian particles in a potential field 

• This is the same as Ito’s diffusion, but it gives a meaning to the drift:

• A potential can for example be produced by heavy molecules • Example: the harmonic potential  

Negative gradient of

 The GBM equation as the Langevin equation  

• The GBM diffusion with drift can be viewed as a Langevin equation 

• The potential for the GBM model is 

• This is the potential of an inverted (negative mass) oscillator! This describes  unstable dynamics! 

• Unstable dynamics never persist in nature for an unlimited time – therefore the  GBM model is incomplete!  

• The unlimited exponential asset growth of the GBM model is in fact a fall in an  inverted harmonic oscillator potential! A fall in such potential can be perceived  as a growth only if we are in an Inverted World ourselves!  

• Non-linearity of the QED model can stabilize the dynamics! 

https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3669972

 Non-linear Langevin dynamics of the QED model  ^!z_t= 0 

• The QED diffusion can be viewed as a Langevin equation (set here)  dX_{t= −} ∂U 

∂X_tdt + X_tσ dW 

U(x) 

• Here is a quartic potential (includes market interactions!):  _{U(X ) = −} ¹₂θ X ² ₊¹₃κ X³ ₊¹₄gX ⁴

• 

 Instantons 

• In quantum field theory (QFT) and quantum mechanics, barrier transitions  (tunneling) are described by instantons: saddle-point solution of classical  equations of motions with an inverted potential (A. Polyakov, 1975), and bounces: instanton-anti-instanton pairs (Coleman and Callen, 1979) 

• George Gamov (1928) explained the alpha-decay as tunneling of alpha particles from a well of a nuclear potential – this is an instanton in the modern language. (See the next slide)  

Digression: How would statistics explain the alpha decay? 

• The alpha decay is a rare event of a spontaneous emission of alpha-particles (helium nuclei) by some heavy nuclei. This is one of the most famous phenomena explained by quantum mechanics 

• A pure statistical approach would describe it as a rare  Poisson event, but would it actually explain anything?  • Many papers in quant finance start with the words ‘We  consider the XYZ process because it fits market data  better…”. This is a Ptolemaic approach – we want instead a  structural/physics-based approach…  

• Now lets go back to Q1 and Q2 from the first slide…  

Q1: Where are non-linearities in market models?

• All interesting complex systems in physics, mechanics,  chemistry, biology etc. are non-linear. 

• Where are non-linearities in finance?  

• Non-linear models are used in finance for some problems  (e.g. a worst-case option pricing, XVA, trade impact models), but are not viewed as critical for baseline/foundational models such as the CAPM, Geometric Brownian Motion model, or the Black-Scholes model 

• On one hand, Machine Learning folks operate with highly nonlinear models, but do it in  

uncontrollable ways… 

• On the other hand, the QED model  

produces a particular quartic non-linearity  

• Instantons are essentially non-perturbative  

phenomena: they cannot be seen within a perturbation theory framework (and also they disappear in the linear limit)!  

Q2: Why equilibrium models? • Take CAPM: 

• This is a cross-sectional regression model that is equivalent to a stationarity assumption of equity returns which are assumed to equilibrate at each time step and produce a Gaussian distribution centered at . Is it a realistic assumption?  

“There are at least five wrong definitions of equilibrium in the economics and finance literature. The first three definitions would be correct were markets stationary, but economic processes are known to be non-stationary”.

• Intermediate conclusion: We want a model of non-linear (and generally non-equilibrium) dynamics that would be similar to the QED model but more analytically tractable. We will do it by mapping the problem onto quantum mechanics! 

From the Langevin equation to quantum mechanics 

• Start with the Langevin equation for log-returns  

• Step 1: Start with the corresponding Fokker-Planck equations (FPEs) for  transition probabilities where corresponds to the  original Langevin Equation (LE), and corresponds to a LE with a  flipped sign of the potential.  

• Step 2: Use the ansatz  

• These produces two Schroedinger equations for the equivalent quantum mechanics (QM) in imaginary time, with serving as a Planck constant 

QM solution for the FPE transition density 

• Let be a complete set of eigenstates (Wave Functions) of Hamiltonian with energies (eigenvalues) . The eigenvalue expansion of the solution of the FPE for is  

• A stationary solution exists If the energy of the QM ground state is zero.  For this case, keeping only a leading pre-asymptotic term, we obtain 

• Here is the energy split between the first excited state and  the ground state with  

• This gives a solution to the FPE equation (and the Langevin dynamics) in terms  of quantum mechanics with wave functions (WF’s)  

• In the traditional approach, the WF’s are determined by a potential that we will do differently, by defining a potential in terms of a ground state wave function  

The Non-Equilibrium Skew (NES) potential 

• The main idea: in a spirit of ML, let’s directly specify the ground state WF by a parameterized function  

• Benefits of physics: everything else (including dynamics) will be fixed in terms  of  

• Use a two-component Gaussian mixture distribution as a model of the QM  ground state WF: For this case, keeping only a leading pre-asymptotic term, we  obtain 

• The vector of model parameters is then, assuming that the two drifts are  

• The choice of the ground state WF now fixes the Langevin potential :

• Our Gaussian mixture model is inspired by a double well potential in QM, one  of the most canonical examples in physics that serves there as a model for a  quantum or thermal tunneling phenomena.  

Quant Finance as Quantum Mechanics Instantons are (almost) everywhere!

• By defining a ground state , we can define the dynamics of returns • For this, need to solve for and other excited states. Let’s use SUSY!  

Supersymmetric Quantum Mechanics (SUSY QM)  

• SUSY QM was proposed by E. Witten in 1981 as a toy model to study supersymmetry (SUSY) in quantum field theory and string theory • In 1982, Feigelman and Tsvelik discovered a hidden SUSY of Langevin dynamics. 

• This means that Langevin dynamics are mathematically equivalent to SUSY QM of Witten!  

• Why is SUSY useful? Many benefits for QFT, but for stochastic Langevin dynamics, the main advantage is a drastic reduction of computational complexity: second order Schroedinger equations are reduced to first-order equations due to factorization of the Hamiltonians 

Superpotential

• Zero-energy (ground-state) solutions for solve first order equations:

• If is known, then the superpotential and potential are known:  

The magic of SUSY 

• SUSY produces degeneracy of eigenstates and eigenvalues of Schroedinger  Hamiltonians : 

• SUSY can be used to replace the problem of finding the first excited state of by the problem of finding the ground state WF of the SUSY partner Hamiltonian, and then obtain by making a SUSY transformation on it.  

• Why is this useful? Because very often a ground state is much easier to find than a first excited state! Many good methods (perturbation theory, WKB etc.) are available in quantum mechanics for a ground state WF.  

• We can then use SUSY QM to build any model of returns in finance!

• Skip a few pages of calculations in the paper, and jump to pictures… • (Details are in https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3724000) 

SUSY and the NES in pictures  

The excited  

state

• The first excited state looks very similar to the first excited state of a QM harmonic oscillator. We will use it to simplify the model even more!  

The NES model: practical implementation 

• Based on the above analysis grounded in SUSY, approximate 

• This gives an approximation for a non-equilibrium transition density:

• Can simplify it even further by replacing, which we can do under the same assumptions. This gives 

• With the Gaussian mixture for, this produces a three-component Gaussian mixture as a non-equilibrium transition density: 

• All parameters in this Gaussian mixture are fixed in terms of the original model parameters. A degree of non-equilibrium is controlled by parameter where is called the Kramers escape rate.  

Option pricing with the NES model   

• Move from the real measure to a risk-neutral measure using the Minimum Cross  Entropy method (the Esscher transform):  

• This is another three-component Gaussian mixture! An option price for a European call is therefore a mixture of three Black-Scholes prices:

• Here the ‘Black-Scholes’ parameters are as follows: 

• All the non-equilibrium effects are now absorbed into Non-Equilibrium Dividends  (NEDs) , where parameter is found from a constraint that has a  risk-free expected return  

• Computational complexity is only about three times more than the Black Scholes! 

 Implied potentials for 1M options on SPX on 07/12/2021, exp 08/9/2021  

Calibrated parameters for puts:  

mape= 3.6% 

Current value of log-return

Calibrated parameters for calls: 

mape= 4.4% 

Implied potentials obtained for puts and calls are different! 

 Implied potentials for 1Y options on SPX on 11/6/2020, exp 09/21/2021  

Calibrated parameters for puts:  

mape= 5.67% 

Calibrated parameters for calls: 

mape= 2.57%

Implied potentials obtained for puts and calls are different! 

Market in distress: potentials for 6M options on SPX on 03/16/2020, exp 09/18/2020   

Calibrated parameters for puts:  

mape= 0.67% 

Non-equilibrium current value

Calibrated parameters for calls: 

mape= 0.72% 

The double well potential describes metastability ! 

Instantons on Wall Street?  

We see that for a stressed market environment, implied potentials can be of a  double-well type, leading to metastability and instantons! 

George Gamow  (1904-1968) 

Non-perturbative finance: non linearities are foundational!

physics -> finance -> RL -> physics

Alexander Polyakov  (b. 1945) 

Can the two alphas be related? 

Summary 

• Quant Finance models should: 

• Be accurate in their definitions (‘equilibrium’ etc.)  • Be explainable! (Compare with financial mathematicians’ approach “We consider the XYZ stochastic process because it fits data better”) 

• Focus more on the drift rather than the diffusion term (did it focus on ‘non-interesting’ models for the last 55 years)? • Model building: We need non-equilibrium and non-perturbative finance! 

• Can combine insights and methods from both machine learning (fitted parameterized functions, Gaussian mixtures, etc.) with physics (quantum mechanics, SUSY, instantons, etc.) 

• The NES model shows that it all can be done in a simple and practical way!  

Written by Dr. Igor Halperin ( Dr. Halperin LIVE at the MIT Quant & Ai Conference )

Science