The Hidden Assumption in the Markowitz‘s Optimization
The optimization of Markowitz, Nobel Prize winner in 1990, who passed away in June 2023, has had a major impact on finance. It influenced discretionary asset management, with better diversification, systematic strategies (risk parity portfolios, minimum variance, maximum diversification, quantitative alternative funds), as well as the restructuring of the industry around indices and ETFs. Also, corporate finance, with the famous CAPM formula (Capital Asset Pricing Model) which has since defined discount rates or required rates of return.
However, there is unanimous agreement that in practice, the Markowitz formula does not work. In my article [1] in the Journal of Investment Strategies, I introduce a second matrix that has an impact on asset management, for those who do not consider the market to be in equilibrium and who want to outperform it. It seems that one should just avoid using the Markowitz formula (the inverse of the correlation matrix) to optimize their portfolio. Practitioners will thus understand why the Markowitz formula does not yield good results, and that it is necessary to arbitrarily reduce correlations or adjust the formula.
As a reminder, portfolio optimization is based on risk minimization while maintaining the portfolio’s return constant. Markowitz then derives the solution by assuming primarily that the expected returns of the different assets are known (and certain) and that risk is measured by the standard deviations of the returns. The mathematically derived solution then simply determines the portfolio’s risk weights according to the expectations normalized by the risks (Sharpe ratios), transformed by the inverse of the returns’ correlation matrix. This transformation will disrupt the construction of the portfolio, overweighting the weights of assets that provide diversification or hedging, for those that are weakly or negatively correlated and underweighting the others. Unfortunately, in practice, the Markowitz formula assumes that expectations and correlations are not uncertain. It will focus on combinations that would generate somewhat aberrant high returns, based on the few misleading correlations and expectations considered safe and certain.
In 2018, an article written by physicists, based on symmetry considerations inspired by physics, claimed that only the ‘rotationally invariant’ portfolio should make sense. However, this portfolio is determined by the inverse of the square root of the correlation matrix, which is inconsistent with the Markowitz formula and the inverse of the correlation matrix. The authors, however, did not demonstrate the optimality of such a portfolio.
The inconsistency between the two formulas intrigued me greatly. I realized that it was necessary to introduce a second correlation matrix to describe the correlations between the expected returns, which were in fact themselves stochastic and uncertain. One of my colleagues then argued that adding uncertainty to expectations only increased variance. However, the uncertainty of expectations does not generate risk, at least not the one measured by practitioners*. The second correlation matrix can also be interpreted as the correlation matrix of returns over large time scales (greater than several years), whereas the first matrix is based on returns taking short time scales (less than a month).
The optimal solution incorporating this second matrix was derived in my article. It depends not only on the correlation matrix of returns but also on this second matrix, which remains unknown because it is difficult to estimate. Then several forms of this second matrix were implicitly identified in several cases, notably to retrieve the Markowitz solution or that of the physicists’ square root, thus unifying the different approaches. To retrieve that of Markowitz, the second correlation matrix must be similar to the first with an invariance of the correlation matrix. To retrieve that of the physicists, the second correlation matrix must have a certain random form. However, the random form seems much more probable**. Thus, the square root of the correlation matrix would be more suitable, which means that when correlations are weak, they are effectively halved in Markowitz optimization. This is indeed a commonly used cooking recipe that is now justified.
In summary, this second matrix introduced has no impact on the CAPM formula where the market portfolio is assumed to be in equilibrium with the consequence of an invariance of the correlation matrix. It is clear that Markowitz was awarded the Nobel Prize because other economists, including Sharpe, continued his work leading to the CAPM, a model of equilibrium, which determines the theoretical expected returns so that the market portfolio (owned by all investors) is the optimal portfolio. Thus, the link between the required return by financiers, the return of the risk-free rate, and the beta of the investment relative to this assumed optimal market portfolio is found.
But now we know, thanks to this second matrix, why it is better to avoid using the inverse of the correlation matrix to optimize one’s portfolio.
[1] VALEYRE, S. Optimal trend-following portfolios, Journal of Investment Strategies 2024 volume 12 N°3
