How do I know if my model is overfitting?
Artificial Intelligence & Machine Learning
The paper titled “Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance” by Marcos López de Prado and others explores the problem of overfitting in financial data analysis and its implications for investors and financial practitioners.
The author begins by highlighting the importance of data analysis in finance, given that investment decisions are often based on the analysis of historical data. However, he notes that such analysis can be easily manipulated by overfitting the data, which involves fitting a model to the data to the extent that it becomes too specific and loses its predictive power on new data.
To illustrate the problem of overfitting, the author provides an example of a fictitious investment strategy that has been back tested on historical data and found to perform well.
However, when the strategy becomes tested on new data, it performs poorly. As a result, suggesting that it became overfitted to the historical data. The author argues that such strategies find themselves prevalent in the financial industry. And often become used to deceive investors.
Furthermore, the author also proposes a new methodology called the “Multiple Hypothesis Testing Framework,” which involves testing a large number of hypotheses and correcting for the multiple comparisons to ensure that the results are not spurious. The author demonstrates the effectiveness of this approach using several examples and argues that it can help to mitigate the problem of overfitting in financial data analysis.
The author also highlights the need for greater transparency in the financial industry, particularly with regard to the disclosure of back testing methods and results. He argues that investors should be provided with more information about the limitations of back testing and the potential risks of investing in strategies that are based on back tested data.
Overall, the paper makes a valuable contribution to the literature on financial data analysis and the problem of overfitting.
The author provides a clear explanation of the problem and its implications for investors and financial practitioners, and proposes a new methodology for mitigating the problem. However, the paper could have benefited from a more detailed discussion of the limitations of the proposed methodology, as well as a more comprehensive review of the existing literature on the topic.
The author’s focus on the need for greater transparency in the financial industry is particularly noteworthy. By highlighting the potential risks of investing in strategies that are based on back tested data, the author is effectively advocating for greater accountability and ethical conduct in the industry.
This is an important message that should become heeded by all financial practitioners. And the author’s proposed methodology provides a useful tool for achieving this goal.
In conclusion, “Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance” is a valuable contribution to the literature on financial data analysis and the problem of overfitting. The author provides a clear explanation of the problem, proposes a new methodology for mitigating the problem, and highlights the need for greater transparency in the financial industry. While the paper could have benefited from a more detailed discussion of the limitations of the proposed methodology, it is nonetheless a thought-provoking and informative read for anyone interested in the intersection of finance and data analysis.
Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance by David H. Bailey, Jonathan Borwein, Marcos Lopez de Prado, Qiji Jim Zhu :: SSRN
Written by Jiatong Li
How do I know if my model is overfitting?
