Dr Michael Kollo & DeepResearch Review “AI-Powered (Finance) Scholarship” by Novy-Marx and Velikov
Artificial Intelligence & Machine Learning
Review of “AI-Powered (Finance) Scholarship” by Novy-Marx and Velikov
Comparison to Existing Literature
Novy-Marx and Velikov’s paper tackles long-standing concerns about data mining in empirical finance. The proliferation of predictive factors, often referred to as the “factor zoo,” raises the risk that many of these anomalies are false positives, a result of extensive data-snooping rather than genuine market inefficiencies. Prior research, such as Harvey, Liu, and Zhu (2016), has highlighted that the standard significance threshold (t≈2) is too low, suggesting that new factors should meet a higher standard (t > 3) to be considered meaningful. Similarly, McLean and Pontiff (2016) demonstrated that many anomalies lose profitability post-publication, reinforcing concerns about data-snooping biases and arbitrage eroding genuine signals.
Novy-Marx and Velikov’s approach extends this concern to an unprecedented scale, mining over 30,000 potential accounting-based signals. While their method aligns with recent machine learning applications in finance (e.g., Gu, Kelly, and Xiu, 2020), it differs by relying on a brute-force search for strong individual anomalies rather than combining weak predictors into a composite model. The results reaffirm the consensus in empirical asset pricing: extensive searching will inevitably uncover in-sample patterns that lack true predictive power. To mitigate this, the authors employ their “Assaying Anomalies” protocol, designed to rigorously validate signals and filter out spurious findings. This protocol builds on prior literature that emphasizes stringent out-of-sample testing and robustness checks for factor discovery (e.g., Ferson, Sarkissian, and Simin, 2003; Harvey et al., 2016). By benchmarking their discovered signals against 200 known anomalies, the authors contribute to the factor model evaluation literature, testing whether their findings provide incremental explanatory power or merely reflect existing anomalies.
Hypothesis Formation and HARKing
A central theme of the paper is the issue of HARKing (Hypothesizing After Results are Known), a practice widely regarded as problematic in academic research (Kerr, 1998). In finance, researchers often uncover empirical regularities first and then retrofit theoretical explanations to justify them. Novy-Marx and Velikov expose this practice by automating post hoc theorizing through AI-generated justifications for each anomaly. This aligns with concerns raised in the literature that many financial theories emerge after empirical results, rather than guiding hypothesis formation a priori. The paper goes further than previous critiques by industrializing this process—each AI-generated introduction constructs a plausible-sounding economic rationale, complete with citations, demonstrating how easy it is to craft post hoc explanations for any observed pattern. This mirrors Campbell Harvey’s (2017) argument that with enough flexibility in storytelling, any statistical result can be framed as meaningful.
By highlighting how AI can generate convincing but potentially misleading narratives, the paper underscores a key issue in empirical finance: if theories can be generated on demand, how do we distinguish real insights from statistical artifacts? This contribution extends existing critiques of data-driven hypothesis formation into the AI era, reinforcing the need for higher methodological standards.
Novelty and Contribution
This paper’s contributions are significant on multiple fronts:
- Scaling Automated Anomaly Discovery
Previous studies have tested hundreds of potential anomalies; this paper expands the scope to tens of thousands of accounting-derived signals. The identification of 96 promising signals from over 30,000 variables is unprecedented in empirical asset pricing. While some of these signals may turn out to be statistical flukes, the approach provides valuable insights into which types of accounting transformations appear predictive under rigorous validation. The accompanying “Assaying Anomalies” protocol serves as a large-scale stress test for anomaly discovery, helping delineate the boundary between genuine return predictors and spurious correlations. - AI-Generated Research and Its Implications
The paper demonstrates how Large Language Models (LLMs) can automate academic writing, producing full research papers from raw empirical results. While AI-assisted writing tools have been used informally, Novy-Marx and Velikov take this further by fully automating the academic paper-writing process, generating tailored theoretical explanations and citations for each anomaly. This raises two key implications:- Potential Efficiency Gains: AI can assist with literature reviews, idea generation, and routine research tasks, freeing scholars to focus on conceptual development and interpretation.
- Ethical Concerns: The same automation could be misused, leading to a flood of convincing but unverified research, exacerbating the replication crisis in finance.
By automating both empirical discovery and theoretical justification, the authors highlight the risk of AI-driven research inflation, where thousands of plausible but weak findings could be published with minimal human oversight. This contribution is both a demonstration of AI’s capabilities and a warning about its potential misuse.
Methodological Rigor and Potential Pitfalls
The paper employs rigorous validation techniques to mitigate data-mining biases, including:
- High statistical significance thresholds for anomaly selection.
- Benchmarking against 200 known anomalies to ensure new signals provide incremental value.
- Consideration of implementation constraints, such as trading costs and short-sale constraints, to assess real-world applicability.
Despite these strengths, some concerns remain:
- Multiple Testing Bias: Even with rigorous filtering, some of the 96 identified anomalies may be false positives. The authors should report a false discovery rate or conduct a randomization test to estimate how many of these signals would appear significant by chance alone.
- Lack of Forward-Looking Validation: A true out-of-sample test, using only data up to a certain year for discovery and later data for validation, would strengthen the robustness of findings.
- AI-Generated Citations: The paper acknowledges that some AI-generated citations are fabricated, which is a serious integrity issue. A methodological improvement would involve restricting citations to verified databases.
While the empirical validation is strong, the AI-generated theoretical narratives highlight a key methodological concern: if AI can generate multiple, conflicting explanations for the same anomaly, how do we determine which, if any, is valid? The paper should explicitly discuss this issue and its implications for anomaly research.
Recommendations for Improvement
- Quantify False Discovery Risk – Implement a false discovery rate analysis or randomization tests to estimate how many of the 96 signals are likely false positives.
- Out-of-Sample Validation – Test anomalies on unseen future data to assess their persistence beyond the initial sample.
- Clustering Analysis – Determine if the 96 signals represent truly independent factors or variations of a smaller set of core anomalies.
- Address Theoretical Inconsistencies – Explicitly acknowledge that AI-generated hypotheses may contradict one another, reinforcing the paper’s cautionary message.
- Improve AI Citation Handling – Restrict AI-generated citations to verified sources to prevent fabricated references from undermining credibility.
- Transparency on Ethical Considerations – Clearly state that the AI-generated papers are not intended for publication, but serve as an experiment to illustrate research risks.
Conclusion and Broader Implications
Novy-Marx and Velikov’s study is a landmark contribution that challenges how financial research is conducted in the AI era. By demonstrating how large-scale data mining and AI-generated narratives can create an overwhelming number of plausible but potentially spurious findings, the paper raises critical questions about scientific rigor, research integrity, and the role of AI in academia.
If published, the paper could:
- Raise the standards for empirical finance research, reinforcing the need for stricter validation protocols.
- Encourage AI-assisted research practices, but with clear guidelines on proper oversight.
- Spark discussions on ethical AI use in academia, potentially leading to new norms for disclosure and verification.
While the paper is already well-executed, minor revisions—particularly in statistical validation, citation accuracy, and theoretical framing—would enhance its credibility and impact. Overall, this study is highly innovative and thought-provoking, warranting strong consideration for publication with minor revisions.
