Large Language Models On Financial Statement Analysis
Ai & Machine Learning / Trading and Investing
Review written by revered Australian quant Michael Kollo!
A new paper from the University of Chicago has tested language models in financial analysis.
The study focused on forecasting the direction of earnings changes (+ve / -ve) one year ahead, using 150,000 firm-year observations from 1968 to 2021. It ensured that each prediction was based solely on information from the financial statements and cash-flow statements of the year leading up to the prediction.
It compared the results with two standard statistical techniques, one achieving 53% accuracy and the other 60% accuracy (assuming a base case of 50% accuracy, indicating the removal of positive drift).
The GPT model, enhanced with a chain of reasoning, achieved 63% accuracy. The authors note that “GPT makes relatively fewer mistakes when predicting the earnings of small or loss-making companies, likely benefiting from its human-like reasoning and extensive knowledge.” This ability to draw upon a broader range of knowledge provides a distinct advantage for the language model.
However, a few biases became noted:
the GPT model itself is trained on news and information from all corpora. So the paper isn’t able to actively limit GPT’s future knowledge in the same way as with other forecasting models. They tested for this and conducted some out-of-sample analysis.
A second concern is what specifically contributes to GPT’s forecasting success. The authors observe that the model frequently calculates standard ratios used by financial analysts and, as instructed by a Chain of Thought (CoT) prompt, generates narratives that analyze these ratios. Further testing found that the narrative insights generated by the model are informative about future performance.
Thus, narratives and analysis of financial ratios are effective at forecasting the direction of earnings—not prices—making this economically significant for financial services firms. It provides a new benchmark for financial analysis and forecasting that demonstrates what AI can achieve.
This result is highly significant, potentially changing research as profoundly as electronic trading transformed the large trading desks of the 1980s and 1990s. It doesn’t imply that research becomes obsolete, but rather that it introduces a new benchmark created by these systems. It also indicates that competition may increase in other areas of financial services for that alpha.
The authors are more optimistic, however, and one sentence is particularly striking: “We find that an LLM excels in a quantitative task that requires intuition and human-like reasoning. The ability to perform tasks across domains points towards the emergence of Artificial General Intelligence. Broadly, our analysis suggests that LLMs can take a more central place in decision-making than previously thought.”
