How Have Models Changed?
Artificial Intelligence & Machine Learning
The significant advancements in modern computing have been pivotal for the evolution of modeling, particularly in statistics. This progress is not just about enhanced memory, swifter CPUs, or more user-friendly programming languages. A crucial aspect became the development of statistical algorithms optimized for efficient computation.
These algorithms, developed over the last half-century, ‘statistical’ in nature, having become conceived within the structure of statistical problems.
The EM algorithm, introduced by Dempster, Laird, and Rubin in 1977, and expanded by Meng and van Dyk in 1997, is a prime example, as are the Gibbs sampler, particle filters, variational inference, and expectation propagation. These methods utilize the conditional independence structures inherent in statistical models to optimize computation.
Algorithms like the Metropolis algorithm and Hamiltonian Monte Carlo, initially formulated for physical sciences, have been adapted for statistical computing, much like optimization algorithms were previously adapted for computing estimates in statistics. Another technique, known as approximate Bayesian computation, circumvents the direct evaluation of the likelihood function by simulating from the generative model, especially when the likelihood’s analytical form is complex or computationally demanding.
The intertwining of data analysis, probability modeling, and computing has been a constant in statistics.
New models often inspire innovative computational algorithms, which in turn enable more complex models and novel inferential ideas. The development of automatic inference algorithms has allowed for model flexibility, enabling changes in the model without necessitating alterations to the algorithm’s implementation.
Decision theory has also evolved, initially framed as a foundational aspect of statistics through utility maximization, error-rate control, and empirical Bayes analysis. Recent developments have built upon these concepts, integrating Bayesian decision theory and false discovery rate analysis. Psychological research on decision-making heuristics and biases has further influenced this field.
Decision-making can become viewed as a statistical application area. Key developments include Bayesian optimization and reinforcement learning. Linked to a resurgence in experimental design for A/B testing in industry and online learning in engineering. Richly parameterized models like Gaussian processes and neural networks now utilized in adaptive decision analysis for large-scale applications. Such as robotics, text generation, and AI for games like Go.
Outside of traditional statistics, methods like nonnegative matrix factorization, nonlinear dimension reduction, generative adversarial networks, and autoencoders have emerged. These unsupervised learning methods are instrumental in discovering structures and decompositions.
Robust inference remains central to modern statistics, focusing on models’ performance under realistic assumption violations. This concept becomes particularly relevant in economics. Where statistical models’ imperfections we find acutely recognized. Robust standard errors and partial identification stand as examples of methods extensively used in applied research.
Overall, the impact of robustness in statistics goes beyond specific methods. It encompasses the evaluation of statistical procedures in a ‘M-open’ world, where the data-generating process does not align perfectly with fitted models. This perspective became crucial for densely parameterized models characteristic of contemporary statistics. Lastly, has broader implications for model evaluation and improvement.
Written by Columbia MAFN Student Ruslan Polovinko
