How to generalize the diffusion model to solve high-dimension problems?
SCORE-BASED DIFFUSION MODELS IN FUNCTION SPACE
Jae Hyun Lim & Nikola B. Kovachki’s paper discussed the probability of using the diffusion model to solve problems in high-dimension function space. The paper proposed the comprehensive implementation and validation of the high-dimension diffusion model’s forward process, generative process, and score function. In addition, verified the applicability of their network in solving a set of problems.
The paper proposed the comprehensive implementation and validation of the high-dimension diffusion model’s forward process, generative process, and score function.
The paper proposed a mathematically rigorous framework called Denoising Diffusion Operators (DDOs) to perform denoising score matching in function space. The work is taken on an infinite-dimensional, real, separable Hilbert space, in which there exists a countable or finite set of basis vectors that can be used to represent any element of the space.
In order to sample from the perturbed measure, the paper considered the infinite-dimensional, pre-conditioned, Langevin equation. Under the appropriate assumption, such an equation has a unique strong solution with continuous paths and an invariant measure ν. In particular, samples from ν can be obtained as the long-time solutions of the equation.
The paper also employed the neural operator framework of Kovachki et al.[1], which suggests that approximating score operators in infinite dimensions is feasible using neural operators.
The paper verified the applicability of their network in solving a set of problems.
The paper proposed numerical experiments using Fourier Neural Operator (FNO) in both Gaussian mixture and solving Navier-Stokes equation. Furthermore, to prove that the network will apply trace-class noise that achieves consistent error that is independent to current resolution. And the model can generalize to high resolutions at no extra cost, performing super-resolution natively.
The paper proved that compared to white noise, which generates noise with an increasing error when scaling the size of the model, the network generates trace-class noise that is consistent regardless of the current resolution(Figure 2).
Figure 2 : Gaussian mixture : (a) Uniform-norm error in the average spectra of samples when using trace-class noise vs. white noise. (b) Generated samples at a resolution of 256 with trace-class noise and (c) with white noise.
As for solving high-dimension problems, the paper used 128×128 data and valuable noise to train the model and use it to generate the answer to the vorticity form of the Navier-Stokes equation (on the 2D-torus with a Reynolds number of 500, which means in a donut-shaped space and the fluid flow is highly complex and dynamic). It proved that by using small resolution data to train, the model can generate high-resolution(1024×1024) without any re-training, and the result is very close to the ground truth value(Figure 3).
Figure 3: Navier-Stokes: (a) Generated sample at the resolution 1024 × 1024 moreover with a model trained at 128 × 128, (b) Spectrum, (c) Pointwise value density, (d) Total kinetic energy density of samples from the model vs. the data.
However, although the paper is well-written, it is difficult for those who is lack of academic background to understand.
Since the results of the thesis are derived on the basis of the diffusion model, and the paper used numbers of Mathematical and Fluid Mechanics Formulas to implement and verify the model, In order to fully understand the implementation, the paper requires the reader to have some level of understanding of machine learning, mathematics, and physics.
The paper needs to use more chapters to discuss the problem solving ability of high dimension diffusion model.
Although the paper proved that the model can provide promising numerical results of Navier-Stokes equations, the paper could provide more similar examples to improve readers’ understanding of their work.
The paper provides rigorous mathematical proof of the model implementation, which is inspiring for the new application of diffusion models.
Most of the work related to diffusion models is based on finite-dimensional space, such as Euclidean. Laslty, this paper provides a new vision of it.
Overall, the paper proposed a innovative operator (DDO) for diffusion model to shine in high dimensional function space, however its application scenarios need to be further discussed.
Read the full paper:
[2302.07400] Score-based Diffusion Models in Function Space (arxiv.org)
Written by Albert Zhan
Reference:
[1] Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces. arXiv preprint arXiv:2108.08481, 2021a.
[2] Lim J H, Kovachki N B, Baptista R, et al. Score-based Diffusion Models in Function Space[J]. arXiv preprint arXiv:2302.07400, 2023.
What is geometric deep learning?
How to generalize the diffusion model to solve high-dimension problems?
