Emergence of diffusion models from associative memory through memorization and generalization

Dense Associative Memories (DenseAMs) are generalizations of Hopfield networks that store training data at local minima of the energy landscape. When the amount of training data exceeds the critical memory storage capacity, new local minima (different from the training data) emerge. These emergent local minima, known as spurious states, can hinder memory retrieval.

We examine diffusion models (DMs) through the DenseAM lens, viewing their generative process as an attempt of a memory retrieval. In the small data regime, DMs create distinct attractors for each training sample, akin to DenseAMs below the critical capacity. As the training set size increases, DMs transition to generalization.

We identify a critical intermediate phase, predicted by DenseAM theory, characterized by the emergence of spurious states. In generative modeling, these previously overlooked states are no longer negative artifacts but rather are the first signs of generative capabilities. Their existence is demonstrated across a wide range of architectures and datasets.

Most of the results presented here were obtained while D.K. was employed by IBM Research. At the time of the submission, D.K. is no longer employed by IBM Research. M.N. was employed at Sapienza University of Rome for most of the time spent working on this project, and completed part of this work following his appointment at CY Cergy Paris Université.

M.N. acknowledges the support of the PNRR MUR project PE0000013-FAIR during his period at Sapienza University. B.P. and M.J.Z. acknowledge the support of the RPI-IBM Future of Computing Research Collaboration program (FCRC). G.R. was funded by the Dutch Research Council (NWO) as part of the CERTIF-AI project (file number 17998).

These authors contributed equally: Bao Pham, Gabriel Raya. Department of Computer Science, Rensselaer Polytechnic Institute, Troy, NY, USA Jheronimus Academy of Data Science, Tilburg University, Tilburg, The Netherlands Department of Physics, University of Rome Sapienza, Rome, Italy Laboratory of Theoretical Physics and Modelling, CY Cergy Paris Universitĕ, Cergy-Pontoise, France Donders Institute for Brain, Cognition, and Behaviour, Radboud University, Nijmegen, The Netherlands The authors declare no competing interests. Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.