Can LLMs store information in a manner akin to Hopfield associative memories (HAM)?

October 23, 2024

Can LLMs store information in a manner akin to Hopfield associative memories (HAM)?

Artificial Intelligence & Machine Learning

The idea that Language Models (LLMs) store information in a manner akin to Hopfield associative memories (HAM) opens up fascinating discussions on the capacity of these models. Let’s break down the key concepts, particularly focusing on the claim that LLMs can store approximately 2 bits of information per parameter, which is reminiscent of capacity scaling laws in physics-based models like the Hopfield networks.

Hopfield Networks and Memory Capacity

Hopfield networks, first proposed in the 1980s, are a form of associative memory where the capacity for storing memories is directly related to the number of parameters (neurons or weights) in the system. It is this understanding of capacity scaling—where the maximum number of stored patterns is proportional to the number of neurons—that earned John Hopfield his Nobel Prize. In a classical Hopfield net, you can store about 0.14N patterns, where N is the number of neurons.

Similarly, transformers, like the architecture behind LLMs, have properties that resemble modern Hopfield nets in terms of memory, where the network can retrieve stored information based on partial inputs (associative retrieval).

LLM Capacity: 2 Bits Per Parameter

The claim that large language models, such as GPT, can store approximately 2 bits of information per parameter is a fascinating result. The “Physics of Language Models” suggests that the scaling laws for knowledge capacity in LLMs are similar to the theoretical capacities we compute for neural associative networks like Hopfield nets.

This translates to the following key observation:

A sufficiently trained language model with 7 billion parameters can store 14 billion bits of knowledge (7B parameters * 2 bits per parameter).
The knowledge stored in the model is not memorized word-for-word but rather flexibly extractable. This means the model doesn’t simply regurgitate facts but can apply its stored knowledge in different contexts (e.g., answering questions or solving tasks).

Capacity Calculation Based on Data Size

In this thought experiment, we take the total available knowledge in English—estimated as 20.5 billion words from sources like Wikipedia and textbooks—and compress it down to 14 billion bits of essential, non-overlapping information. This compression takes into account redundancies in natural language where much of the information can be deduced or inferred from context.

Given that an LLM with 7 billion parameters can theoretically store 14 billion bits of knowledge, the idea is that a well-trained LLM can hold and process the entire body of available English-language knowledge (from Wikipedia, textbooks, etc.). Importantly, this storage is distributed across the layers of the network, involving both the self-attention mechanisms and the MLP blocks within the transformer.

Hopfield-like Behavior in Transformers

The reason why transformers are compared to modern Hopfield networks lies in their ability to distribute knowledge across many parameters, where retrieval doesn’t depend on memorizing exact sequences but rather relies on flexible, pattern-based storage and retrieval. The distributed nature of this information storage is what allows the model to:

Extract specific knowledge on demand (e.g., answering factual questions),
Apply knowledge to downstream tasks (e.g., summarizing or translating),
Use the stored information in nuanced and context-sensitive ways, such as understanding relationships or comparing concepts.

Training Exposure and Robustness

Another key point raised is the necessity for repeated exposure during training. For the model to effectively store a piece of knowledge, it must encounter the data around 1000 times during training. This high number of exposures ensures that the knowledge is deeply embedded in the model’s parameters and can be recalled or applied flexibly.

However, a significant challenge comes from the quality of the data:

If the training data is of poor quality, it severely impacts the model’s capacity, potentially reducing it by a factor of 20x. This means that noisy or uninformative data takes up much more memory “space” in the model, reducing its overall effectiveness in storing useful knowledge.

Applications and Scaling

The practical implications of understanding memory capacity in LLMs are vast. For instance:

Quantization effects: Lower precision in representing weights can impact the model’s memory capacity and retrieval ability.
Scaling laws: As models grow in size (e.g., GPT-3 with 175 billion parameters), we can expect to store even more knowledge, but the efficiency of this storage is contingent on the quality of the training process and data.

This framework of thinking also applies when comparing LLMs to other architectures, where transformers might outperform less optimal networks due to their Hopfield-like retrieval capacity.

Conclusion

The memory capacity of LLMs follows a simple scaling law similar to that of Hopfield associative memories. With 2 bits of information stored per parameter, a well-trained 7B parameter model could theoretically surpass the knowledge stored in Wikipedia and other English texts. However, the model’s effectiveness depends not only on the size but also on the quality of the training data and repeated exposure to the information. The flexibility and distributed nature of knowledge storage in LLMs allow them to perform sophisticated tasks that go beyond simple memorization, reflecting the deep parallels between LLMs and Hopfield networks.

This evolving understanding of the “physics of language models” opens the door to more precise and optimized architectures for knowledge storage and retrieval in artificial intelligence systems.