Was Einstein Just Autocomplete?
How a New Representation Becomes New Information for a Bounded Mind
Often LLM debate comes down to one question: can these models produce anything genuinely new, or are they only predicting text?
The answer depends on what you mean by “new.” In the technical sense, an LLM may not create new information at all. But humans are cognitively limited. We can absorb only a tiny fraction of the information available to us, and even processing that fraction requires substantial time and mental effort. An LLM can process vast amounts of existing data and may surface structures humans have missed or are incapable of constructing on their own.
By representation I mean a structured way of organizing information that changes what can be seen or inferred. Language is itself a representation of reality. It maps the world and our thoughts onto symbols. I’m using “language” broadly here: any system of symbols and rules that lets us encode, compress, and transmit complex mental representations, whether spoken, written, or mathematical.
A new representation can open up cognitive territory that was closed before, and in that sense it becomes genuinely new information for the mind that encounters it. This isn’t the trivial sense in which any fact you happen not to know is “new.” A representation earns the word because it changes what you can infer next, the way a single law predicts motions you’ve never observed.
Einstein
Before Newton, people knew apples fell and planets moved. What didn’t exist was a representation that compressed those observations into a single coherent law human thought could use effectively. Newton supplied one.
Newton was not only a master of representation but also a brilliant experimentalist. He built instruments, performed measurements, and interacted directly with the physical world. His discoveries emerged from a combination of observation and representation.
Einstein pushed much further toward the representational side.
Einstein worked almost entirely inside language. His breakthroughs came from thought experiments performed not on objects but on representations carried in language. Riding alongside a beam of light. Falling inside an elevator. Two clocks drifting apart as the people holding them move past each other. None of these happened in a lab; they happened inside language. By manipulating representations instead of apparatus, Einstein explored possibilities that had never been seriously considered before and eventually arrived at structures that made relativity thinkable.
So here’s the question. If a new representation reorganizes existing information into something human thought couldn’t reach before, is the representation itself a kind of new information? Or was Einstein just doing autocomplete, reshuffling old data into a new arrangement?
Calling Einstein autocomplete feels strange. Of all scientists, Einstein is remembered for his willingness to reject prevailing assumptions and depart from conventional thinking. He succeeded precisely because he did not follow the dominant intellectual trajectory of his time.
Yet this only pushes the question one level deeper. Where did that tendency come from? Einstein’s insights did not emerge from some transcendent source. The observations, theories, and contradictions were already there. What distinguished Einstein was that he picked up on structures and implications that most of his contemporaries either missed or ignored.
Perhaps the real question is not whether Einstein was doing autocomplete, but what selected one continuation over another. What made him follow the unlikely path?
LLMs
A current large language model is perhaps the closest thing we have built to a system that operates primarily within language, much like Einstein did. It isn’t trained by letting it experience the world. It’s trained on the accumulated record of how people have written about the world. Whatever ends up in the model’s parameters is a compressed version of that record.
Information about the world is spread across millions of pages. Training compresses that scattered record into the model’s weights, and the corpus is far larger than any human mind could absorb directly.
LLMs are enormous nonlinear functions with randomness baked in, so they aren’t confined to replaying training data. Physics repeatedly shows that nonlinear interactions among many components can produce emergent structures that are not obvious from the underlying elements — turbulence, evolution, pattern formation. As a physicist, I am not surprised that a nonlinear system of this scale, with so many interacting components, would produce structures and representations nobody has seen before.
A model can produce continuations, some insightful and some hallucinated, that almost certainly never appeared in its training text. Nonlinearity can generate novelty, but it does not necessarily generate truth. The challenge is producing representations that are both novel and useful, representations that reveal something that was not previously visible.
Einstein’s achievement was not simply generating novel representations. It was selecting the ones that survived contact with reality. Even for Einstein, that process was not automatic. He explored dead ends, pursued incorrect ideas, and discarded representations that ultimately failed. Creativity generated possibilities; judgment and experiment selected among them.
When you prompt an LLM, the structure encoded in its parameters shapes a path through the space of possibilities, and a representation comes out the other end. The answer often arrives at a conclusion the person asking did not have a moment earlier.
The ingredients from which that representation emerged were already encoded in the model’s parameters, so in the technical sense no new information was produced. Yet the specific representation that surfaces for a particular prompt did not exist until the model ran.
This is where the information-theoretic perspective misses something important. Human cognition is severely limited, which is exactly why information theory alone cannot tell us what makes a representation valuable to a human mind. Most of the time an LLM produces useful arrangements of what we already had. Whether it can produce representations that change what becomes thinkable on the scale of a scientific revolution is still an open question. Given what we know about emergence in large nonlinear systems, the possibility is difficult to dismiss.
Whether an LLM can produce something genuinely new matters less than what it does with what already exists. Reorganizing familiar information into a new representation can open entirely new horizons of understanding.