New research from Brown University has shown that language models are capable of forming representations of the plausibility of events—despite being trained on contradictory data.
Chatbots are trained on vast amounts of text, where facts, errors, and absurdities are mixed, but new research from Brown University has shown that language models can still develop an understanding close to that of the real world.
Researchers analyzed the internal states of models (GPT-2, Llama 3.2, and Gemma 2) when processing ordinary, unlikely, impossible, and nonsensical events. In larger models (with over 2 billion parameters), distinct vectors for different categories of plausibility formed, allowing them to distinguish even unlikely and impossible events with about 85% accuracy.
The models also reflected human uncertainty in ambiguous cases: if among people there were 50% supporters of each category, the models assigned approximately 50% probability to each option. This indicates that AI is not merely predicting the next word but encoding something akin to causal relationships of the real world.
Although the work does not prove that AI understands the world in the same way humans do, it suggests that something more structured is forming within the statistical mechanisms.
This means that behind the external 'guessing of words' lie more complex processes: AI is gradually building internal models of reality that help it navigate the logic of events—even if it does not perceive them as humans do, writes bb.lv.
Leave a comment