Everything that’s swept the world since the 2022 release of ChatGPT — “the AI economy”; data centers; chatbots acting as best friends, doctors, and mathematicians; talk of existential risk — it all starts, in some sense, with an intern in a University of Montreal lab named Dzmitry Bahdanau. The 23-year-old Belarusian, who had been coding competitively since he was a kid but hadn’t done academic research before, was asked to help figure out how to get a computer to translate sentences from English to French. Bahdanau was working with artificial neural networks, which had already shown promise in pattern-recognition tasks such as deciphering handwriting and identifying objects in photos.
The networks learn to recognize patterns by strengthening and weakening connections between nodes in response to training data. Bahdanau quietly doubted their potential to achieve anything like human-level intelligence, but his adviser, the computer scientist Yoshua Bengio, and the postdoc on the project, Kyunghyun Cho, were believers. And translation was widely seen as a formidable and interesting problem for neural networks to try to solve.
Translation is not as simple as swapping each English word for a French one. Languages have different grammatical structures and are riddled with idioms and connotations, all of which had long stymied attempts to automate the process. You can’t simply implement rules of translation because there are none.
But neural networks seemed promising because they learn by example. They can be trained on huge databases of paired English and French sentences to, eventually, absorb the quasi-rules, exceptions, and subtleties. Results had been mediocre, though; the longer the sentence, the worse neural networks performed.
That’s because researchers were trying to cram each sentence’s data into a fixed amount of memory before decoding the sentence in the target language. It was a bottleneck that limited how complex and sophisticated translations could be. The network architecture just wasn’t right.
So Bahdanau tried something different. From the very first attempt.” Bahdanau’s trick, a mathematical operation that he, Cho, and Bengio referred to as “attention,” enabled their neural network to translate long sentences just as well as short ones. That meant the approach was scalable.
Extract — continue reading at the source.
