Google has unveiled PaLM (Pathways Language Model), a new language model capable of programming, solving mathematical problems, and explaining jokes with a learning efficiency % that puts it ahead of any language models developed to date.
According to Google, PaLM was trained using the Pathways model, which allowed him to train a single model with numerous Tensor Processing Units (TPU) Pods efficiently, as noted on his blog. It’s based on ‘few shots’ learning, which cuts down on the number of instances required in task-specific training to match a particular application.
For this, a database with 780 billion tokens was created, which incorporates “a set of multilingual data” that includes web papers, books, Wikipedia, discussions, and GitHub code, in addition to English. There’s also a ‘lossless’ vocabulary that “preserves all whitespace,” which is “particularly crucial” for programming, as well as the byte division of Unicode characters not included in the vocabulary.
This new AI has 540 billion parameters, which is higher than OpenAI’s GPT-3 language model, which Google credits as a pioneer in demonstrating that these can be utilized for learning with “amazing outcomes.”
According to Google, its new language model uses 6.144 TPU v4 chips on Pathways, making it the “biggest TPU configuration” in history. PaLM also achieves a training efficiency of 57.8% when using flops in ‘hardware,’ which is “the highest obtained so far for language models at this size,” according to the researchers.
This is made feasible by a combination of “parallelism technique and transformer block reformulation,” which allows the attention and forward layers to be computed in parallel, speeding up “TPU compiler optimizations.”
“PaLM has proved novel skills in multiple and very tough tasks,” the technology firm claims, citing several examples ranging from language comprehension and generation to reasoning and programming activities.
One of the examples Google provides is asking PaLM to guess a movie based on four emojis: a robot, an insect, a plant, and the planet Earth. The AI chooses the best option out of all the possibilities (LA Confidential, Wall-E, León: the Professional, BIG, and Rush): Wall-E.
In another, he is asked to choose two terms from a list that are connected with the word’stumble,’ and he correctly chooses ‘fall’ and ‘trip.’
AI can also answer simple math problems and even explain jokes by contextualizing and breaking down the pieces that occur in them in order to make sense of them.
Finally, PaLM can program by translating code “from one language to another,” creating code based on “a natural description of the language,” and “correcting compilation faults,” according to Google.