In the world of artificial intelligence chatbots, the common mantra is “the bigger, the better.”
Large language models such as ChatGPT and Bard, renowned for generating authentic, interactive text, progressively enhance their capabilities as they ingest more data. Daily, online pundits illustrate how recent developments – an app for article summaries, AI-driven podcasts, or a specialized model proficient in professional basketball questions – stand to revolutionize our world.
However, developing such advanced AI demands a level of computational prowess only a handful of companies, including Google, Meta, OpenAI, and Microsoft, can provide. This prompts concern that these tech giants could potentially monopolize control over this potent technology.
Further, larger language models present the challenge of transparency. Often termed “black boxes” even by their creators, these systems are complicated to decipher. This lack of clarity combined with the fear of misalignment between AI’s objectives and our own needs, casts a shadow over the “bigger is better” notion, underscoring it as not just obscure but exclusive.
In response to this situation, a group of burgeoning academics from the natural language processing domain of AI – responsible for linguistic comprehension – initiated a challenge in January to reassess this trend. The challenge urged teams to construct effective language models utilizing data sets that are less than one-ten-thousandth of the size employed by the top-tier large language models. This mini-model endeavor, aptly named the BabyLM Challenge, aims to generate a system nearly as competent as its large-scale counterparts but significantly smaller, more user-friendly, and better synchronized with human interaction.
Aaron Mueller, a computer scientist at Johns Hopkins University and one of BabyLM’s organizers, emphasized, “We’re encouraging people to prioritize efficiency and build systems that can be utilized by a broader audience.”
Alex Warstadt, another organizer and computer scientist at ETH Zurich, expressed that the challenge redirects attention towards human language learning, instead of just focusing on model size.
Large language models are neural networks designed to predict the upcoming word in a given sentence or phrase. Trained on an extensive corpus of words collected from transcripts, websites, novels, and newspapers, they make educated guesses and self-correct based on their proximity to the correct answer.
The constant repetition of this process enables the model to create networks of word relationships. Generally, the larger the training dataset, the better the model performs, as every phrase provides the model with context, resulting in a more intricate understanding of each word’s implications. To illustrate, OpenAI’s GPT-3, launched in 2020, was trained on 200 billion words, while DeepMind’s Chinchilla, released in 2022, was trained on a staggering trillion words.
Ethan Wilcox, a linguist at ETH Zurich, proposed a thought-provoking question: Could these AI language models aid our understanding of human language acquisition?
Traditional theories, like Noam Chomsky’s influential nativism, argue that humans acquire language quickly and effectively due to an inherent comprehension of linguistic rules. However, language models also learn quickly, seemingly without this innate understanding, suggesting that these established theories may need to be reevaluated.
Wilcox admits, though, that language models and humans learn in fundamentally different ways. Humans are socially engaged beings with tactile experiences, exposed to various spoken words and syntaxes not typically found in written form. This difference means that a computer trained on a myriad of written words can only offer limited insights into our own linguistic abilities.
However, if a language model were trained only on the vocabulary a young human encounters, it might interact with language in a way that could shed light on our own cognitive abilities.
With this in mind, Wilcox, Mueller, Warstadt, and a team of colleagues launched the BabyLM Challenge, aiming to inch language models towards a more human-like understanding. They invited teams to train models on roughly the same amount of words a 13-year-old human encounters – around 100 million. These models would be evaluated on their ability to generate and grasp language nuances.
Eva Portelance, a linguist at McGill University, views the challenge as a pivot from the escalating race for bigger language models towards more accessible, intuitive AI.
Large industry labs have also acknowledged the potential of this approach. Sam Altman, the CEO of OpenAI, recently stated that simply increasing the size of language models wouldn’t yield the same level of progress seen in recent years. Tech giants like Google and Meta have also been researching more efficient language models, taking cues from human cognitive structures. After all, a model that can generate meaningful language with less training data could potentially scale up too.
Despite the commercial potential of a successful BabyLM, the challenge’s organizers emphasize that their goals are primarily academic. And instead of a monetary prize, the reward lies in the intellectual accomplishment. As Wilcox puts it, the prize is “Just pride.”
