Towards a Large Language Model: The 500-Language Model and its Impact on Google Search, Wordcraft, and Test Kitchen

The 1,000 Language model was announced at the showcase. The company also shared new research on text-to-video models, a prototype AI writing assistant named Wordcraft, and an update to its AI Test Kitchen app, which gives users limited access to under-development AI models like its text-to-image model Imagen.

Language models are already integrated into products like Google Search, while fending of criticism about the systems. There are flaws to language models which include a tendency to regurgitate harmful societal biases, and a lack of ability to translate language with human sensitivity. The researchers were fired for publishing papers detailing these problems.

Speaking to The Verge, Zoubin Ghahramani, vice president of research at Google AI, said the company believes that creating a model of this size will make it easier to bring various AI functionalities to languages that are poorly represented in online spaces and AI training datasets (also known as “low-resource languages”).

The model is exposed to and trained in many languages, which makes it much better for the low resource languages. “The way we get to 1,000 languages is not by building 1,000 different models. Languages are like organisms, they’ve evolved from one another and they have certain similarities. Zero-shot learning can be improved with the use of data from a new language in the model, and the ability to translate what it learns from a high-resource language to a low-resource language.

The access to data is a problem for training across many languages, and in order to support the work on the 1,000-language model, it will be funding the collection of data for low-resource languages.

The company does not have any plans to apply the model at this moment, only that it will have a wide range of uses across the products.

“One of the really interesting things about large language models and language research in general is that they can do lots and lots of different tasks,” says Ghahramani. “The same language model can turn commands for a robot into code; it can solve maths problems; it can do translation. The really interesting things about language models is they’re becoming repositories of a lot of knowledge, and by probing them in different ways you can get to different bits of useful functionality.”

The Google seizure faux pas makes sense given that one of the known vulnerabilities of LLMs is the failure to handle negation. A few years ago, Andrew Ettinger demonstrated this with a simple study. When asked to complete a short sentence, the model would answer 100% correctly for affirmative statements (ie. The statements are incorrect for negative statements. “a robin is not…”). It became evident that the models could not differentiate between the scenarios because they provided the exact same responses. This remains  an issue with models today, and  is  one of the rare linguistic skills models do not improve at as they increase in size and complexity. Such errors reflect wider concerns raised by linguists on the effectiveness of such artificial language models, and how much of the English language may look like gibberish without any linguistic capability to show it.

Additionally, the creators of such models confess to the difficulty of addressing inappropriate responses that “do not accurately reflect the contents of authoritative external sources”. The benefits of eating crushed glass and how crushed porcelain added to breast milk can support the infant’s ileum are some of the things that will be written about in a paper. In fact, Stack Overflow had to temporarily ban the use of ChatGPT- generated answers as it became evident that the LLM generates convincingly wrong answers to coding questions.

Yet, in response to this work, there are ongoing asymmetries of blame and praise. Model builders and tech advocates alike think that the output of a mythically autonomously produced model is a technological marvel. The human decision-making involved in model development is erased, and model feats are observed as independent of the design and implementation choices of its engineers. But without naming and recognizing the engineering choices that contribute to the outcomes of these models, it becomes almost impossible to acknowledge the related responsibilities. As a result, both functional failures and discriminatory outcomes are also framed as devoid of  engineering choices – blamed on society at large or supposedly “naturally occurring” datasets, factors  those developing these models will claim they have little control over. But it’s undeniable they do have control, and that none of the models we are seeing now are inevitable. It is possible for different choices to have been made and a completely different model to be developed and released.

According to a report from CNBC, Alphabet CEO Sundar Pichai and Google’s head of AI Jeff Dean addressed the rise of ChatGPT in a recent all-hands meeting. One employee asked if the launch of the bot — built by OpenAI, a company with deep ties to Google rival Microsoft — represented a “missed opportunity” for the search giant. According to reports, Dean said that the company had to move less conservatively than a small startup because of the reputational risk posed by the technology.

Openai, too, was previously cautious in developing its technology but changed it’s stance with the launch of ChatG PPT, giving access to the public. The company eats a lot of money keeping the system free-to-use, but that has led to a storm of publicity and hype.

In 2023, we will see Codex and other large AI models used to create new “copilots” for other types of intellectual labor. The applications are potentially endless, limited only by one’s ability to imagine scenarios in which such productivity-assisting software could be applied to other types of complex, cognitive work—whether that be editing videos, writing scripts, designing new molecules for medicines, or creating manufacturing recipes from 3D models.

Historically, computer programming has been all about translation: Humans must learn the language of machines to communicate with them. Codex gives us the ability to use natural language and the machine will translate it into code. The human imagination and a piece of software with an access point can be communicated in this way.

By applying the same underlying technology used to create GitHub Copilot, it will be possible to build Copilots for virtually any complex, repetitive aspect of knowledge work, allowing knowledge workers to spend their time on higher-order cognitive tasks, and effectively transforming how a great many of us interact with technology to get things done.

In and of itself, that’s a truly remarkable step forward in productivity for developers alone, a community of knowledge workers who are wrestling with extraordinary complexity and unprecedented demand for their talents. It is the first step in a series of steps that will take place in the year 2023.

Our increasingly complicated and information- dense world requires more knowledge work every year, increasing the demands on those workers in every field. Productivity gains have been very little since the invention of the personal computer and the internet, and copilots for everything could be a genuine revolution for types of work.

Transformers for Natural Language Understanding: Challenges, Opportunities, and Challenges in Scaling Up Artificial Intelligence Neural Networks (Extended Abstract)

Artificial intelligence has promised much, but there has been something holding it back from being used successfully by billions of people: a frustrating struggle for humans and machines to understand one another in natural language.

Transformers are neural networks designed to model sequential data and generate a prediction of what should come next in a series. Core to their success is the idea of “attention,” which allows the transformer to “attend” to the most salient features of an input rather than trying to process everything.

These new models have delivered significant improvements to applications that use natural language like language translation and summarization. In the past, each required bespoke architectures. Now transformers are delivering state-of-the-art results across the board.

Although Google pioneered transformer architecture, OpenAI became the first to demonstrate its power at scale, in 2020, with the launch of GPT-3 (Generative Pre-Trained Transformer 3). The largest language model ever created was created at that time.

Parameter counts are a rough proxy for a model’s capabilities. So far, we’ve seen models perform better on a wide range of tasks as the parameter count scales up. Models have been growing by almost an order of magnitude every year for the past five years, so it’s no surprise that the results have been impressive. However, these very large models are expensive to serve in production.

Source: https://www.wired.com/story/artificial-intelligence-neural-networks/

Language in the 21st Century: Predictions for Google, Bing, and Other Next-Generation Technologies and Opportunities for Silicon Valley

The area I’m most excited about is language. Humans have had to input their thoughts using interfaces designed for technology, not humans. With this wave of breakthroughs, in 2023 we will start chatting with machines in our language—instantly and comprehensively. Eventually, we will be able to converse with all of our devices. This promises to fundamentally redefine human-machine interaction.

Since the 1980’s, we have been teaching people how to code and teach the language of computers. That will be important. We will flip the script in a few years, and computers will speak our language. There are a lot of tools for creativity, learning, and playing.

On February 8st, artificial intelligence integrations will be announced for the company’s search engine. It is free to watch live on the internet.

Google commanded the online search business for years, while Microsoft’s Bing remained a distant competitor. Microsoft plans to add Artificial Intelligence to its search engine in a bid to differentiate it from its competitors. Will this be a good year for Bing? More text is likely to be created by artificial intelligence as users navigate through their search engine of choice.

If you are familiar with text and images, they are only the starting point. More and more information is being shared by Google about its research into the possibilities for audio and video. Plenty of startups in Silicon Valley are also vying for attention (and investment windfalls) as more mainstream uses for large language models emerge.

What new discoveries from the James Webb Space Telescope can we tell my 9 year old about? Check out Bard, Bing and Baidu in the new search wars

“This highlights the importance of a rigorous testing process, something that we’re kicking off this week with our Trusted Tester program,” a Google spokesperson told CNN in a statement Wednesday about the factual error. We will combine external feedback and internal testing to make sure Bard’s responses meet a high bar for quality, safety and groundedness.

In the demo, which was posted by Google on Twitter, a user asks Bard: “What new discoveries from the James Webb Space Telescope can I tell my 9 year old about?” Bard responds with a series of bullet points, including one that reads: “JWST took the very first pictures of a planet outside of our own solar system.”

According to NASA, however, the first image showing an exoplanet – or any planet beyond our solar system – was actually taken by the European Southern Observatory’s Very Large Telescope nearly two decades ago, in 2004.

The inaccurate response from Bard was the reason why Alphabet’s shares fell as much as 8%.

In the presentation Wednesday, a Google executive teased plans to use this technology to offer more complex and conversational responses to queries, including providing bullet points ticking off the best times of year to see various constellations and also offering pros and cons for buying an electric vehicle.

In case you’ve been living in outer space for the past few months, you’ll know that people are losing their minds over ChatGPT’s ability to answer questions in strikingly coherent and seemingly insightful and creative ways. Want to understand quantum computing? Need a recipe for whatever’s in the fridge? Can’t be bothered to write that high school essay? ChatGPT has your back.

The all-new Bing is similarly chatty. Demos that the company gave at its headquarters in Redmond, and a quick test drive by WIRED’s Aarian Marshall, who attended the event, show that it can effortlessly generate a vacation itinerary, summarize the key points of product reviews, and answer tricky questions, like whether an item of furniture will fit in a particular car. It is not as bad as Microsoft’s useless Office assistant, which may bother readers whenever they created a new document.

In the newai search wars, China’s biggest search company is Baidu. It joined the war by announcing another competitor, or “Enie Bot,” in English. Baidu says it will release the bot after completing internal testing this March.

There are already huge resources used in internet searches, but the need for a different kind of firepower is what requires the inclusion of artificial intelligence. It requires processing power, as well as storage and efficient search. Every time we see a step change in online processing, we see significant increases in the power and cooling resources required by large processing centres. I think this could be such a step.”

There is a lot of computation required to train these models, according to Carlos Gmez-Rodrguez. Only the Big Tech companies can train them.

While neither of the companies have said what their computing costs are, third-party analysis concludes that the training of GPT 3 used more than a ton of electricity and led to more than a ton of carbon emissions.

The thing is not that bad, but you have to consider it. It’s more than just a matter of training, it’s also the ability to serve millions of users and execute it.

There is a big difference between integrating the chat product into Bing and using it as a stand-alone product.

It will have to change in order to meet the requirements of search engine users. “If they’re going to retrain the model often and add more parameters and stuff, it’s a totally different scale of things,” he says.