What’s New in the World of Android, What Does It Mean to Be? Speaking on Android at Google Meet and Talking with David Burke and Sameer Samat

Gemini is Google’s AI assistant that is in many ways eclipsing Google Assistant. Really—when you fire up Google Assistant on most Android phones these days, there’s an option to replace it with Gemini instead. So naturally, I asked Burke and Samat whether this meant Assistant was heading to the Google Graveyard.

Users will soon be able to ask questions about videos on-screen, and it will answer with an automatic caption. For paid Gemini Advanced users, it can also ingest PDFs and offer information. There are a lot of updates coming over the next few months.

There is a catch if you are looking at a PDF, and that is that you need to access the paid version of Google’s paid version, Gemini Advanced. That’s because the feature ingests the entire PDF, so it requires the long context window available to Gemini Advanced subscribers. You can use the PDF as an expert on whatever topic it is you are interested in, for instance, your dishwasher owner’s manual. Gemini Advanced is part of the $20 per month Google One AI Premium plan.

The Now on Tap feature was first showcased in the year 2008 and allows users to tap and hold the home button to access contextual information related to things on the screen. Talking about a movie with a friend over text? Now on Tap could get you details about the title without having to leave the messaging app. If you’re looking for a restaurant, you can look at it on yelp. The phone could surface OpenTable recommendations with just a tap.

“I think what’s exciting is we now have the technology to build really exciting assistants,” Dave Burke, vice president of engineering on Android, tells me over a Google Meet video call. I don’t think we have the technology anymore to make a computer that understands what it sees, since we did not have it back then. We do now.

I got a chance to speak with Burke and Sameer Samat, president of the Android ecosystem at Google, about what’s new in the world of Android, the company’s new AI assistant Gemini, and what it all holds for the future of the OS. The update is a chance to rethink what the phone can do and how the rest of the operating system works.

Circle to Search is the new way of approaching Search on mobile. Circle to Search is more interactive than typing into a search box and similar to the experience of Now on Tap. (You literally circle what you want to search on the screen.) It is fun to use and it skews younger as well because of it.

Samat made it clear Gemini wasn’t just providing answers but was showing students how to solve the problems. More complex problems like diagrams and graphs will be solved by Circle to Search later this year. This is powered by the learning models from the internet giant, LearnLM.

Google will roll out “AI Overviews” — formerly known as “Search Generative Experience,” a mouthful — to everyone in the US this week. Now, a “specialized” Gemini model will design and populate results pages with summarized answers from the web (similar to what you see in AI search tools like Perplexity or Arc Search).

Google Lens already lets you search for something based on images, but now Google’s taking things a step further with the ability to search with a video. That means when you take a video of something you want to look up, ask a question, and get relevant answers from the internet, you will be helped by the Artificial Intelligence of Google.

A new feature on the internet giant is expected to benefit just about anyone who has looked at photos for years. “Ask Photos” lets Gemini pore over your Google Photos library in response to your questions, and the feature goes beyond just pulling up pictures of dogs and cats. The CEO asked the person what his license plate number was. He had to get a picture of the number and the response so that he could make sure that was correct.

Google’s Project Astra is a multimodal AI assistant that the company hopes will become a do-everything virtual assistant that can watch and understand what it sees through your device’s camera, remember where your things are, and do things for you. It’s powering many of the most impressive demos from I/O this year, and the company’s aim for it is to be an honest-to-goodness AI agent that can’t just talk to you but also actually do things on your behalf.

Google’s answer to OpenAI’s Sora is a new generative AI model that can output 1080p video based on text, image, and video-based prompts. There are many different styles of videos that can be produced, like aerial shots or timelapses. Veo is offered to creators for use in YouTube videos, but the company is also trying to get it used in films.

Gems: A Chatbot Creator with AI-Generated Content and SynthID Embedded Video Detection for Android

There is a new chatbot creator called Gems that is on the way from Google. Users have the option to give instructions to a program, just like the GPTs from OpenAI. You will be able to do that if you’re a Gemini Advanced subscriber.

If you’re on an Android phone or tablet, you can now circle a math problem on your screen and get help solving it. It won’t solve the problem for you but it will make it easier for you to complete it, so students wont cheat on their homework.

It will be possible with the help of artificial intelligence on your phone to look for red flags, like conversations that are suspicious, and then give you a real-time warning. The company promises to offer more details on the feature later in the year.

Google announced that it’s adding Gemini Nano, the lightweight version of its Gemini model, to Chrome on desktop. The built-in assistant will be able to help you generate text for social media posts, product reviews and more from within the browser.

In order to detect AI-generated videos, SynthID will be embedded into the content created with the Veo video generator, according to the company.