Google Adds Multimodal Capability To AI Mode With Lens Search

Google Lens visual search illustration. (photo: Fok. google)

JAKARTA - Google is rolling out multimodal capabilities on the AI Mode feature that is integrated with Google Lens' visual search. This feature allows users to take or upload images, ask questions related to the image, and receive context-rich answers with links relevant for further exploration. In addition, AI Mode is now available to millions of Google Labs users for free in the US.

Last month, Google released a dedicated AI Mode experimental feature for Premium Google One AI customers. Now, the search engine giant is expanding access to millions of Google Labs users in the United States.

Not only that, Google is also improving AI Mode capabilities by adding multimodal features, which means users can now upload images or take photos, then immediately ask about the contents of the image and receive an in-depth explanation of Google's search results.

Google says that this multimodal search feature is supported by Google Lens and a special version of the AI Gemini model. Now, this system can understand the entire context of an image, including the relationship between objects, materials, colors, shapes, and arrays in the image.

With Google expertise in visual search, Lens is able to accurately identify each object in the photo. This technology uses an approach called fan-out query, where AI conducts various additional searches to provide more insight.

More Than Just Ordinary Visual Search

An example shared by Google shows AI Mode can recognize all books in bookcase photos, then seek information about these titles and suggest similar books that receive high reviews. The results are not only in the form of a list of books, but also links to buy them as well as additional details.

Users can also ask further questions, such as: "I want books to be read quickly. Which of these recommendations is the shortest?"

This feature is considered Google's response to AI-based search services such as Perplexity and ChatGPT Search, which provide a summary of answers from the search index. Google plans to continue to improve its search experience and expand its AI Mode feature in the months to come.