Google AI told me to drink & drive

Google's foray into AI-powered search has been marred by inaccuracies and inconsistencies, raising concerns about the reliability and trustworthiness of the technology.

May 25, 2024

The Facts:

Google has recently introduced a new feature called "AI Overview" within its search engine, which allows the company's artificial intelligence to directly generate answers to users' queries, rather than just providing links to relevant websites. This seems to be an attempt at replicating the success that Microsoft has recently seen with their integration of OpenAI’s GPT engine within Bing’s search functionality.

This feature of Google’s has been rolled out to users in the United States, and it represents a significant change in their search functionality. The AI Overview feature is designed to provide users with concise, AI-generated answers to their queries, particularly in the realm of health and medical information.

“At the end of the day, they're the 800-pound gorilla [Google] on this [search engine]. And I hope that with our innovation, they will definitely want to come out and show that they can dance. And I want people to know that we made them dance.”

- Satya Nadella

However, the rollout of this feature has not been without its challenges. Users have reported a number of inaccuracies and bizarre responses from the AI system, ranging from claims that dogs have played in the NFL to the assertion that President Andrew Johnson had 14 degrees from the University of Wisconsin at Madison.

These erroneous responses have not been limited to trivial or humorous queries, but have also extended to more serious health-related questions. The AI has provided advice that could be potentially harmful, such as suggesting that rocks are safe to eat or that chicken is safe to consume at a temperature of only 102 degrees Fahrenheit.

While Google has stated that it has an "even higher bar for quality" when it comes to health-related queries, the examples of incorrect or misleading information that have surfaced suggest that the company may be struggling to maintain that high standard.

The inconsistency of the AI's responses has also been a concern, with some health-related queries triggering an AI Overview, while others do not. For instance, the tool refused to provide information on the symptoms of breast cancer, but then later offered details on the symptoms of lung and prostate cancer. This inconsistency raises questions about the reliability and trustworthiness of the AI system, especially in a domain as sensitive as healthcare.

Furthermore, the potential for real-world harm is a significant issue when it comes to the AI Overview feature. As an AI research scientist tweeted, the concern is not just about "gotchas" or obvious errors, but about "clearly foreseeable harms" that could result from even minor inaccuracies or flaws in the AI's reasoning. For example, an AI-generated response that misses the signs of an allergic reaction could have serious consequences for a user seeking immediate medical advice.

The View:

Google's decision to introduce the AI Overview feature, which allows its artificial intelligence to directly generate answers to user queries, is a bold and ambitious move that underscores the company's commitment to leveraging advanced technologies to enhance the search experience.

However, the rollout of this feature has been marred by a series of concerning issues, which raise questions about the reliability and trustworthiness of the AI system, particularly when it comes to sensitive domains like healthcare. The examples of inaccurate and bizarre responses generated by the AI Overview feature are troubling, as they demonstrate the challenges that tech giants like Google face in ensuring the quality and safety of their AI-powered tools.

While the company has stated that it has a higher standard for health-related queries, the reported instances of incorrect or misleading information suggest that the system may not yet be up to the task of providing reliable medical advice. The inconsistency in the AI's responses further compounds the issue, as it undermines the user's confidence in the tool's abilities.

The fact that the AI may refuse to answer certain health-related queries, only to then provide information on similar topics, highlights the need for more robust and reliable systems that can consistently deliver accurate and trustworthy information.

Ultimately, the rollout of Google's AI Overview feature serves as a cautionary tale about the perils of rushing to deploy AI-powered tools, particularly in sensitive domains where the potential for real-world harm is significant.

As Gary Marcus, a neural science professor at NYU noted, the final 20% of accuracy required to achieve truly reliable and trustworthy AI systems may be the most challenging to attain, requiring advanced reasoning and fact-checking capabilities that current language models may not possess.

Google's reputation and leadership in the search industry are undoubtedly at stake as it navigates the challenges posed by the AI Overview feature.

The company must demonstrate a commitment to rigorous testing, robust content moderation, and a willingness to learn from its mistakes if it hopes to regain the trust of users and maintain its position as a trusted source of information in the rapidly evolving landscape of AI-powered search.

Lastly, until and unless AI models are capable of fact-checking themselves, it seems as though it’ll be a while before we can expect to see any valuable suggestions to complex queries.

TLDR:

Google has introduced an "AI Overview" feature that allows its AI to directly generate answers to user queries, rather than just providing links.
The rollout of this feature has been marked by a number of inaccuracies and bizarre responses from the AI system, ranging from trivial claims to potentially harmful medical advice.
The inconsistency in the AI's responses, with some health-related queries triggering the AI Overview while others do not, further compounds the issue of reliability and trustworthiness.
The potential for real-world harm is a significant concern, as even minor inaccuracies or flaws in the AI's reasoning could have serious consequences, especially in sensitive domains like healthcare.
Google has stated that it has a higher standard for health-related queries, but the reported instances of incorrect or misleading information suggest that the system may not yet be up to the task of providing reliable medical advice.
The rollout of Google's AI Overview feature serves as a cautionary tale about the perils of rushing to deploy AI-powered tools, particularly in sensitive domains where the potential for real-world harm is significant.
Google must demonstrate a commitment to rigorous testing, robust content moderation, and a willingness to learn from its mistakes if it hopes to regain the trust of users and maintain its position as a trusted source of information in the rapidly evolving landscape of AI-powered search.

Insights From:

Google Is Playing a Dangerous Game With AI Search - The Atlantic

Google scrambles to manually remove weird AI answers in search - The Verge