Searching for information on Google today means encountering AI Overviews, the Gemini-powered search robot that appears at the top of the results page. AI Overviews has faced a lot of trouble since its launch in 2024, angering users due to its scattershot accuracy, but it is getting better and usually provides the correct answers. However, this is a low level. A new analysis from The New York Times attempted to assess the accuracy of AI observations, finding that it is correct 90 percent of the time. The flip side is that 1 in 10 AI answers is wrong, and for Google, that means hundreds of thousands of lies being spread every minute of the day.
The Times conducted this analysis with the help of a startup called Oumi, which is deeply involved in developing AI models itself. The company used AI tools to test AI overviews with the SimpleQA assessment, a common test to rank the factuality of generic models like Gemini. Released by OpenAI in 2024, SimpleQA is essentially a list of over 4,000 questions with verifiable answers that can be fed into an AI.
Omi started its testing last year when the Gemini 2.5 was still the company’s best model. At the time, the benchmark showed an 85 percent accuracy rate. When the test was rerun after the Gemini 3 update, AI Overviews answered 91 percent of the questions correctly. If you add this miss rate to all Google searches, AI observation is generating millions of wrong answers per day.
The report includes several examples where AI observations went wrong. When asked about the date when Bob Marley’s former home became a museum, AI Overviews cited three pages, two of which did not discuss the date at all. The last year, Wikipedia, listed two conflicting years, and AI overviews confidently chose the wrong year. The benchmark also prompts models to produce the date on which Yo Yo Ma was inducted into the Classical Music Hall of Fame. While AI Overviews cited the organization’s website which listed Ma’s induction, it claimed that there is no such thing as a Classical Music Hall of Fame.
<a href