Google Faces Backlash Over Use of Same Crawler for Search Indexing and AI Results

The tension between Google and content creators is still simmering, with publishers accusing the tech giant of "stealing" content to fuel its AI ambitions while simultaneously eroding the very traffic that sustains them. At the center of the dispute is Google's web crawler, which publishers say the company refuses to separate for its AI and search functions.

Neil Vogel, CEO of People, Inc. (formerly Dotdash Meredith), one of the largest digital publishers in the United States, publicly branded Google an "intentional bad actor" for this practice. Speaking at a recent conference, Vogel argued that the company's use of a single bot, Googlebot, for both search indexing and AI content harvesting creates an unfair and unethical dynamic.

"They have one crawler, which means they use the same crawler for their search, where they still send us traffic, as they do for their AI products, where they steal our content," Vogel stated.

The AI existential threat

For decades, the relationship between Google and publishers was a symbiotic, if uneasy, one. Publishers provided the content that made Google Search useful, and in return, Google sent back a flood of valuable web traffic.

However, with the rise of generative AI, that fundamental bargain is breaking down. Publishers, including People, Inc., are now exploring ways to block AI crawlers from scraping their content without permission.

Cloudflare has even developed tools that allow publishers to block non-paying AI bots. Vogel revealed that adopting the solution has already prompted interest from other major AI providers for potential content licensing deals.

The problem, he noted, is that Google’s unified bot cannot be selectively blocked. If a publisher uses their robots.txt file to block Google's AI from scraping their content, they also risk cutting off their search traffic.

Declining search traffic

Vogel's concerns are backed by a stark reality: the share of traffic from Google Search to publisher sites has plummeted in recent years. Vogel revealed that for People, Inc., search traffic has fallen from as much as 65% to the "high 20s."

This decline is exacerbated by Google's own "AI Overviews" and other generative features that provide users with summarized answers directly on the search results page, removing the need to click through to the original source.

While some publishers are pursuing legal action against AI companies, the core issue with Google remains. The company's refusal to split its crawler places publishers in an impossible position: either continue to provide free content for Google's AI models or sacrifice their remaining search visibility.

The standoff highlights the existential crisis facing digital media. As AI models become more adept at synthesizing information, they risk disintermediating the very creators they rely on.

Publishers are now fighting for a new business model, one where they are fairly compensated for the content that fuels the AI economy, rather than being treated as an open, free resource.