Amazon quietly blocks AI bots from Meta, Google, Huawei and more
Amazon is escalating efforts to keep artificial intelligence companies from scraping its e-commerce data, as the retail giant recently added six more AI-related crawlers to its publicly available robots.txt file.
The change was first spotted by Juozas Kaziukėnas, an independent analyst, who noted that the updated code underlying Amazon’s sprawling website now includes language that prohibits bots from Meta, Google, Huawei, Mistral and others.
“Amazon is desperately trying to stop AI companies from training models on its data,” Kaziukėnas wrote in a LinkedIn post on Thursday. “I think it is too late to stop AI training — Amazon’s data is already in the datasets ChatGPT and others are using. But Amazon is definitely not interested in helping anyone build the future of AI shopping. If that is indeed the future, Amazon wants to build it itself.”
The update builds on earlier restrictions Amazon added at least a month ago targeting crawlers from Anthropic’s Claude, Perplexity and Google’s Project Mariner agents, The Information reported. Robots.txt files are a standard tool that websites use to give instructions to automated crawlers like search engines. While restrictions outlined in robots.txt files are advisory rather than enforceable, they act as signposts for automated systems — that is, if the crawlers are “well-behaved,” they are expected to respect the block, according to Kaziukėnas.
Amazon did not respond to a request for comment by press time.
The move highlights Amazon’s increasingly aggressive stance toward third-party AI tools that could scrape its product pages, monitor prices or even attempt automated purchases. For Amazon, the stakes are significant. Its online marketplace is not only the largest store of e-commerce data in the world but also the foundation of a $56 billion advertising business built around shoppers browsing its site. Allowing outside AI tools to surface products directly to users could bypass Amazon’s storefront, undermining both traffic and ad revenue.
Amazon’s changes come shortly after Shopify, the e-commerce technology provider, introduced a warning to the robots.txt file of its merchants’ sites, including Brooklinen, Alo Yoga and Allbirds, Modern Retail scooped in July. Shopify’s “Robot & agent policy” requires that “buy-for-me” agents include a human review step and directs developers to integrate Shopify’s checkout technology into their tools. Rather than naming particular AI companies, Shopify’s policy applies broadly to automated agents. Shopify has partnered with Perplexity and is reportedly preparing to integrate with OpenAI to enable transactions through AI chatbots, per The Financial Times.
By comparison, Amazon appears to be taking a more stringent approach. Rather than accommodating outside AI firms, it has moved to keep them at arm’s length as it builds out its own in-house tools like Rufus, a shopping chatbot now being tested with advertising features, Adweek reported last year. As Modern Retail previously reported, Amazon is also testing its own “buy-for-me” feature that can purchase items from third-party websites for customers. The implication is that Amazon would rather control how AI is used for shopping on its website.
Amazon’s stance is notable given that major retailers like Walmart and eBay have not made any changes blocking AI bots from their sites, according to a review of their robots.txt files conducted by Modern Retail.
Still, the question remains whether such restrictions can hold. Much of Amazon’s catalog has already been scraped into existing AI training datasets, and robots.txt files rely on voluntary compliance. As Kaziukėnas put it, “Amazon is a treasure trove of e-commerce data. It is notable that Amazon seems to be the only one actively fighting this.”