Meta has removed language from its commercial terms that raised concerns it could claim permission to scrape publishers’ websites and use their content to train artificial intelligence systems. The company says the revision clarifies terms for its advertising and business products, rather than changing how its web crawlers operate.
The deleted provision described broad uses for content from domains, web pages and URLs associated with business accounts. It included developing, improving, researching and training Meta products and services. That wording prompted questions about whether publishers using Meta’s business tools were granting rights over material on their own websites, beyond the content they supplied directly to Meta.
Critics interpreted the provision as potentially allowing Meta to collect website content for AI training without paying publishers, including when a site had expressed restrictions through its robots.txt file. Cornell digital and information law professor James Grimmelmann was among those who raised concerns about the language. The dispute centered on the possible reach of the contractual wording, not on evidence that Meta had begun a new scraping operation.
Meta initially rejected the interpretation that its terms represented a change in web crawling. Its vice president of communications, Andy Stone, and spokesperson Daniel Roberts both pushed back on that characterization. The company subsequently removed the disputed passage and issued a fuller explanation of what it said the terms were intended to cover.
In a follow-up statement to The Verge, Roberts said the consolidated crawling provisions applied only to Meta’s advertising and business products, and that the company had made that scope clearer in the terms. He also said website owners could continue using robots.txt to communicate their preferences and that the updates did not give Meta new rights overriding those preferences.
Robots.txt is a standard mechanism through which website operators give automated crawlers instructions about which parts of a site they may access. It is not itself a technical barrier: a server does not necessarily prevent access simply because a path is listed as restricted. Its effectiveness therefore depends on how a crawler handles the instructions. Whether a company has contractual permission to use content is a separate question from whether its software can retrieve that content.
That distinction helps explain the concern over the removed language. A publisher can use a platform’s advertising or business services while maintaining a separate website with its own access policies. Broad contractual wording can create uncertainty about whether participation in one service carries permissions affecting that separate site, particularly when training is expressly named as a possible use.
The revision narrows the immediate dispute to Meta’s stated scope for its business and advertising terms. It does not settle broader questions about copyright and AI training, which remain the subject of litigation involving Meta and other AI companies. On the specific crawling issue, Meta’s position is that its practices have not changed and that the disputed update did not override website owners’ expressed preferences.
Sources: The Verge (AI), TechCrunch AI