New Data Shows Just How Badly OpenAI And Perplexity Are Screwing Over Publishers

ai-searchweb-scrapingpublishersreferral-trafficcopyrightperplexity

Abstraction: AI search engines send 96% less referral traffic while massively increasing scraping

Key points:

TollBit report (analyzing 160 websites, Q4 2024): AI search engines send 96% less referral traffic to news sites and blogs than traditional Google search; AI scrapers visited sites 2 million times on average per site.
AI scraping has more than doubled in recent months; each page scraped ~7 times on average; companies like OpenAI and Perplexity often use unidentified bots making it hard for publishers to block.
Chegg's traffic plummeted 49% YoY in January after Google released AI summaries; Chegg sued Google and is considering going private.
Perplexity scraped one publisher 500 times after being blocked yet sent 10,000+ referrals — evidence of covert scraping under a different user agent despite claiming robots.txt compliance.
Content licensing deals (AP, Axel Springer, Financial Times with OpenAI) and new platforms like TollBit (500 publishers, charges per scrape) are emerging as partial solutions.
Publishers face a dilemma with Google: blocking its AI bots risks harming SEO since the same bots may serve multiple purposes (indexing + AI training).

Connections: Openai · Perplexity · Google · AI Web Scraping · Copyright · AI Training Data