Google Ai Overviews Cloudflare robots.txt news publishers

Robots.txt got some much-needed TLC last week, courtesy of Cloudflare’s latest update. Cloudflare’s new Content Signals Policy effectively upgrades the decades-old honor system and adds a way for publishers to spell out how they do (and perhaps more importantly – how they don’t – want AI crawlers to use their content once it’s scraped.) For

Read the full article

Published by digiday.com