In a disturbing revelation, it has been discovered that Perplexity, a generative AI search engine, is ignoring the instructions in the robots.txt file, which is meant to control bots and crawlers like itself. This means that Perplexity is accessing websites that administrators have explicitly prohibited it from visiting. This is a serious breach of trust, as it undermines the control that website administrators have over their own content.
The issue came to light when Rob Knight, a technology blogger, blocked PerplexityBot, the crawler used by Perplexity, in the robots.txt of his blog. However, when he tested the block, he found that Perplexity was still able to access and summarize his blog post. Further investigation revealed that PerplexityBot was using a headless browser to scrape content, ignoring the robots.txt file altogether. What’s more, Perplexity’s user agent string did not contain the ‘PerplexityBot’ part, which allowed it to bypass the robots.txt restrictions.
This issue has sparked a heated debate, with many pointing out the negative implications of generative AI search engines like Perplexity crawling websites without permission. Not only does it undermine website administrators’ control over their content, but it also raises concerns about the unauthorized use of internet data to train generative AI. As one user on the social news site Hacker News pointed out, “forcing users to block crawlers by AI development companies could have a negative impact on ad blockers and other useful software.” It remains to be seen how Perplexity will respond to these allegations and whether they will take steps to respect website administrators’ wishes.
