Amazon Web Services (AWS) has initiated an investigation into Perplexity AI, a startup backed by significant investors like the Jeff Bezos family fund and Nvidia, for allegedly violating AWS rules. The issue revolves around Perplexity AI’s use of web scraping techniques on websites that have explicitly prohibited such activities through the Robots Exclusion Protocol. Although this protocol is not legally binding, terms of service are, and AWS mandates its customers to respect these protocols while crawling websites.
Perplexity AI has been accused of scraping content from various news websites, including Condé Nast, which blocks such activities through its robots.txt file. An IP address linked to Perplexity was found to have accessed Condé Nast’s servers hundreds of times in recent months. This IP address is traced back to an Elastic Compute Cloud (EC2) instance hosted on AWS. Perplexity’s CEO, Aravind Srinivas, claims that the scraping was conducted by a third-party company under a nondisclosure agreement and stressed the complexity of the issue.
The scrutiny intensified following a Forbes report accusing Perplexity of stealing content, which was confirmed by WIRED’s investigations. This has raised questions about the ethical use of AI in web scraping and the responsibilities of companies using such technologies.
