Good Search Steals?

The WIRED Analysis of the Perplexity AI: A Social Machine That Does Not Crawl the Web, But Does It Exploit Robots Exclusion Protocol?

The WIRED analysis and one carried out by Robb Knight suggest that Perplexity is able to achieve this partly through apparently ignoring a widely accepted web standard known as the Robots Exclusion Protocol to secretly scraper parts of websites that Operators do not want accessed byBots, despite claiming that it won WIRED observed a machine tied to Perplexity—more specifically, one on an Amazon server and almost certainly operated by Perplexity—doing this on wired.com and across other Condé Nast publications.

The WIRED analysis also demonstrates that despite claims that Perplexity’s tools provide “instant, reliable answers to any question with complete sources and citations included,” doing away with the need to “click on different links,” its chatbot, which is capable of accurately summarizing journalistic work with appropriate credit, is also prone to bullshitting, in the technical sense of the word.

The Perplexity is more specific. Prompted to describe what Perplexity is, it provides text that reads, “Perplexity AI is an AI-powered search engine that combines features of traditional search engines and chatbots. It provides concise, real-time answers to user queries by pulling information from recent articles and indexing the web daily.”

Web crawling—the act of indexing information across the internet—has been around for decades. It has been used by search engines and nonprofits to catalog the contents of the internet and make it accessible through a website. Until recently, the practice of web crawling has not been seen as controversial, as websites depended on the process as a way to find their content. Crawling tech is no longer used by companies like Perplexity artificial intelligence, because they use it to absorb whole articles that are fed into their machines.

The National Thoroughbred League: Hacks on Max, Randall Lane, Kate Knibbs, Boone Ashworth, and GadgetLab

The National Thoroughbred League is Randall’s new horse racing league. Kate likes the book by Boryga. Lauren recommends the show Hacks on Max.

Randall Lane can be found on social media @RandallLane. Kate Knibbs is @Knibbs. LaurenGoode is a person. Michael Calore is @snackfight. Bling the main hotline at @GadgetLab. The show is produced by Boone Ashworth (@booneashworth). Our theme music is by Solar Keys.

If you want to get every episode for free, you can use the audio player on this page.

If you have an iPad or an iPhone, you can simply tap this link to open the Podcasts app. You can also download an app like Overcast or Pocket Casts, and search for Gadget Lab. You can find us in the app if you use the phone’s browser. We’re on Spotify too. And in case you really need it, here’s the RSS feed.