Reddit is Blocking Major Search Engines Except for Those That Pay

Reddit has begun aggressively blocking search engines, sparing only those it has commercial arrangements with, such as Google. This development, dating back to last February, follows Google's $60 million extended partnership with Reddit. Under this deal, Google can utilize Reddit data to enhance its AI models, giving Reddit content more visibility on Google Search, thereby facilitating participation in Reddit’s discussions and communities. This is prepared by SSP.

Key to this partnership is Google's access to Reddit's Data API, which provides real-time content. Consequently, non-Google search engines like Bing, DuckDuckGo, and Mojeek are now denied recent Reddit content or restricted from displaying complete search results. Microsoft spokesperson Caitlin Roulston affirmed Microsoft’s adherence to the robots.txt standard, which guides web crawlers on site access permissions. Adhering to Reddit’s updates on July 1, Bing ceased crawling Reddit’s pages.

Despite Reddit’s public and commercial maneuverings, Reddit spokesperson Tim Rathschmidt stressed that the search engine blocks were independent of the new Google partnership. According to Rathschmidt, discussions with multiple search engines fell through because some could not or would not comply with enforceable terms regarding the usage of Reddit's data, including for AI purposes.

To reinforce its anti-scraping stance, Reddit updated its robots.txt file last month to alert web crawlers without agreed terms to refrain from accessing its data. Ben Lee, Reddit's chief legal officer, explained this move signals crawlers, disallowed without an agreement, from processing Reddit data.

Blocking engines other than Google reflects Reddit’s quest to capitalize on data access and align with new investors' revenue expectations since its IPO. The change may frustrate users habituated to tagging "Reddit" for authentic human responses, especially affecting those searching through Bing.