Navigating the Digital Maze: User-agent: GPTBot Disallow: /

In early March of this year I started talking about the importance of implementing “robots.txt” and the “nofollow” attributes because the great AI transcoding was in full effect. The time has clearly come to use code as a explicit statement around IP it looks like this : User-agent: GPTBot Disallow: / A Verge piece looks at the NYT implementing this. The New York Times blocks OpenAI’s web crawler / The NYT’s robot.txt page that controls how it appears to automated bots built to index the internet now specifically disallows OpenAI’s GPTBot.