There’s been a lot of coverage in the last few weeks about just how much AI bots have proliferated. And this doesn’t include the “nicer” bots which respect you blocking them in your website’s robots.txt (on that see our article from last year). A lot of these bots don’t respect any restrictions and so can flood your website with traffic, wasting resources and increasing your costs. This include’s Meta’s AI crawler.

For more info see this post by Gergely Orosz about how AI crawlers are wrecking the open internet. He may need to close down his side project because his bandwidth has increased 7-fold to the point where he’s paying $90/month just to train AI bots. Wikipedia is also drowning with 65% of their most expensive images to serve (rare ones that aren’t cached) are requested by bots.

If you use Cloudflare for your website, you might want to check out this new feature where they use AI to fight AI. You can enable the option for it to add some links to your website that will only be visible to bots. These are auto-generated and so AI bots will be trapped in a labyrinth of fake pages rather than consuming the resources of your real pages. This can help keep your bandwidth for your actual visitors. This is available on Cloudflare’s free plan.