A sad reality of running a website these days is that most sites gain the attention of AI crawlers and other bots. As the world has decided that generative AI, powered by scraping the Internet, stealing other people’s work, is the future, for the moment at least, it has meant that everyone wants to do it, and there are times that a website can be brought down by the sheer weight of the bots and crawlers sucking it dry for information.

This is, of course, a problem for website operators, but there are a number of solutions that can help mitigate the problem.

Your website should have a robots.txt file in the root. This is a long established standard that was developed to tell search engines what to crawl and not to crawl on a website. Most crawlers do, remarkably, honour this, so a robots.txt file with a list of bot user agents and and the simple instruction ‘Disallow’ can go a long way towards controlling what scrapes your site.

Known user agents are collected by, among others, a service called Known Agents (formerly Dark Visitors) who make their lists publicly available as well as providing a very good managed service. However, their public list is converted into a usable robots.txt file here: https://github.com/ai-robots-txt/ai.robots.txt, which can be imported into your site. There are also defence mechanisms for all major web servers that can be integrated into your site to reinforce the robots.txt instruction.

If your site is suffering from scrapers, this is just the first line in defending against them. Moortech can help you develop a strategy to defend against them ranging from simple solutions such as the above, to web application firewalls and hosted services.

Contact us to find out what we can do to help you defend against bots and scrapers.