This is going to be a long post, so here's a short version:

tl:dr: Bots are bad. I wrote a plugin.

I noticed already a long time ago that the visitors statistics I am getting from tools like DokuWiki's own Statistics plugin, or an external web analytics tool like Matomo, differ by a very large margin from the low-level pageload numbers I get from my hosting provider's backend interface. I didn't think much of it at first – after all, what I have seen in Google Search Console is somewhat in line with the numbers in statistics, if take into account that Google is of course not the only source of traffic to my sites. So far so good.

At the time I switched from the Statistics plugin to Matomo (developing my own Nu Matomo Plugin to integrate it with DW), I however had to dig a bit deeper into this issue. Where does this ghost traffic that never shows up in the statistics actually come from?

I decided to log the traffic in more detail using a custom-made plugin. The way this worked is that it writes every page load to a dedicated log on the server itself (not relying on JavaScript). Then also write another log using JavaScript, and finally comparing these logs to see how many visitors are never actually executing any JS code. I then deployed this to several of my wikis (I run a couple of them) to see what happens. Most importantly, I could compare two very similar wikis that mostly differ in the language: One is in German and the other is English.

The results were eye-opening. Across pretty much all but the most obscure sites, the non-JS visitors outnumbered the ones that actually run JS code by about 100:1. Moreover, when checking the geolocation of these visits, they mostly seem to come from China and Brazil. Why would there be so many Chines and Brazilians interested in a German-language wiki?

The answer is: bots. These are automated processes that keep scraping the Internet for content. In most cases presumably to train AI models. In the process, they cause a significant load on the server, slowing down things for actually legitimate users. In some instances, they may incur costs for the owner (read: me!) if the hosting contract needs to be upgraded ... not to mention that they never even asked for permission to use my content for training their AI models.

A first idea was to simply block them. There are two mechanisms available for webmasters: Firstly, the [robots.txt](https://leib.be/sascha/projects/dokuwiki/botmon/blocking/robots.txt) file, which tells compliant bots to please refrain from scraping this site – and the [.htaccess](https://leib.be/sascha/projects/dokuwiki/botmon/blocking/htaccess) file which takes a slightly more radical approach, by blocking access altogether.

The first observation was: Most of the bots that are open about their identity (think: GoogleBot, BingBot, etc.) do respect the robots.txt instructions. Good. Adding the few that don't care about it to .htaccess pretty much takes care of those. The problem is that there are so many of them, and making sure that you add new bots to the blocklist as they appear is a lot of work.

And of course that leaves the other bots, which hide their identity by using random user-agent strings and jumping from one IP-address to another. Simply blocking the IP-ranges that they normally come from does not seem to help. They will just start scraping from other IP addresses. At some point I had almost all of South-America and a good part of East-Asia geoblocked for my wikis – yes, it reduced bot traffic, but also blocked a lot of legitimate traffic, which is not good.

At that point I came up with the idea that instead of blacklisting the bot operators, I should really get the bot operators to blacklist me – or rather: my wikis. The question is: how can I pull off that trick?

What if instead of seeing my wiki as a juicy source of high-quality content, I get them to see it as something they don't want to use. Ideally even something that could damage their precious AI models.

This is where the "Dada generator" comes in (though I also like to refer to it as "project brainrot" :-)

The "Dada generator" does exactly what its name suggests: it generates nonsensical texts that somewhat resemble the language of the wiki page (it uses word frequency lists as a source), but are, well, just utter nonsense. Using it to train an AI would actually degrade the model. This gives an incentive for bot operators to simply blacklist my site, rather than risking adverse effects to their AI. So far, this has reduced the bots-to-humans ratio to something like 20:1. Still not great, but a step into the right direction. And this tool only exists some a few weeks – I expect the number of bots to decrease further as more operators realize that there is something bad going on here.

The "Dada generator" is so far available in English, French, German and Latin – with Latin being the fallback language in case your page language is not available. Adding more languages is easy, though – please contact me if you want to help bringing your language to it as well.

If you want to be nice to our future AI overlords, you can also simply serve a standard "Lorem ipsum" text, which I hope they will recognize more easily as a placeholder text (though these bots are so simple that I don't have very high hopes on that). This is also the better option for wikis with very high visitor numbers, as it generates less load on the server.

The way this works is that if a certain cookie is not set, the server will serve this placeholder text instead of the actual page content. In addition, it triggers a JavaScript-based Captcha that is designed to be particularly easy to solve by humans (you just have to click a checkbox!). This then creates an encrypted cookie and reloads the page. The next time around the server sees the cookie, and if is meets certain expectations, the proper page is served.

Note that this will fail if you use some kind of proxy tool like Varnish or similar. In these cases, the proxy has to handle the Captcha instead.

To be clear: a skilled developer could easily write code that allows bots to bypasses the Captcha. The idea is that bot operators who didn't care about bots running any JS code at all are unlikely to put much work into writing a specialized script for a relatively small number of web sites, if they can also just blacklist them and extract content elsewhere. For a small CMS like DokuWiki, this should work. For a large one, like WordPress, the Captcha probably needed to be more advanced than this.

In fact, I don't even care much if there is one or another bot that bypasses the Captcha. It is the high number of bots that is a problem. If that goes down from hundreds to one or two, it is a great success :-)

Last but not least, data on all of these is also made available via a neat dashboard. It currently lacks long-term statistics, but it can already serve as a simple statistics module with information about the latest day of traffic – with a focus on "bot traffic", of course.

I should add that this Dashboard still needs more than a bit of work – in particular, better explanations what all these numbers specifically refer to. But it is pretty useful already as it is.

The next step is of course to further fine-tune this tool to better monitor, and better block bots. For this I would be interested in other people's experiences. It would be great if some of you could also test this and provide feedback.

I should also add that this is not a "install and forget" tool. It is a tool for hunting down and blocking bots, but it requires active management and monitoring. If you are ready to do that, have a look at: Bot Monitor Plugin.

As always: feedback, comments, complaints, but reports, suggestions, pull requests, etc. are very much welcome! :-)

Global DokuWiki Links