Added some cloudflare verifications on forum games

[Pleeb]

We use cloudflare for our DNS, that's nothing new.  It's also how I implemented rate limiting, but with the bots really hitting forum games hard, it's been dragging the site down to several seconds of page loading.  I've added a cloudflare managed challenge to the forum games and profile views.

 

The first time you view something that hits it, it should just say "We're verifying that you're human", and it should only happen maybe once per session.  Please let me know if it causes anyone grief, but I'm hoping this causes things to finally work normal again.

 

NOTE: if you are viewing a specific post, for example if the URL ends in "#comment-477346" it will also trigger. I added out to the rules too because the bots would spend 1 million requests looking at the exact same topic but with `#comment-477346` (different number every time) at the end.

[Ranger]

Oh thank god

Invision's activity feed likes to link post numbers instead of the actual thread, so that could make things interesting

[Reisen]

It works out pretty perfectly so the crawlers only do exactly what they're supposed/intended to and don't get stuck in loops

 

The re-verification seems to trigger every 60 minutes right now, and seems random whether we'll have to click or not

Might be based on if you have "human" mouse movement in that time, not sure

Edited June 15, 2025 by Reisen

[Ido]

Works with TBB 'Standard' settings. I'm not happy having to go to such a low security setting that allows websites to detect stuff like mouse movements - or any JS-related stuff in general, but if bots are botting it can't be helped. I guess the only way to lock them out would be to make topics like forum games accessible to logged in users only. But then they'll just go to another high-reply topic.

I also still don't understand the rationale of those bots, it appears to happen in a lot of places. Thousands of 'guests' = bots online looking at a prominent topic with many replies.

Porque?

Scraping the internet for AI training material? Even then they should be done in a few hours and not make permanent requests for months. It's basically a DDoS attack.

Edited June 16, 2025 by Ido

[Ranger]

2 hours ago, Ido said:

It's basically a DDoS attack.

 

It may just be a DDoS attack. Something something maybe Jade is still trying after all these years

[Reisen]

Crawlers for indexing are as old as time, they're how search results exist

I imagine they also crawl for tons of other random reasons too though, and now with the advent of the AI boom the problem has only gotten much worse

They're SUPPOSED to listen to a robots.txt on the root directory of a site to know what they should crawl though, badly behaved ones do not do that

And poorly made ones will get stuck doing nonsense like treating every #comment-477346 URL as a new page, etc.

 

Cloudflare does a good all-around job of helping thwart this with its customizability

Although.. The number of users/guests on the Who's Online list tanked to <20 after the cloudflare thing, a whole digit lower than it's basically ever been. I wonder if, while hypothetically crawling only the un-cloudflare'd threads, they're clicking the Latest Post comment links indiscriminately and getting walled from the site, unable to do the crawling we meant for them to do?

Well, oh well, we already don't show up when googling "tulpas" anyway, so I don't think the crawlers could do anything good for us. (As for AI training... I wouldn't personally mind, but it's on them for coding lousy trackers that don't work right)

 

- Oh, we do show up much quicker on duckduckgo vs google's low page 2 though

Guess Pleeb can choose to care if he cares & can do anything, or not if not

Edited June 16, 2025 by Reisen

[Pleeb]

https://www.theregister.com/2025/06/17/bot_overwhelming_websites_report/