diremptiness
1 month ago
*goes to websites* *searches for Random button, fails* *finds Random in dropdown* *clicks update* *tabf5 is first listing* Ah yes, working as intended
tabf5
1 month ago
please ignore the $20 bill i slid under the table to Penelope the Cat and Kyle Drake
archieiscool
1 month ago
ohhhh that 20 bucks??? i def didn't steal it afterwards noooo that's crazy XD
strawberry-transneu
1 month ago
im an actual irl victim of war + it's crimes here; can we PLEASE be careful w war crimes jokes in this era rn? If you wanna be outlandishly silly saying "I'll invest these into crimes against humanity" is a better option. these are much more vague AND severe than war crimes ever were
strawberry-transneu
1 month ago
20 dollars is a great investment into crimes so better pick your crimes wisely
spotties
1 month ago
wait what? there's something i'm failing to understand, what's the point of adding the checkbox anyway if that's what you have the robots.txt file for?
tabf5
1 month ago
I think the checkbox is also a server-level redundancy, meaning that you're adding a second layer of AI-blocking along with the robots.txt
spotties
1 month ago
hmh, that does raise some concern,, i hope however this doesn't develop into a big controversy (๑•́ -•̀)
venthefrog
1 month ago
Noticed this the other day too. Very strange that they leave it unchecked, but at least they give us the option
wwwdotdeepblueme
1 month ago
Thanks, I got mine to the setting I prefer. They leave it unchecked because little to no one will voluntarily allow their sites to be scraped by AI. I do not know why neocities does it in the first place because it seems like a waste of money to allow AI to scrape sites.
sinproexe
1 month ago
literally just noticed it today... no idea since how long it was there (???) i have a robots.txt file but im not against additional protection, i guess... odd choice to default it unchecked though i agree 🤨
anypup
1 month ago
wait so if i turn off everything on my robots.txt, but leave the box unchecked, will bots still crawl my site?
gobbolab
1 month ago
Enabling this setting will trigger neocities to return HTTP 403 to known AI crawlers. This is separate from robots.txt in that it is a hard server side block. robots.txt is a preferences file you can put on your site which will tell potential crawlers what you do don't allow. Nothing forces a crawler to respect your robots.txt file however. Of note, not all crawlers are bad, example, the Wayback machine.
win1999
1 month ago
@anypup, if you have an empty robots.txt file and the box is unchecked, you will get bot scrapers.
win1999
1 month ago
Not all web crawlers harvest data for AI training, as gobbolab mentioned, many perform more essential functions like indexing search results, archiving web history, and monitoring site security. While a robots.txt file requests that bots refrain from crawling, it relies on voluntary compliance rather than technical enforcement, meaning malicious scrapers can simply ignore it.
tabf5
1 month ago
yeah I avoid using robots.txt because I still want my website to show up in search engines, but I did tick the checkbox to block the actual AI scrapers
win1999
1 month ago
@tabf5, I’d suggest a customized robots.txt file. You can target specific bots rather than making it an all-or-nothing thing. If you check that box in neocities, it'll block bots designed for Search Engine Optimization. It won't affect pages that are already indexed, but it'll prevent future crawls.
semi-sentient-goose
1 month ago
it being off by default is yet another addition to the nekoweb list lol
tabf5
1 month ago
Oh man, I forgot that part was in the tutorial. Maybe try asking @resrc for a guide on how to do that? He wrote one that also displays a viewcounter but with a new link that doesn't require a Cloudflare login. You might be able to reuse that part with dann's code too actually
viridianf
1 month ago
I actually got it to work by doing the simple instructions in your fix. Thanks so much!
maehat
1 month ago
reminds me of the guy in my college who has been playing undertale songs on the cafeteria piano just a floor below so i always hear it lol
tabf5
1 month ago
not all that much actually but i've heard his music so much that it's enjoyable to recognize in public!
Thanks! same with you