I rebuilt my blog's cache. Bots are the audience now
- Personally, I think this is a good idea. But the core problem is this:
How is a newcomer supposed to build reputation now?
Without exaggerated business promises or capital, basic online reputation usually depends on writing. In fact, my own first step into freelancing came because someone found the articles on my Korean blog interesting.
So the question is: if the subscribers are bots, what benefit do they actually give me?
If bots become the readers, then what matters is whether they can provide any kind of symbolic capital or real capital. I can build caching with Redis without much difficulty, but I worry that if this continues, the result may simply be that LLMs learn from my writing while no benefit returns to me.
People write partly to organize their thoughts, but also partly to gain symbolic capital. That is one reason why I write my own posts instead of using an LLM to write them for me.
by pavel_lishin
3 subcomments
- > Not because I expect a person in Singapore to shave 200ms off their pageload, but because the next request for that page is more likely to come from a retrieval system than a browser, and the request after that, and the one after that.
Why do I care if I shave off 200ms from a crawler's request, instead of a human's?
by chrismorgan
2 subcomments
- I’m very confused about why you’d have such a complex cache arrangement. Sounds like you’re using Cloudflare and Fastly to do roughly the same thing. That sounds like a recipe for more expense and more problems.
For the sort of thing you’re doing, it should be as simple as “throw it behind Cloudflare/Fastly/Bunny/whichever private CDN you like” and that’s it.
Also the diagram near the end is pretty much incoherent. GenAI, I presume.
- my tiny blogs no one reads have been racking up a huge deno deploy and vercel bills ($40-50/mo each) bc I ran them "naked" without a cache or cloudflare or static builds - it didnt matter bc I got like hundreds of visitors a month. they were just hono or whatever api pulling from my backend which could be notion or airtable - super simple, though kind of slow
now I suddenly I have 10k visitors a month hammering my apis and causing massive egress and cpu usage - so i had to get them behind cloudflare and now build everything statically - cut the costs back down from 90+ cpu hours to about 0.2 cpu hours a month
crazy times
(also, all donw w/ claude code's help, or it would have taken a week for me to figure out)
by cullumsmith
4 subcomments
- I simply block all AI crawlers with a user-agent check in nginx.conf.
- I denylisted traffic from Singapore. As far as I could discern it was all Windows 11 machines running Chrome accessing pages sequentially. I’d love to not do that, but trying to sift through that is quite the task.
by faangguyindia
1 subcomments
- Yesterday I logged into cloudflare and found that Cloudflare had blocked chatgpt and claude from accessing my site. https://macrocodex.app
This is bad because there are fitness guides on my domain
https://macrocodex.app/guides which newbies often put in chatgpt and asks to simplify.
I enabled crawl for LLMs. There is lot of misinformation in fitness field so it's better if LLMs get their content from people who atleast have experience in the field
- the core value of internet was some one discovering you via your content, agents as primary consumer might looks good for now, but we are definitely making internet dead for many SMBs.
by steve_adams_86
1 subcomments
- I went through a similar process recently. For a while I saw readership of my site gradually increasing, and eventually it became clear that it wasn't human beings.
I also used Claude to help me drill into what's going on. Bizarrely, about 80% of my traffic comes from Singapore, which the author mentioned. I don't know why. A lot of the traffic looks real; it stays for a while, clicks different links in different orders. But no one in Singapore has ever read a thing I've written on my site as far as I'm concerned.
I thought Cloudflare would help protect my site from bots, but it utterly fails. I'm not sure if they're too sophisticated or people overestimate how well CF works for these things. I paid for advanced features for a while and reverted to the free plan once I realized it made no difference. It's a great platform in general, but hasn't been great for allowing me to see how many humans actually read my content.
I know some do because they email me occasionally. If I had to guess, of the ~200 visits per week reported in analytics, around 15 are real.
- Why would I want bots to read my blog?
- It is time rewrite to X to optimize Y :)
by Hackbraten
3 subcomments
- Why do I get just an empty page?
- The writership of the blog is also changed and seems to be mostly machine as well. It is painful to read something that lacks human presence on the other side.
Hacker News