FRESH

Hacker News

Home

Fix your robots.txt or your site disappears from Google

159 points by bobbiechen

by WmWsjA6B29B4nfk

5 subcomments

Google docs are pretty clear (https://developers.google.com/crawling/docs/robots-txt/robot...):
> Google's crawlers treat all 4xx errors, except 429, as if a valid robots.txt file didn't exist. This means that Google assumes that there are no crawl restrictions.
This is a better source than a random SEO dude with a channel full of AI-generated videos.

by franze

0 subcomment

Fake or just miss-informed
this is the support page https://support.google.com/webmasters/community-video/360202...
this is the creators linkedin https://www.linkedin.com/in/iskgti/
he does not work for google, just a seo somewhere that creates videos and posts his hypothesis in forums
this is his youtube account https://m.youtube.com/@saket_gupta
nice high quality - propably ai created videos - still no relationship to reality

by jimberlage

2 subcomments

I remember back in the day, when SEO was a more viable channel, being surprised at how much of the game was convincing Google to crawl you at all.
I naively assumed that they would be happy to take in any and all data, but they had a fairly sophisticated algorithm for deciding "we've seen enough, we know what the next page in the sequence is going to look like." They value their bandwidth.
It led to a lot of gaming of how you optimally split content across high-value pages for search terms (the 5 most relevant reviews should go on pages targeting the New York metro, the next 5 most relevant for LA, etc.)
I'm surprised again, honestly. I kind of assumed the AI race meant that Google would go back to hoovering all data at the cost of extra bandwidth, but my assumption clearly doesn't hold. I can't believe I knew all that about Google and still made the same assumption twice.

by skybrian

2 subcomments

Not sure if this is reliable.
- What does "unreachable" mean, exactly? A 404 or some more serious error?
- What is a "Diamond Product Expert" and do they speak for the company?

by cj

2 subcomments

Not having a robots.txt is fine as long as it's a 404. If it's a 403, you'll be de-indexed.
I have a feeling there's more to the story than what's in the blog post.

by Aardwolf

1 subcomments

If true, this would mean more websites with genuine content from the "old" internet won't show up (since many personal websites won't have this), while more SEO-optimized content farms that of course do put up a robots.txt will...

by senko

1 subcomments

> Your robots.txt file is the very first thing Googlebot looks for. If it can not reach this file, it will stop and won't crawl the rest of your site. Meaning your pages will remain invisible (on Google).
This implication (stopped crawl means your pages are invisible) directly contradicts Google's own documentation[0] that states:
> If other pages point to your page with descriptive text, Google could still index the URL without visiting the page. If you want to block your page from search results, use another method such as password protection or noindex.
What I get from the article is the big change is Google now treats missing robots.txt as if it disallowed crawling. Meaning you can still get indexed but not crawled (as per above).
My cynical take for this is this is a preparation for a future AI-related lawsuit. Everyone explicitly allowing Google (and/or other crawlers) is a proof they're doing it with website's permission.
Oh, you'd want to appear in Google search results without appearing in Gemini? Tough luck, bro.
[0] https://developers.google.com/search/docs/crawling-indexing/...

by forinti

2 subcomments

My logs tell me that Google ignores my robots.txt.

by sidnarsipur

0 subcomment

I've made a nice tool (https://botblock-puce.vercel.app/) to generate custom robots.txt easily if anyone is interested.

by estimator7292

4 subcomments

What I'm hearing is that if I tweak robots.txt I can exclude my site from google? Excellent news!

by geldedus

0 subcomment

I don't even bother with Google indexing anymore. They massively de-index perfectly valid and useful pages. And this happens to thousands of sites. Google becomes less and less relevant.

by dazc

0 subcomment

I've witnessed a few catastrophes that have resulted in mistakes made via robots.txt, especially when using 'disallow' as an attempt to prevent pages being indexed.
I don't know if the claims made here are true but there really isn't any reason not to have a valid robots.txt available. One could argue that if you want Google to respect robots.txt then not having one should result in Googlebot not crawling any further.

by nextlevelwizard

0 subcomment

Who cares? Top results are anyway paid ads

by efilife

0 subcomment

"Here's the video from Google Support that covers it:"
This Google Support is another indian spammer that generates tens of nonsense videos and uploads them to YouTube: https://www.youtube.com/watch?v=2LJKNiQJ8LA
This guy is not affiliated with Google in any way other than spamming on their help forums like indian people tend to do
https://www.iskgti.com/
His own website has 92 score in SEO on lighthouse despite his claim he's a "SEO expert"
From the article:
> I don't have a robots.txt right now. It hasn't been there in a long time. Google still shows two results when I search for files on my site though:
guess why

by gmiller123456

2 subcomments

Sounds like great news. Users will eventually figure out other search engines produce more relevant results and Google's dominance will fade. Hopefully they never "fix" it.

by mrroryflint

0 subcomment

Thanks for sharing - I took a look at my own robots.txt and realised the sitemap was broken.

by vicpara

0 subcomment

A lot of websites have robots.txt and sitemap.xml protected by cloudflare if you can imagine that. That's crazy.

by ArcHound

3 subcomments

To reach my site, users need to get through the AI summary first. Spoilers: they don't get through more often than not. This is based on the drop of views since AI summary started.
And honestly, I don't blame them. If the summary has the info, why risk going to a possibly ad-filled site?

by ThinkBeat

0 subcomment

Mildly odd given the desperate bots trying to mop up the entire internet that leeches anything.

by Animats

1 subcomments

So Google Search is now opt-in? Good.

by Bengalilol

0 subcomment

...and not a single link to any Google dev page...

by bflesch

0 subcomment

Don't invest any second of your time into the US tech monopoly. That time is much better spent deploying non-US alternatives and backing up your data from US clouds, which could be blocked for us any moment.
Google is a rent-seeking parasitic middleman leeching off productive businesses, let them hang out with their best friends at the US administration.

by crazygringo

1 subcomments

This is interesting and unexpected if true.
My only thought is that virtually all "serious" sites tend to have robots.txt, and so not having it indicates a high likelihood of spam.

by mwkaufma

0 subcomment

Yes it's _our_ fault Google search was enshittified.

by wackget

1 subcomments

> The top Stack Overflow answer on robots.txt has a discussion about Allow: / not being valid according to the spec. The only date for the comments is "Over a year ago" but given that the question is from 2010 the comments are probably from around that time.
Firstly, I detest that stupid "feature" of showing only relative dates. It makes screenshots impossible to date, and it's frankly useless for humans as proven by OP's article.
Secondly, you can hover over the relative date string to see the actual date. But don't let that stop you from hating it.

by nikanj

1 subcomments

I remember how religiously people used to care about their Google ranking. It's almost shocking to realize how fast that has changed. People used to spend tons of effort gaming site load speed, optimizing sitemaps and writing blog content.
All of that is fast getting completely irrelevant, people see ads on their favourite TikReels app, find their holiday presents on Temu and ask their questions from ChatGPT

by josefritzishere

2 subcomments

The irony is that their AI bots still hoover up all your site content regardless.

by shevy-java

0 subcomment

We need to fix Google.

by Onavo

1 subcomments

Is this a compliance issue? I can't imagine why they would willingly not scrape.

by brianbest101

0 subcomment

[dead]

by linolevan

3 subcomments

This is a crazy change. I wonder if part of the reasoning is that sites without a robots.txt tend to be very low-quality. Search is a very hard problem and in a world of LLM-generated internet, it's become way harder.

by Igor_Wiwi

0 subcomment

Thanks for the heads up. I am releasing 10 projects every month it's really easy to miss some of the SEO fundamentals, to fix it I created a Chrom extension to verify basic stuff https://chromewebstore.google.com/detail/becgiilhpcpakkecdho...