FRESH
Hacker News
Hacker News Hacker NewsNot too be pedantic here but I do have a noob question or two here:
1. One is building the index, which is a lot harder without a google offering its own API to boot. If other tech companies really wanted to break this monopoly, why can't they just do it — like they did with LLM training for base models with the infamous "pile" dataset — because the upshot of offering this index for public good would break not just google's own monopoly but also other monopolies like android, which will introduce a breath of fresh air into a myriad of UX(mobile devices, browsers, maps, security). So, why don't they just do this already?
2. The other question is about "control", which the DoJ has provided guidance for but not yet enforced. IANAL, but why can't a state's attorney general enforce this?
Google used by 90% or the world?
~20% of the human population lives in countries where Google is blocked.
OTOH, Baidu is the #1 search engine in China, which has over 15% of the world’s population… but doesn’t reach 1%?
These stats are made measuring US-based traffic, rather than “worldwide” as they claim.
It remains to be seen how or if the remedies will be enforced, and, of course, how Google will choose to comply with them. I am not optimistic, but at least there is some hope.
As an aside: The 1998 white paper by Brin and Page is remarkable to read knowing what Google has become.
>Because direct licensing isn’t available to us on compatible terms, we - like many others - use third-party API providers for SERP-style results (SERP meaning search engine results page). These providers serve major enterprises (according to their websites) including Nvidia, Adobe, Samsung, Stanford, DeepMind, Uber, and the United Nations.
The customer list matches what is listed on SerpAPI's page (interestingly, DeepMind is on Kagi's list while they're a Google company...). I suppose Kagi needs to pen this because if SerpAPI shuts down they may lose access to Google, but they may already have utilize multiple providers. In the past, Kagi employees have said that they have access to Google API, but it seems that it was not the case?
As a customer, the major implication of this is that even if Kagi's privacy policy says they try to not log your queries, it is sent to Google and still subject to Google's consumer privacy policy. Even if it is anonymized, your queries can still end up contributing to Google Trends.
Crazy for a company to admit: "Google won't let us whitelabel their core product so we steal it and resell it."
Kagi's AI assistant has been satisfying compared to Claude and ChatGPT, both of which insisted on having a personality no matter what my instructions said. Trying to do well-sourced research always pissed me off. With Kagi it gives me a summary of sources it's found and that's it!
https://storage.courtlistener.com/recap/gov.uscourts.dcd.223...
Will Kagi file an amicus brief in support of the plaintiffs
Perhaps Google will fund amici in support of their position as they did in the Epic appeal
https://www.law.com/nationallawjournal/2025/01/10/fight-over...
What even is market rate? Kagi themselves admits there's no market, the one competitor quit providing the service.
Obviously Google doesn't want to become an index provider.
Should actually be - Layer 3: Paid, ad-free, subscription-based search. (It's a subtle omission that indicates the direction Kagi search will eventually take).
There is no way the government provides a search engine that doesn’t become a political football or weapon.
Maybe in a different age.
I completely agree that monopoly remedies, such as fair open paid licensing, are needed. I prefer that to breakups, when this kind of cooperative/competitive leveling works.
My hope is that the powers that be figure out how to monetize these products with dollars instead of attention. Google’s ad-driven business model ruined the internet - we don’t need that in our AI products too.
Exa, Parallel and a whole bunch of companies doing information retrieval under the "agent memory" category belong to this discussion.
And Marginalia Search was not mentioned? Marginalia Search says they are licensing their index to Kagi. Perhaps it's counted under "Our own small-web index" which is highly misleading if true.
This would not only allow better competition in search, but fix the "AI scrapers" problem: No need to scrape if the data has already been scraped.
Crawling is technically a solved problem, as witnessed by everyone and their dog seemingly crawling everything. If pooled together, it would be cheaper and less resource intensive.
The secret sauce is in what happens afterwards, anyway.
Here's the idea in more detail: https://senkorasic.com/articles/ai-scraper-tragedy-commons
I'm under no illusion something like that will happen .. but it could.
> 14 renowned European research and computing centers have joined forces to develop an open European infrastructure for web search. The initiative is contributing to Europe’s digital sovereignty as well as promoting an open human-centered search engine market. [1]
> The Open Web Index (OWI) is a European open source web index pilot that is currently in Beta testing phase. The idea: Collaboratively and transparently secure safe, sovereign and open access to the internet for European organisations and civil society. The index stores well structured open web data, making it available for search applications and LLMs. [3]
Because text matching was so difficult to search with, whenever you went to a site, it would often have a "web of trust" at the bottom where an actual human being had curated a list of other sites that you might like if you liked this site.
So you would often search with keywords (often literals), then find the first site, then recursively explore the web of trust links to find the best site.
My suspicion has always been that Google (PageRank) benefited greatly from the human curated "web of trust" at the bottom of pages. But once Google came out, search was much better, and so human beings stopped creating "web of trust" type things on their site.
I am making the point that Google effectively benefited from the large amount of human labor put into connecting sites via WOT, while simultaneously (inadvertently) destroying the benefit of curating a WOT. This means that by succeeding at what they did, they made it much more difficult for a Google#2 to come around and run the exact same game plan with even the exact same algorithm.
tldr; Google harvested the links that were originally curated by human labor, the incentive to create those links are gone now, so the only remaining "links" between things are now in the Google Index.
Addendum: I asked claude to help me think of a metaphor, and I really liked this one as it is so similar.
``` "The railroad and the wagon trails"
Before railroads, collective human use created and maintained wagon trails through difficult terrain. The railroad company could survey these trails to find optimal routes. Once the railroad exists, the wagon trails fall into disuse and the pathfinding knowledge atrophies. A second railroad can't follow trails that are now overgrown. ```
This comment (https://news.ycombinator.com/item?id=46709957) points out that Google got its start via PageRank, which essentially ranked sites based on links created by humans. As such, its primary heuristic was what humans thought was good content. Turns out, this is still how they operate.
Basically, as people search and navigate the results, Google harvests their clicks, hovers, dwell-time and other browsing behavior -- i.e. tracking what they pay attention to -- to extract critical signals to "learn" which pages the users actually found useful for the given query. This helps it rank results better and improve search overall, which keeps people coming back, which in turns gives them more queries and data, which improves their results... a never-ending flywheel.
And competitors have no hope of matching this, because if you look at the infrastructure Google has built to harvest this data, it is so much bigger than the massive index! They harvest data through Chrome, ad tracking, Android, Google Analytics, cookies (for which they built Gmail!), YouTube, Maps, and so much more. So to compete with Google Search, you don't need just a massive index, you also need the extensive web infra footprint to harvest user interactions at massive scale, meaning the most popular and widely deployed browser, mobile OS, ad footprint, analytics, email provider, maps...
This also explains why Google spends so many billions in "traffic acquisition costs" (i.e. payments for being the Search default) every year, because that is a direct driver to both, 1) ad revenue, and 2) maintaining its search quality.
This wasn't really a secret, but it turned out to be a major point in the recent Antitrust trial, which is why the proposed remedies (as TFA mentions) include the sharing of search index and "interaction data."
We all knew "if you're not paying for it, you're the product" but the fascinating thing with Google is:
- They charge advertisers to monetize our attention;
- They harvest our attention to better rank results;
- They provide better results, which keeps us coming back, and giving them even more of our attention!
Attention is all you need, indeed.
There are other times (usually not work related) when I want to explore the web and discovering some nice little blog or special corner on the net. This is what my RSS feed reader is for.
It may be impracticable to share the crawled data, but from the stand point of content providers, having a single entity collecting the information (rather than a bunch of people doing) would seem to be better for everyone. Likely need to have some form of robots.txt which would allow the content provider to indicate how their content could be used (i.e research, web search, AI, etc.).
The people accessing the crawled data would end up paying (reasonable) fees to access the level of data they want, and some portion of that fee would go to the content provider (30% to the crawler and 70% to the crawler? :P maybe).
Maybe even go so far as to allow the Paywalled content providers to set a price on accessing their data for the different purposes. Should they be allowed to pick and choose who within those types should be allowed (or have it be based on violations of the terms of access)
It seems in part the content providers have the following complaints:
* Too many crawlers (see note below re crawlers)
* Crawlers not being friendly
* Improper use of the crawled data
* Not getting compensated for their content
Why can't Microsoft retain Bing traffic when it's the default on stock Windows installs?
* Do they not have enough crawled data?
* Their index isn't very good?
* Their searching their index isn't good
* The way they present the data is bad?
* Google is too entrenched?
* Combination of the above?
[1] https://en.wikipedia.org/wiki/Search_engine#2000s–present:_P... (2019)
Here are some examples:
- Discord
- WeChat (is it the web?)
- Rednote
- TikTok (partially)
- X (partially)
- JSTOR (it finds daily, but you find more stuff on the website directly)
- any stuff with a login, obviously.
Meanwhile, users pay a premium to pretend they're not using Google
Fascinating delusion
...
>We tried to do it the right way
This sign-up to retrieve better information idea will never take-off the way they think it will. A white label search will get you nowhere. They are silently failing because they're just too stubborn to do it the hard way. Kagi needs to pivot and succeed on useful and interesting edge cases first. Build us out a subject-relevant search, such as displaying vetted content from forums when searching a product/service, and then tying it into Facebook Marketplace for local items or services and Amazon for new. That is called building a product for yourself that others will use. Now you have your very own cashflow for clicks; use that cashflow to buy more corporate access, thereby proving you can succeed without any other search business propping you up and into relevancy. You don't need to start with the giants either. Start with something that works on local hunting, fishing, shooting, and knitting forums. When grandmothers need high quality green yarn today, make their muscle memory point to Kagi local, not Google.
On one hand, I really want Kagi to succeed. They very often, do seem to care about the parts of the world and internet that I care about. But on the other... to me, willingly associating, and financing a company that willingly brags about ignoring consent, is a non-starter for me.
https://news.ycombinator.com/item?id=46681985
https://news.ycombinator.com/item?id=44546519
I'm going to send this idea to my legislators, the EU, Sam Altman, Tim Sweeny, and Elon Musk, et al., I just haven't had time to put this together yet.
Google is a monopolist scourge and needs to be knocked down a peg or two.
This should also apply to the iPhone and Android app stores.
But Kagi funds Yandex which fund the RU government, and I think it should be known to anyone looking to use it.
https://ounapuu.ee/posts/2025/07/17/kagi/
https://kagifeedback.org/d/5445-reconsider-yandex-integratio...