I did some research on this about 10 years ago. I spent 2 days hand labelling data from scraped news sites. Then built a good old fashioned Random Forest model to classify html nodes based on some feature engineering. turns out the P tag and the number-of-words threshold get you 90% of the way there, on news sites anyway. Great thing about RF models is they tell you which features are the most important. fun little project (apart from the 2 days of data labelling).
by esafak
1 subcomments
Why does the 'Quality vs Cost of Web Content Extraction' chart not have zero cost at the origin? Up to the right does not have to mean better; we can read.
by zaptheimpaler
1 subcomments
So this is tailored towards kind of a "reader view" for models right? Can it handle images, tables, shadow DOMs too? Like there are 3 use cases I have now - one is a simple text view for models to understand it, one is a "web clip" mode which would ideally preserve images and media, and one is to extract tabular data from web pages. Which ones is this good at?
by kocamaz
1 subcomments
It's good looking, and I liked it. The trial page accessed from the hugging face website is a very inefficient experience when I use Mozilla and the dark theme, FYI.
by wiradikusuma
1 subcomments
Does it work with ecommerce for product scraping? E.g. Amazon, or Shopee (big in SEA)
by andrethegiant
3 subcomments
Why not use a plain old html ā markdown converter? You can easily strip out ads using CSS /jQuery-like selectors. That would cost zero dollars.
by geniium
0 subcomment
Amazing I was just looking for something like this to be able to import web page content into Whisperit
by grillermo
0 subcomment
Iām implementing this for readitsoon my web to kindle app. Thanks for this!
by lnenad
0 subcomment
Very nice! Thank you for building this.
by tyzoid
1 subcomments
How does this work on pages that require JavaScript in order to render?
Hacker News