I wanted to share my recent project: Tadpole. It is a custom DSL built on top of KDL specifically for web scraping and browser automation. I wanted there to be a standardized way of writing scrapers and reusing existing scraper logic. This was my solution.

Why?

    Abstraction: Simulating realistic human behavior (bezier curves, easing) through high-level composed actions.
    Zero Config: Import and share scraper modules directly via Git, bypass NPM/Registry overhead.
    Reusability: Actions and evaluators can be composed through slots to create more complex workflows.

This is a fully running example, @tadpole/cli is published on npm:

tadpole run redfin.kdl --input '{"text": "Seattle, WA"}' --auto --output output.json

  import "modules/redfin/mod.kdl" repo="github.com/tadpolehq/community"

  main {
    new_page {
      redfin.search text="=text"
      wait_until
      redfin.extract_from_card extract_to="addresses" {
        address {
          redfin.extract_address_from_card
        }
      }
    }
  }

    Control Flow: Add maybe (effectively try/catch) and loop (while {}, do {})
    DOMPick: Used to select elements by index
    DOMFilter: Used to filter elements using evaluators
    More Evaluators: Type casting, regex, exists
    Root Slots: Support for top level dynamic placeholders
    Error Reporting: More robust error reporting
    Logging: More consistent logging from actions and add log action to global registry

    Piping: Allowing different files to chain input/output.
    Outputs: Complex output sinks to databases, s3, kafka, etc.
    DAGs: Use directed acylic graphs to create complex crawling scenarios and parallel compute.

I've also created a community repository for sharing scraper logic: https://github.com/tadpolehq/community

Feedback would be greatly appreciated!

So KDL is like another JSON or Yaml. FSL-1.1-ALv2 is an, almost but not really, open source license that after a 2 years becomes available under a real open source license. It's to prevent free loading from companies or something. Sounds fine to me actually.

Most of my webscraping code is in Python, do you have plans to implement bindings to other languages?