I just tried the OCR capabilities with a photo of a DIN A4 page which was written with a typewriter. The image isn't the easiest to interpret. The text perspective is distorted because the page is part of a book and the page margin toward the spine of the book is very small. There are also many inline corrections due to typing errors while the page was written (backspace couldn't erase characters back then, and arrow keys couldn't be used to add text in between existing words). Over the past months I've tried to use several LLMs on this very same image already (1 out of 200 pages that seek digitization). The result is by far the most accurate so far. Only some very minor errors (which are also non-trivial for human translators) were made.

This page induced costs of about 25 cent. I assume I could tweak the input image a little more to consume less input tokens. OCR-ing all 200 pages would otherwise cost a juicy 50$ - although there is a generous 20$ of free credits.

Induced cost: 108.8k Input tokens => 16,32 cent 24.5k Output tokens => 8,58 cent

// Edit: I just re-tried the same task utilizing a capability of the API to only run a specific part of the model (e.g. _only_ OCR). This cuts cost by 3x (to ~8c/page) but significantly worsens the result. The result is missing entire lines of the original document. There are also many error in the text that was recognized.

The output was correct, and seemed deterministic, although I ran it only 2-3 times on the same image.

Main problem is response time: it took about 20-25 seconds for a simple structure of 5 fields. As such unusable at scale, let alone "real time" processing.

Other problem is cost, it is considerably more expensive than more established models for the same document, like flash-light.

Shame, the architecture is very interesting.

Does code extraction and manipulation fit in that? Would interfaze be the agent that a coding agent uses?

The graph doesn't exactly make it clear but it describes a pipeline that goes beyond the LLM, so the CNN could be a separate model there.

The first OCR example returns output that does not detect the article columns - the bounding box is the entire first line.

https://docs.mumbli.app/benchmarks

It'll be interesting to see it on my coding evals as well. Can't do it yet but will try later.

how do I run it locally?