A deterministic layer like a compiler or generator of code with some kind of IR that the LLM generates and feeds it with.

I feel we will be seeing this more and more in the near future.

My understanding is that Vega was already an expressive DSL for visualizations and its probably already well spread through LLM training data.

This podcast (I only have a spotify link) has a really good short interview on this:

https://open.spotify.com/episode/18dHTAxCCeIaLOTch6tRld

The interview is with Frank Elavsky who seems to be a rock star in the field (and no, I don’t know him and am not him) and made a project called Chartability relating to this which has heuristics, principles, and guidelines for a11y audits:

https://chartability.github.io/POUR-CAF/

N of only a few of us working on an analytics agent, I don't think we've been finding this to be the case. We've been impressed with just how good LLMs (even smaller open weight models) are at using Python and R for visualization. Often any shortcomings go away if we iterate a bit to about ambiguity. Are there any threads of research that could better support this claim or highlight where issues might be?

Ok, Microsoft is conflating two different things here: LLMs don't really care about code being low level and verbose, they can read things like Assembly and SPIR-V just fine: visualization is the real issue in that LLMs have no natural understanding of spatial composition through visual comparison because they literally "see" things differently than humans, so the way to get around that is provide them with "visualization" in code form that they can easily reason about and understand, so basically anything that's not deeply nested and has hidden states that they have to reason about.

Also, Flint being stringly typed in JSON is a decision that I don't think I agree with. Looking at the actual spec, this could have just been a normal, human usable TypeScript library, and it would have been 100x better. Using their own example (excuse the formatting):

type SemanticType = "Category" | "YearMonth" | "Profit";

type ChartType = "Heatmap" | "BarChart" | "LineChart" | "ScatterPlot"; // extend as needed

interface ChartEncodings { x: string; y: string; color?: string; size?: string; tooltip?: string; }

interface ChartProperties { colorScheme: string; [key: string]: unknown; // allow other optional properties }

interface ChartSpec { chartType: ChartType; encodings: ChartEncodings; chartProperties: ChartProperties; }

type SemanticTypes = Record<string, SemanticType>;

interface ChartConfig<TData = Record<string, unknown>> { data: TData; semantic_types: SemanticTypes; chart_spec: ChartSpec; }

// The actual typed object literal: const chartConfig: ChartConfig = { data: {}, // replace with your actual data shape/type semantic_types: { game: "Category", period: "YearMonth", newUsers: "Profit", }, chart_spec: { chartType: "Heatmap", encodings: { x: "period", y: "game", color: "newUsers", }, chartProperties: { colorScheme: "redblue", }, }, };

EDIT:

Went and actually looked at the source instead of just eyeballing it from the docs, and it was a lot more complete and sophisticated than my assumed mockup already.

Core complaint (string-keyed JSON vs. a real generic authoring surface) still stands, but the specific types I posted aren't what Flint has. My bad.

Isnt graphviz there for the same reason?

Edit: I see it is using JSON as the declaration language, I am OK with llms being "good at json" but a syntax also consumable by humans it is not!

MCP setup: https://microsoft.github.io/flint-chart/#/mcp

[1] landed on pretty much the same spec for higher level tether charting on vegalite

I'm seeing more attempts at standardizing how AI displays data via presets. It's practical, I guess, and hopefully it will make things less error-prone.

but enterprise

Ppl who think this is a good idea should be compelled to maintain some M4 or other configuration "files". These configuration files ALWAYS devolve into a full language if they are actually used. So now you have the language you wrote your app in and then you have another, shittier language for your configs ... that you have to maintain.

Please stop doing this.

What should we do? In this case just express the chart in a natural language. Where there are uncertainties, the LLM should ask questions to clarify and record the results.

It's machine friendly but not exactly LLM friendly.

Here is a skill using that (tested)

I'm not sure if Flint is the right tool for me. I'd like to have a tool that expresses code in visual form for me. For example, right now I need to reverse engineer some code for debugging purposes.

I already found out there are three tasks:

    * Task one fills task two's queue and waits for an event to get notified
    * Task two reads from its queue, forwards elements to task three's queue.
    * Task three reads from its queue and sends a success/fail message back to task two's queue
    * Task two then notifies the waiting task one.

Which ML tools suited best for that?

and then that spec would be rendered either to a Bubble TUI via NTCharts or to HTML/SVG via ECharts. That Echarts HTML could be naturally served by a Golang http service.

But Flint goes much deeper with semantic layers and settings optimizations. Perhaps a NTChart, or whatever terminal chart, could be a rendering target? I'll add it to the list to explore...

https://github.com/NimbleMarkets/ntcharts/blob/spec/spec/REA...

The only „for AI agent” solution that will gain adoption is/should be measured by two dimensions: token usage and correctness.

If the solution doesnt use less tokens than generating chart.js code for example - why should I use it?

Same for correctness, if the generated chart spec is correct only 90% of the time - why should I use it?

It’s still early but I think this is the direction we should be thinking about „for AI agents” libraries and projects.

Functions extremely well and the result is a very clear (and consitent) human-readable "output layer." Cool idea, fun to see people converging on similar concepts in the space.

It might lead to good or miss leading.

While LLM ability boundry extends, it might invalid for next turn.

Best Qiuwu

Agents, npm, typescript, MCP. All buzzwords are there. Will anyone look at the slop charts? Of course not, the tokens are the goal.

MSFT stock is at 2024 levels. Maybe someone should produce a flint chart and present the agentic work to Nadella. No one buys this AI slop any more.

Make something people want.

I’ve been building https://smalldocs.org for this exact reason. It’s an office suite for AI agents - but my main use case is giving a cli based LLM the canvas to express itself - charts, mermaid diagrams, etc. I’ve extended it a bit further to be a format for all types of work so the agent can embed slides and spreadsheets in a document.

Sample document: https://smalldocs.org/blogs/what-is-a-smalldoc

Source: https://github.com/espressoplease/smalldocs