"Where's the data on accuracy?"

Backtesting is difficult to do correctly with LLMs, but because this is marketed as being for macro investing, I would expect to see a level of rigor and quantitative analysis consistent with that.

The Monte Carlo simulation engine sounds really cool, but is there evidence to indicate that it generates superior results to expert predictions, or to LLMs alone?

I actually think it would be totally fine for your beta version to have low accuracy numbers. After all, this seems to be something in the very early stages. But to have no quantitative analysis of your system's performance definitely makes me uneasy to trust it.

First, let's address the elephant in the room: we were inspired by George Soros' theory of reflexivity and how human tendencies affect markets more prominently than expected. Yes, there's a corny backronym [0]. No, this is not a political statement or endorsement of his views.

Coming back to the main point, we (the founding team at Lookback Labs) have both spent a long time at the intersection of financial markets, technology, and machine learning. During that time, one key thing that kept bothering us [1] was simply this: when a geopolitical crisis breaks, an investor's actual problem is not really to find out "what is happening now" — it's more of "which scenario plays out, how likely is each one, and what do I buy, sell, or hedge under each? For how long?"

There are a ton of existing tools and services that seek to answer the first question reasonably well (newsletters such as StratFor, publications such as Foreign Affairs and Foreign Policy, Bloomberg terminals for breaking news, etc.).

None of these answer the other questions particularly deftly. Sure, one can engage with ChatGPT (or Claude if one prefers), and play through multiple scenarios. You will, of course, miss out on the grounded structural model that powers Soros' analysis, along with the simulations that serve up the relative probability estimates.

Also, one of the worst things purely LLM-based ad hoc frameworks do is assume that countries are monolithic decision-making units from a game-theoretic perspective. This is hardly the case - "Iran" doesn't make choices, Mojtaba and the IRGC faction does. "China" doesn't decide, the Politburo Committee does. And so on.

There are of course formal analytical frameworks that dig deeper, studying groups, factions, organizations that are jostling to gain control (Bruce Bueno de Mesquita's Expected Utility Model and selectorate theory [2] is the most academically serious and is a prime inspiration for our system design), but they are extraordinarily hard to operationalize in real time, and produce no market implications.

To sum up, the choices are stark: ask AI and hope for the best, or build out your own systematic framework to organize evidence, assumptions, and implications. We chose the latter path.

Zooming out, our mission at Lookback Labs (https://www.lookbacklabs.com/) is to build "the intelligence layer for AI-native investing"; accordingly, Soros is the first of several agentic systems that we are designing across the systematic and discretionary spaces, that are both usable and useful from the get go, and not merely demo eye candy.

* Some minor details:

(1) We are currently in private beta for Soros and are onboarding selectively.

(2) The static demo is not completely static; you can still chat with the analysis (up to 20 messages a day per IP).

(3) We are still working on pricing: something that captures the value Soros provides.

(4) We want this to work for individual investors as well, not just institutional desks, and would love to price accordingly.

We're curious to hear what the HN community thinks about our approach. AUA!

Feel free to reach out offline if you'd like! We are, sadly enough, on LinkedIn, but are also available via email (anshuman/karen@lookbacklabs.com)

PS: As is probably obvious to the diligent reader :), every token in this post has been lovingly handcrafted by the Lookback Labs team.

[0] Scenario-Oriented Reasoner for Opportunity Synthesis. Lol.

[1] Many things bothered us. Buy us drinks, get stories.

[2] We heartily recommend two of BdM's books: "Predicting Politics" and "The Dictator's Handbook"