It's hard to critique this paper directly because its claims are so incoherent and decorated with so much obnoxious verbiage[1] that people aren't going to believe me when I point out what the claims actually are.

Regardless, this is their paper:

First, they conflate the substrate with the presentation layer. Then, they point out that Turing equivalence means you can run an LLM on anything, with a pointless aside where they nerd out about making a logic gate in AoE II. This lets them conclude that you can use anything as the presentation layer.

Then they claim that it's natural to ascribe human-like attributes to outputs from some presentation layers, like abstract letter symbols on a computer screen, but not to most other things, like patterns of goats on AoE II, or LEGO. Yes, this seems to imply if your partner writes something heartwarming to you using LEGO, you're meant to laugh at them and point out how LEGO isn't intelligent so this isn't evidence of anything.

Then they do a thing where they say that assuming substrate independence is true (or false) prevents proving whether substrate independence is true or false, and from this, but just by vibes AFAICT, make it sound like everything one could learn about attributes of a system from its outputs is circular.

Then they write a bunch more incoherent text and mercifully then it ends.

[1] 'from an epistemic perspective, we argue that a generalised conclusion such as that necessarily requires a well-designed experiment' — the whole thing is like this.

Something about it seems to abuse the power of analogies to draw connections, treating view from 10,000 feet comparisons like they're proof of identity. So I do think a paper like this is perfect for the moment and just in time (if not a little late) because it responds to arguments of a form that are currently rampant all over Substack.

in its tone however it's written as if it's a brutal takedown of... somebody's perspective. It's hard to tell whose or what perspective exactly. Maybe I'm just misreading the writing style.

(Personally, I think the general case here is one of the better objections to computationalism about consciousness. You can make it even more absurd.

There exists some isomorphism between the velocities of the molecules in a glass of water, and the states of a Turing machine simulating a human mind. So is the glass of water conscious? Actually there are many such isomorphisms to many possible conscious minds, so is every glass of water simultaneously having every possible conscious experience?)

See also, https://en.wikipedia.org/wiki/China_brain

The interaction with those LLMs would be excruciatingly slow, probably spanning over multiple human lifetimes, but yes, why not?

That's not a property of LLMs thought, that's a trivial property related to turing completeness. I don't get what LLMs have to do with this - if LLMs have anthropomorphic properties, so does AoEII, duh, it's called emergence

Why does it imply that? That doesn't sound right to me. Unless we define "sufficiently powerful" as by definition producing that outcome, which seems unhelpful.

e.g. there have been experiments training transformers on things other than language, and it's not clear that this produces LLM-like qualities (nor does it seem likely to me).

---

Edit: I have misunderstood. The point was that LLMs can be run on any hardware (or in this case, emulator) that can do the actual computations. So the author picked AoE because it's an obviously silly example that goes against the tendency to anthropomorphize.

So basically it's the "substance/structure" question. (GPT-5 running on human neurons. Conscious or nah? Human neurons simulated on NVidia. Conscious or nah?)

But by the same argument, if you simulate a human brain in AoE, then what?

( Or for that matter, the universe containing all human brains: https://xkcd.com/505/ )

If we find out the universe is being run on a computer made out of legos, does that suddenly make all of us not sentient for some reason?

Now of course as an atheist I do not believe in soul and humans are just following evolutionary programming but we will never see chatGPT do standup comedy.

I thought this was going to be about NPCs in video games. NPCs, by intent, have human-like attributes. It's not hard to do. I've done a bit of that, pre-LLM. It doesn't even require anything near intelligence. Some NPCs are better than that. Unreal has demoed some that, if asked about it, can be made to understand that they are NPCs in a game world, and will talk reasonably about it.[1]

[1] https://www.youtube.com/watch?v=4sCWf2VGdfc

hence, what matters is the reversibility of the semblence, not the semblence.

LLMs do not do this readily, even if you can instruct them to, say, talk like a vampire, they wont just follow along. humNs winn.

The damn thing is a huge approximation function from input to output. It learns morality from correct inputs. Remember Microslops's Tay chatbot? Remember MechaHitler?

The whole industry is a Scientology cult by people who have read too much cheap SciFi. Unfortunately finance bros, who obviously believe none of this nonsense and laugh at the nerds, think they can milk it.

It's a take that is just disconnected from reality.

Ask a LLM whether bombing hiroshima was justified and you'll likely get a nuanced response. Ask AoEII the same...and well it doesn't even have an interface to ask that let alone answer.

...the entire premise is just gibberish