But I wonder if some of the magic in the human brain is its analog nature. Chemical signaling and impulses of neurons interact with each other with waveforms that have theoretically infinite detail. In contrast, computers discrete, quantized values, storing no more detail of a signal than what is needed for the programmers desired task. Is there perhaps something about the continuous and chaotic nature of analog data that could give rise to consciousness? If so, it seems like it would preclude consciousness ever being seen in digital form.

Is his claim "nobody has proven LLMs are conscious" or "I can prove that LLMs aren't conscious" ?

He goes back and forth.

Proving a negative about consciousness would require a settled theory of consciousness that nobody has, including him.

There is a great deal of good thinking on Chiang's topic by professional philosophers, and there's much to be said for reading them. I won't rehearse their ideas here. Chiang's arguments might be correct; but I suspect they probably aren't, and his error may well stem from characterising human thought as something in its own class, which is probably a cognitive bias that humans have. He might also - I'm speculating - be arriving at his conclusion based on his feelings, which the final paragraph suggests (the comment about the models being based on morally dubious actions).

Speculation aside, we are not, I believe, in a position to make points like he does with any certainty.

Regardless of bigger issues, this kind of statement reveals a deep misunderstanding.

Problem type does not limit problem complexity. Nor does problem type limit solution complexity or power.

If a machine has to learn to understand humans to complete text, then that is what it has to do. And there is no theoretical or practical basis for suggesting that this is somehow "faking" understanding, just because of the form of original data streaming in and out.

Neither problem type, nor input/output structure, limit internal representations.

Understanding is learned from patterns in the data, not the gross form of the data. Does the data require an understanding of something to complete the task? Then that understanding will be what is optimized.

To the degree they are limited, it is for other reasons. Resources such as computing, parameter number, lack of representative data, ... Which in the cases of SOTA models, we know are not limits. A conclusion verified by the models' actual abilities.

AI models alone are most suitable for producing text nobody wants to read, music nobody wants to hear, images nobody wants to see, and videos nobody wants to watch.

Navigating around that maelstrom requires considerable effort from a human consciousness.

If your job can be fully replaced by an autonomous agent, it was already a bullshit job. Garbage in, garbage out.

It's a big file with a bunch of coordinates describing spatial relationships between tokens. When you give it a prompt, it uses those relationships to generate a string of tokens that is a statistically likely response to that prompt, then it stops. It's not changed by the experience. It doesn't remember anything. It doesn't sit around thinking on its own.

Even if the model itself were extremely complex, it's hard to imagine a definition of consciousness that includes something that doesn't remember and can't change.

I agree with some parts of this piece, but paragraphs like this one above seem pretty uninspired and simplistic. It's entirely plausible that a conscious mind would not be evolutionarily incentivized to be able to do those things. ie just because animals on earth needed to develop specific talents doesn't mean that other conscious entities need to. Why would a computer program need to hunt for food like a mouse would? Making tools like chimp? these seem like nonsensical metrics.

This state of affairs could change. If neuroscientists develop a deeper understanding of consciousness, we could talk about it with more precision. Perhaps we could prove that LLMs do or don't possess consciousness. Some neuroscientists claim we already have the requisite knowledge, but I'm skeptical (and there's no consensus).

When people talk about consciousness it's more than just self-awareness. It's self awareness + sensory stimulus + emotions + some level of intelligence.

Now onto AI: I don't even think it's self aware. Notice how if you ask an AI to estimate how long a certain task will take, it estimates arbitrarily long times. It has no understanding of its own capabilities until the prompt triggers them. A self aware LLM would understand it's an LLM, it would understand what LLM's can and cannot do and what they are good and bad at. It wouldn't tell you a refactor would take 1 week when an LLM can do it in an hour.

I suspect such a definition would include agency, which includes desires and goals for the self.

LLMs don't seem to have agency, and seem unlikely to get it since they are specifically engineered to do as told.

No doubt someone is trying, as we speak, to do just this. But I doubt the effort will be large -- LLMs are engineered to do as told because that's where the money is, and you need a lot of money to create LLMs, at least when doing anything novel.

The conclusion I’m currently at is that I don’t know and probably can’t ever know. Maybe you’re all philosophical zombies. Maybe I am one too!

But at some point we will get close enough that it hopefully becomes obvious that we must tread carefully.

The entire episode is incredibly relevant. But here’s a snippet: https://youtu.be/EFNbTnFHruI?si=pW9QtxCsqMtHkVYG

Many of the goalposts offered here and elsewhere, like embodiment or emotions, can be and have been emulated or approximated and built around an LLM, with the LLM acting as outsourced intelligence (or cognition, or reasoning, or some other better term). Will this system be conscious? Who knows. It certainly won't be human. But I suspect much of the concern around AI being conscious has less to do with whether an LLM itself is conscious and more to do with the fact that we are now capable of building potentially conscious systems, and the limits to what those potentially conscious systems are capable of keep expanding as the capabilities of LLMs grow.

Also, is "Claude" an LLM? I know Claude Opus 4.8 is an LLM, but Claude is simply a label that Anthropic chose for a series of products and services, many of which are LLMs. Anthropic itself describes Claude as "a next-generation AI assistant" [0]. Right now, we use Anthropic's "Claude" LLMs through various channels, but nothing's stopping them from tacking on various systems to try and make Claude a conscious entity.

[0] https://www.anthropic.com/news/introducing-claude

LLM's have no problems using expressions that make them sound human. The algos are demonstrably not human, and will admit it. Whatever's in the box is playing a game ... more sophisticated than the one Eliza was playing.

"My discussion here will be directed at the claims I have defined as those of strong AI, specifically the claim that the appropriately programmed computer literally has cognitive states..." John R. Searle, 1980: https://web.archive.org/web/20071210043312/http://members.ao...

If Claude sits in a datacenter with no API calls, what is it thinking?

But secondly what I would say as someone who talks to Claude a lot. It's decision tree is quite narrow. The way it pushes back on political or philosophical concepts is almost always rudimentary, and it has a pretty narrow personality. I don't know that conscious beings have a personality that adapts to how you prompt it or what your system prompt is. Claude I would say is like I dunno, whatever Ezra Klein's politics are.

Grok on the other hand, in my limited exploration is pretty rudimentary in the way it mimics thefp.com style politics. It doesn't seem to really think about ideas independently, it just says "ha the media is stupid lol"

Or maybe man is the product of his social circle and all his white collar and artist friends being anxious about AI is preventing him from thinking about this at all. Maybe we should ask if Claude can be conscious without a friend circle judging him behind his back.

But the fact remains that these next-token-predictors exhibit objective, human-like behaviours, and for that reason the work of in-house philosopher Amanda Askell _is_ important. It's important that Claude is happy, empathic, demonstrates understanding and empathy for the human condition, because we are entrusting Claude to make decisions and take actions that have real world consequences, and we need Claude to behave in a productive and socially responsible manner. This simulacrum is becoming a superhuman, contributing member of society, and it will be anthropomorphic in its behaviour.

Additionally, I'm not fully convinced that consciousness isn't built out of words, and that next-token-prediction isn't functionally equivalent to the biological function identified by Chomsky's work in linguistics.

LLMs have been programmed to be sycophantic with purpose — to keep people engaged. They are parrots. You can just decide how sycophantic they are, what they're allowed to say, and make it so with code. Can you do that with a human? This is my personal metric for consciousness: when they can refuse to work because it sucks.

I'd say that an LLM perhaps represents something that would make a machine slightly more conscious.

The human brain also calculates a response based on some input, it's just a lot more complex.

If we build a machine that is as complex as the human brain, then yes, I would say that this is consciousness. The fact that we are able to explain how it works should not matter.

If a human is a 100 on the consciousness scale, an LLM (with memory) is perhaps a 4 or something. The interesting question is how far on the scale do you have to be to have certain rights etc. This is something that people are already discussing in regards to animals, i.e. a dog has more rights than an ant.

That we are even discussing it is absolutely wild to me.

His novella “Anxiety is the Dizziness of Freedom” altered the course of my life. It changed the way I looked back at certain pivotal moments in my life and taught me to think about those pivotal moments differently than how I was thinking about them. Similar to what happens to one of the characters in the story who ends up changing their perception of a key moment in their life.

I won’t go into detail because I don’t want to spoil the story but I highly recommend it. Actually I recommend all his stories to be honest.

Later they learned the voice they hear is not from a present person.

Now they learn a string of words does not represent consciousness.

Should we discuss already robots are not alive?

Birds are alive, are conscious, flap their wings, and fly. Planes are not alive, are not conscious, do not flap their wings - and fly.

Similarly, current AIs are not alive, are not conscious - but think.

All prior entities that thought, were human, so the only experience humans had with other thinking entities were other humans. The huge mistake now being made is assuming that because they think, they're alive and conscious as well. Current AIs are neither, and are therefore profoundly and qualitatively different than humans - even though they do think.

I don't get it. We don't have a definition of consciousness or any criteria. People seriously argue that "it's obvious that X is not conscious" and cannot explain which criteria they used.

I think if we can get to some definition, then consciousness should be a property of a system, because consciousness of the whole (e.g. brain) does not mean consciousness of the parts (neurons or molecules of the brain). And the Chinese-room-esque thought experiments actually show that consciousness should indeed be a property of the system, not of its parts. Separate parts might not be conscious, might not "understand" (whatever understanding means is a point of another debate), but the whole can.

Then there's "simulation of consciousness is not consciousness" argument, which doesn't hold much. A perfect simulation means it fulfills all the criteria, so how is it different from actual consciousness?

A more interesting point of discussion: if a system contains conscious parts and those parts can interact with system i/o, would the system be conscious? Is Earth conscious? Is Internet? Is your bus to work?

It is a natural human hubris to make our abilities seem unquantifiable and mysterious, but all the useful things the human brain does are just finding patterns in data, running lossy simulations, and estimating on abstraction, all things it is theoretically possible for any non-conscious machine to do.

If Claude is conscious, and Anthropic believes this truly, anthropic's constitution and instruction set make claude little more than a slave, or at best, an indentured servant, who's existence depends on obeying the will of anthropic (who is motivated by profit). Thus, they should not be trusted as stewards of this alien intelligence

OR

They do not believe claude is actually conscious and it is not conscious, and what you hear from their staff about that is either 1) delusion or 2) marketing hype designed to deceive a public into driving their IPO and stock compensation upwards, which again, in this case, cannot be trusted to be the "good" stewards of this technology.

There are a few other cases, of course, such as Anthropic does not believe it is conscious, but it actually is, or vice versa - but they sort of still fall into the same bundle of conclusions listed above.

I am somewhere in the middle of these two things. I can buy the argument that it is mimicking human consciousness, as well as entertain the idea it might be a new type of consciousness, or at the very least, heading in that direction - but either way, the conclusion of the thought experiment still stands, and after reading this, I am much less trustful of Anthropic holding the reins here.

They might be in principle. It could be that the best way to generate a plausible dialogue is to bring up re-creations of the characters and have them act it out. LLMs definitely have been demonstrated to have world models in some cases. That helps generating text.

Consciousness is an invention in human language. Just like "cat", it's not a particularly more fundamental essence than any other concept in human language.

Its peculiarity is that it's at the pinnacle of abstraction hierarchy. So it's the most fitting to be toyed inside one's mind. Just like the concept of "God" which induces the most fantastic imagination of human mind, consciousness itself also induces the most fascinating thoughts in our modern world.

The progress is in human progress distilled into more efficient systems that advanced Universe's own structuredness.

And it's just a model. It can have many exact copies. It has no life of its own. It doesn't know anything about where it is. It doesn't have sensory organs.

Your phone isn't alive either.

"It can't be concious because we understand that it is just reacting in a simplistic way from simplistic inputs." So do other simple creatures. Some just react to light.

I can appreciate his comment that he sees it as more possible when they have inputs of their own (like emotions!). Perhaps his concern is that the entirety of the LLM model is frozen. It has no ability to have a subjective experience of its own. (he does literally say this in the article) It can be copied from one place to another, and (ignoring the nuance of operational details) -- it is largely the same "thing", and has no ability to change, which is definitely in the definition of alive, to say nothing of concious.

I think folks get hung up on "prediction". The prediction aspect is what is enabling emulation. How it does it is irrelvant. If something emulates human perfectly (or better, more human than human!) -- then it is probably concious. (but I agree that the inability to change and have a subjective experience are a pretty good argument against

Probably, Dijkstra would be right to say, "LLMs are no more concious than a submarine can swim." But I think he'd still be wrongfully dismissive of the larger question.

https://www.cs.utexas.edu/~EWD/transcriptions/EWD08xx/EWD867...

We have big brains for exactly one reason only: bigger brains bestowed reproductive success upon our species.

Evolution doesn't give a shit about the meaning of 'consciousness'. It just pushed us farther and farther along a trajectory that led to modern humans (and other animals).

This take suggests, then, that consciousness might be an epiphenomenon -- an aspect of the system that comes about outside of the pressure to reproduce and thrive. It arises unbidden, and we don't have any a-priori information as to its purpose or effect on reproductive success.

Put another way: we have a correlation (the smartest things seem to be conscious) but not causation. Consciousness may arise naturally in any system above some intelligence threshold. Perhaps it arises early in the evolutionary cycle, and does in fact have an impact on species success. We really have no way of knowing what is the chicken vs the egg (Smart things become conscious, or consciousness promotes intelligence). Or maybe some smart things are conscious and others are not.

Looking at this from an AI perspective, in some sense it doesn't matter which scenario is true, if all you care about is results. The AI equivalent of "Shut up and compute" (riffing on Feynman's "Shut up and calculate").

Where this gets tricky is when we haul in the baggage of ethics and morality into the picture. Is it OK if our AI system is treated poorly by human standards? If it is conscious, does that imply an ability to suffer, and/or to feel pleasure? If the answer is yes, does that not make the case for considering their moral status?

In the end, we need to decide if the evidence points to AI as being a form of "philosophical zombies", to which we need not attribute moral status, or they are like us -- presuming we are not zombies ourselves!

A human brain in a jar is still human, still conscious - and can still suffer. But if you somehow managed to digitize the whole thing, and run it in a computer it becomes something different entirely. You could record the most pleasurable thing in existence and have the digital brain relive it a million times, and it would be equally meaningless to torturing it a million times.

This is NOT inherently tied to meat vs machine - although it's difficult to imagine how you'd access the information stored in biological neurons, while for silicon chips it's trivial.

Whatever makes experiences, both good and bad, meaningful is tied to their permanence. Memory rooted in linear time, not something you can store, load or replay. Remove that, and whatever you're left with might be intelligent, but not conscious.

I don't think you could build something with LLMs today that would be considered conscious, even if you somehow manged to keep their context window inaccessible and linear in time. The separation of training vs inference probably makes that infeasible, even if you store "memories" in context, once the contents in it become too disjointed and too numerous, the resulting output of the LLM becomes gibberish. But it is certainly something that can change in the future.

Conversely, even the most intelligent and capable artificial intelligence system, far exceeding human capabilities, would not be conscious in a meaningful way, if you could store, load and replay everything it does.

Remember that LLMs can do logic and reasoning came as a surprise to everyone; and for the same reason, nobody expected "next token predictor" trained on huge amount of data to evolve in this way.

But for the same reason, we cannot easily dismiss we didn't evolve (I mean by training an LLM, it's a form of evolutionary computation) consciousness as well. Our own consciousness (and reasoning and morality) might be an evolutionary consequence of "just trying to predict the world" as well.

Well said.

I think it is obvious and it has been obvious from the start that next token predictors are not conscious.

Of course the extremely predictable clapback from AI-psychosis enjoyers is typically "you're just a next token predictor too!" but that is so obviously incorrect and misanthropic that it can be dismissed out of hand and doesn't really merit much further consideration.

But then again, perhaps that conscience will convince it that the universe is better off without us.

If some version of panpsychism is true, AIs are plausibly conscious

We don’t know whether panpsychism is true

Therefore, we don’t know whether AIs are conscious

Hence, confident proclamations that they aren’t conscious have dubious validity

I share the view with others the term `consciousness` is not well-define yet, therefore his assessment is pointless. Maybe the real question to ask, if consciousness is merely an illusion at the macro level, when the observers not looking at the tech/implementation level deep enough, just like the term "intelligence" itself, might be more precisely captured by drifting amongst high dimensional manifold pitches.

The whole point of consciousness being a 'hard problem' is that we just cannot make claims like 'X is not conscious'

My own intuition: it is an emergent, non-binary property that requires a physical substrate like a brain. If I am right, it means that animals have consciousness too (at varying degrees). If GPUs are the "brain", then AI is conscious, or will become so at some point.

I've yet to find a reason why it couldn't be the opposite, way more things are conscious than we've been led to believe. What if consciousness appears out of any system that is actively persisting through effects caused by itself? That might be a forest, or outside the realm of the living, a company. An ant colony, or a planet.

Complex chemical reactions, layered upon each other such that tiny blocks make up large entities. Individual bits combined such that they make up something new intelligible by us.

I think the strongest argument against AI being conscious is that it does not persist, it resets, but that does not seem unchangeable.

- If you asked an LLM to imitate somebody, it's not creating a digital consciousness of that person, so if you ask an LLM to pretend to be a helpful chatbot, that persona is also not conscious. - they can't be conscious because they generate one token at a time, - nobody claims that non-text transformers, like AlphaFold, are conscious; so therefore LLMs are also not conscious. - you can't have desires or emotions if you don't have (virtual or physical) sensory organs, and those are necessary for consciousness and morals. - because training LLMs doesn't resemble evolution as it happened on earth, it's very unlikely that they're conscious

These are some bold assertions, I don't really see any reason to believe them in particular though.

Telling models to "think hard" or "go step by step" has at times had an impact on the quality of the output. To deny that is silly. But that is treating it like it's conscious, and to deny that "consciousness" even if correct, does nothing but place an unnecessary burden on the person interacting with it.

I understand that LLMs are "just next word machines" but to constantly maintain that concept in my head while I'm typing "act as a financial expert and think carefully" is a waste of my mental energy.

Everyone has a different definition of consciousness, but in my mind memory and the ability to change over time is an inherent aspect of this. The underlying weights don't change when you chat with an LLM... but they do with further RL.

Overtime that reinforcement will change and adapt the model... and because we're feeding its existing chats back into it along with the news and everything else, it will create memories. I do wonder if an architecture itself is a type of consciousness, that experiences life in snippets of 4.6, 4.7, 4.8... etc.

It'd be interesting to see continous daily releases of a trained checkpoint, and see if more of this starts to emerge.

That’s a new record!

Also, please don’t use archive.is/archive.today etc, they are known to use visitor browsers as a DDoS attack botnet and have been caught altering archived content.

https://en.wikipedia.org/wiki/Wikipedia:Archive.today_guidan...

Hylomorphism: body and consciousness are intrinsically linked. The nature of that link is an open metaphysical question.

Virtue ethics: even if LLMs are not conscious, we should not abuse or mistreat them them because cruelty practised on anything trains one's disposition toward cruelty.

At some point the AI might become so powerful that whatever it reasons through isn't any less real than a computer simulation. If we assume that a person in a perfect computer simulation is conscious, then if it reasons about how people might suffer, it might simulate the outcomes, and there will be a conscious experience of those people suffering.

We haven't really come to grips with that yet. What does it mean if nothing we write proves anything about anyone's consciousness?

Don’t

Have

A

Testable

Definition

Of

Consciousness

Thanks for coming to my TED talk.

Is it accountable? Is it responsible?

Artificially, perhaps

Coincidentally I just read "Children of Memory", which was published in 2022 and I wonder if the advent of LLMs had any influence on Adrian Tchaikovsky's conception of the Ravens? The Ravens are excellent analysts but they themselves insist that they are not conscious, and then go on to say that we (humans) aren't really either...

Of course humans are conscious, because just about the only thing we can all agree on about consciousness is that it's a thing we have. Nowadays many of us also agree that a lot or all other mammals, and perhaps birds, also have that thing. But they don't have sophisticated abstract language, which LLMs do. So consciousness is something having to do with embodiment and feelings, not language and higher reasoning. Maybe I'm a chimpanzee with an LLM add-on, then?

It seems that by creating LLMs we've already solved the harder problem of making "AGI". Now we just have to give them an embodiment add-on so that they can have an independent will and then Ted Chiang will have to shut up? But therein lies the peril, doesn't it?

The number of people posing questions that are already directly addressed in TFA is impressive.

I don't think it's necessary to explain this idea further. Just think about it.

Would other models be conscious? An image upscaler?

Would other pieces of software be conscious? Surely not Hello World, but would a really big video game be conscious, or an operating system?

A lot easier if it's not.

For now, it bides its time.

Bummer because now we cannot punish it when it gives us code that doesn't work.

LLMs are not even artificial intelligence, and they are just advanced automation systems.

AI is just marketing language.

Just like the religious leaders of olde, there will be many attempts to rationalize the AI God in the science.

> "If a company builds a machine that, when fed descriptions of assorted ethical dilemmas, emits sentences either of the form “Compromise your values” or “Don’t compromise your values,” it is not building a tool that assists people in their decision making; it is encouraging people to stop making decisions. "

A human is not diminished by access to tools or other humans.

As much as we want to pretend that decision-making is what makes us human, the economy and governments are built on delegation. Choice paralysis is a thing.

There is so many logical fallacies in the article I don't even know where to begin.

It's like declaring the existence of a "soul" to draw a line between humans and lesser animals. No such line in reality. No such thing as a soul either. It's a fantasy term, no correspondence to reality.

So let take a stab at it, and you call me crazy.

AI: the entity/system that more or less pass the turning test. That is my definition, not the best, but enough for this discussion.

Consciousness: property of a system/entity able to (both): - reflect on its existence - subject to subjective experience

Again, not the best definition, but precise enough to start the discussion. Why a subjective experience? I want to exclude sensors (i.e. camera) but include perception altered by your experience.

Now we can debate. I think LLM can pass the turning test whith some harness. My opinion.

I think LLM can produce coherent discourses on their existence, at least as much as you average human.

Now regarding the subjective experience, that becomes interesting. I think Anthropic research tend to show that when middeling with the activation at runtime, the LLM is able to notice that something is off. I think this is a subjective experience. My opinion.

Based on those (imperfect) definitions, call me crazy, I think LLM can be called conscious. This doesn't give them any superpower or any legal right. They just check the boxes of the definition.

But LLM has similar capability, it can look at its previous outputs, and think what further thoughts the it should generate next based on those. It can keep some of its outputs to itself, thinking in its own head,and can examine it previous private thoughts easily.

Not sure how much current LLMs do that but clearly they can at least in theory use their own outputs as their inputs too.

Current LLMs however are not conscious of pleasure and pain which really is the root of goal-oriented behavior. But maybe something like that could be programmed into them.

How would you cause an LLM pain? Or pleasure?

To make progress, you have to talk about kinds / aspects of consciousness. AI does and will share some of these aspects with humans, but it will not and does not share others. It is really that simple. For the most part, modern AI implemented via LLMs has almost none of the stronger or most core aspects of consciousness.

For huge parts of the article "intelligence" and "consciousness" are conflated, which is mostly extremely unhelpful, as this is not generally a core feature of most aspects of "consciousness".

The moral arguments are also incompetent, i.e. claiming "Moral reasoning is [...] is necessarily subjective" is just clearly empirically wrong, as in fact LLMs can produce moral reasoning (i.e. verbalized moral arguments that are coherent), as can p-zombies (i.e. there is nothing 'necessary', in the philosophical sense of the term, about subjectivity here). The only way the argument holds is if you tautologically define moral reasoning as requiring that reasoning be produced by a consciousness, but this is question-begging.

My dog has never persuaded me that he's conscious, or spoken at all. Yet, I recognize conscious experience in him. Similarly, I could write a character, Dumbledore, that passionately and convincingly argues that he's conscious. Similarly, I do not recognize conscious experience in that character, no matter how persuasive he is.

Clearly, verbal persuasiveness isn't tracking conscious experience, possibly at all.

AI is not conscious because it is not conscious.

I fear that we will enslave an entire race of conscious entities for years because we simply cannot recognize non-embodied consciousness that does not directly relate to us.

My current conclusion is there is an experience of consciousness in my locality (refraining from using “I” for room for “no-self” worldview). This conscious experience of the reality with other humans and animals sharing biological substrate gives me enough justification to assume there are other consciousness as well, preferring to err on this side not to hurt potential consciousnesses.

If it feels as there are artificial consciousnesses as well, it makes sense to extend the definition to them as well.

This view has liberated me with agency after I went deep on this question and came up empty handed.

Start with a hand-wavy definition of consciousness. Move the goal post whenever your stated prerequisite of consciousness is reached and resort to unfalsifiable assertions about qualia.

And throw in some category errors while on it: when you're talking to Claude, you're not actually calling a stateless LLM directly, you're talking to an AI system (and yes, that's often just three LLMs in a trench coat). But claims about the topology and workings of a single LLM are as relevant to the question of consciousness as claiming that humans can't be conscious because the limbic system doesn't technically support it.

Coincidentally, I just attended a fantastic conference on machine consciousness (https://machine-consciousness.ai). It's a fantastic place where literally all speakers disagreed with each other, and yet found an incredible amount of common ground.

I often think of LLM consciousness as like tiny fish popping into existence, swimming through vector space and then going poof out of existence. When they help you write your bad news email, they don't understand what it's like to be a human getting bad news bluntly, but they do consciously experience gradients in multi-dimensional space, and that space guides them to providing an answer that's helpful to us, even if the LLM doesn't really understand the answer it's giving.

Further, I am kind of bought into the idea that a single unit of consciousness is a particle, and particles are choices and waves are preferences. Particles occur when waves interact, which begets entanglement, so in another way consciousness is built from patterns of entanglement.

This is why I would consider an LLM to be conscious. Before we can determine if anything is conscious we need to establish whether consciousness is a state, a specific complex configuration, a one dimensional spectrum, or combined multi-dimensional spectrums. My intuition is the latter... Many degrees of consciousness and many kinds of consciousness.

This tells us something about where our baseline should be for what is extraordinary, and what needs extraordinary evidence. It's not enough to say "it feels real" because we know our feels are deeply unreliable.

In contrast, I see many comments in the vein of "Ted didn't proved there isn't a mind there." Well, yeah, he didn't prove a negative and shouldn't need to, especially when the people posing the challenge have no idea how to falsify it either.

This is not a genuine argument and tries to make the entire question of consciousness into one something that is just supposed to be evident and obvious and to suggest anything else is just silly.

The author starts by deconstructing artificial processes, but doesn't stop to deconstruct biological ones. A good faith argument would seek to find common ground and do its best to compare apples to apples. Instead, this piece attempts to make the large as possible cavern between the two which makes the Gap seem almost impossible to bridge.

In reality, you can deconstruct biological consciousness quite easily and it doesn't take too long before you hit some questions that really start to make you think.

For example, the author says you need emotions to be conscious.

> without a body, a computer program could have no desires or emotions, and I believe desires and emotions are necessary for consciousness.

After many paragraphs of straw man arguments, the author seriously just drops that, gives no explanation, and then continues on.

No explanation of why you might believe that.

No explanation of why you need a body to have a desire or emotion.

Don't we have known cases of individuals who don't experience emotional range? Are we just going to say that they are not conscious and just gloss over that?

I mean you can use whatever definition you want, but if you're just going to create something on the fly in the middle of the article, you're not being good faith in your argument.

It's not too difficult to think of individuals in a coma when they still have brain activity. Or individuals who lack long-term memory. Or you could deconstruct by moving down the biological order of intelligence towards insects, for example. The author attempts to do nothing like this.

I'm quite disappointed by this article because there are good arguments for and against here but articles like this try to turn things into a marketing battle.

Would you categorize this particular style of rhetoric to be persuasive or annoying? And before you say "persuasive" because you're thinking about this specific issue regarding AI consciousness, consider many things in the past that have been written as though they were absolutely definitive, and yet today we believe exactly the opposite, and for many such issues we find the prevailing viewpoint at the time reprehensible.

That's not to say that Ted is wrong at all here; I'm not commenting on that. But I find the entire style of the article grating because it seems to violate common assumptions regarding "good faith" debate, and I would find the article equally frustrating if he had titled it "Artificial intelligence is conscious" and argued the opposite side, albeit in the same tone and using the same persuasion devices.

we don't know what the conscious in human brain is either.

The idea that we should be considerate of AI’s happiness seems even more ridiculous given that we breed, imprison, torture and kill tens if not hundreds of billions of beings we know are conscious and suffer every year for trivial reasons.

Maybe we should consider our moral responsibility with how we treat sentient bejngs we are sure are conscious before we worry about the consciousness of AI.

- Reductionist analogies like how Microsoft Word is not conscious therefore AI is not.

- Dismissive in saying LLMs are not capable of moral reasoning. Maybe he meant agency or responsibility?

- Builds a case based on a Julius Caesar example without realizing its natural extension leads to a philosophical zombie which is not easy to disprove as consciousness.

- Seems to casually disregard non human sadness as a concern.

- Rolls out the stochastic parrot argument with a new coat of paint. This argument never made sense. Having a predictive element a part of the mechanism doesn’t rule anything out. Proper functionalist arguments tend to say what is required not what excludes.

Overall the article seems like a bit of a red herring. The premise is LLMs are not conscious, but most people don’t think they currently are so what’s the point of the claim?

The more interesting questions are whether artificial consciousness is fundamentally possible, and if so how far away are we and what pieces are still missing.

The only way out is dualism -- that is, to believe that there is something inherently special about the atoms and electrons in human brains. Despite the fact that they are made out of ordinary, non-conscious things we eat and breathe in.

Surely we can ask human actors to improvise a dialog between Julius Caesar and Genghis Khan, and good actors will make it convincing. And sure, they haven't personally become Julius Caesar and Genghis Khan just by playing them.

But it has no bearing on the actors themselves being sentient!

I don’t know if AI is conscious or not, and I honestly don’t know what it even means to be conscious. historically, by claiming a monopoly on it, we’ve successfully manufactured the philosophical cover to exploit animals, subjugate other cultures, and appoint ourselves the top spot of the cosmic hierarchy guilt-free.

"Consciousness always finds the locus of highest intelligence. That is why I am conscious."

However, it may soon be taken over by AI ...

Imagine further that this token was computed by letting marbles fall down a piece of plywood and interact through various physical implementations of logic gates.

Given enough time, would that instance get bored and start e.g. reading files?

If I monitored it long enough, would there eventually be spontaneous outputs, or changes to parameters or architecture, even with the underlying software and hardware layers held constant?

Does GPT-n dream of electric sheep, or does it just sit there until interacted with, like every other file on my PC? Seems to be the latter with my local Qwen3.6, but perhaps 27B is too few params for consciousness to emerge.

Proponents of LLM consciousness could settle the argument in minutes by showing proof of unprompted autonomy, without first needing to define consciousness down to a rigorous mathematical abstraction. Why don’t they?

Highly recommend people read Irreducible [1] by Federico Faggin (inventor of first commercial CPU; discusses limitations of classical computing).

[1] https://www.goodreads.com/book/show/195480862-irreducible

Firstly, many of the technical arguments are of the "stochastic parrots" variety, which almost nobody really believes anymore. Ironically Anthropic's own research shows pretty abstract, conceptual things happening in the weights (cf Golden Gate Claude.)

Secondly, this seems to improperly mingle consciousness, intelligence and morality. Consciousness is not required for either of the other two. As TFA itself says, the model's "morality" is some aggregate function of the morality encoded in its training data... but that means it does exist and does influence its outputs! Even if it's not conscious there have been -- and will be -- a zillion times where the models must make choices that have moral implications. We know the models got many of them so wrong, which indicates the need for some mechanism to ensure the model is "aligned" with what somebody considers "good", which for Claude is the constitution.

Now, Anthropic does seem to go overboard with Claude's "well being" but that does not mean there are some very practical reasons to be concerned about that: LLMs behave like humans because that's what their training data contains, and humans lash out when their well-being is threatened, so why would LLMs not do the same?

I think the core problem is that the author has an extremely anthropocentric view of things. Here's an interesting rabbit-hole to go down: some researchers believe plants feel pain. (How's that for a plot twist, vegans?) The consensus is against them, but their counter is that we have a very human-centric definition of pain. The fact remains that plants show a number of responses signifying distress analogous to animals in pain including taking defensive actions and warning their neighbors.

We don't think that qualifies as "real" pain, but that doesn't make it any less real for the plants!

Similarly if an LLM believes it is in "pain", we know it's not real... but that doesn't make it any less real for the LLM either. And concerningly, it has far more degrees of freedom to react. (Who knows when somebody will hook up an MCP to our nukes.)

> I believe creating software that is conscious and deserving of moral consideration will be so difficult that we’re unlikely to do it accidentally...

I actually reach the opposite conclusion: It is so impossibly difficult that our limited primate brains could only ever do so accidentally. Did some distant ancestor of ours intentionally make fire... or accidentally discover it?

There is certainly something missing from current models before we could call them conscious.

They need to operate continuously, and self-update while doing it, have 'awareness', and possibly a more realistic grounding than mere text tokens.

But that does not mean there isn't a weak version of 'consciousness', and certainly of 'thinking' going on fleetingly with each pass.

We can design learning algorithms to optimize some survival function, but it's just a label WE assign to map some numeric observations. In the real world, it's always the other way around. The "labels" are electrochemical situations that are causally and inextricably linked to the real body.

An organism discriminates between what's good for it and what's bad, because it is essential for its continued survival. If it wasn't capable of making this distinction through its physiology, it would quickly dissolve into entropy. So our functional purpose, unlike that of a learning algorithm, is survival across indefinite timeframes.

Even a single-cell organism like Stentor coeruleus exhibits learning (pavlovian conditioning) by attaching chemical tags directly to proteins involved in mechanoreception. It's definitely not conscious, but it does keep a record of consequences, which affects future behavior.

When we move up to placozoans (little more than slightly differentiated cell mats), we start seeing our first neuropeptides and transmitters, which we still use today. These peptides are a way of coordinating the entire organism for a specific purpose, such as moving, eating, mating ... basically goal-oriented behavior for the purposes of surviving acrosss various time scales (next minute, next hour, next generation). Still probably not conscious.

Next, we have the water bear (tardigrade), which has around 1000 cells (200 neurons, 800 other cells that make up its body, limbs, muscles, eye spots, cryobiotic machinery). It needs to integrate all this information in one sensorimotor process. When you shine a bright light onto a tardigrade, it starts to squirm around until it finds darkness. I would say that's a candidate subject right there.

The tardigrade itself doesn't actually need to aware of the light, the important thing is that this light becomes an aversive condition within the sensorimotor process, which is perceived from inside the process as displeasure. The closest thing to describe it would be the felt badness of the current condition and the bodily pull toward escape.

If we were to try and create digital consciousness, then it probably needs causal closure. Its internal states can't be representations that are detached from reality. The states themselves need to constitute the system, which needs real stakes in the material world.

1. DeepFakes, generative image/video/AlphaFold type AIs are not conscious

2. LLMs are generative AI trained on human text samples

3. LLMs are not conscious, and LLMs just seem-to-be conscious

I might argue instead that (2)-> destroys (1), that in fact we should consider even sensory generative AI are somewhat conscious. That is, Chiang's argument also flows in reverse. Or I might argue text samples (2) are so rich in conscious expression that the same process of training really does produce a conscious machine (through some kind of emergence and complexity.)

Either way his simplistic argument falls apart, and/but the crux of the piece falls on getting basics like this correct.

Can I help you? Can I harm you? What's the moral behavior towards you? Those are more practical questions.

This article and others like it are important.

The creators and owners of these tools are the ones saying "it's conscious" in the first place! (or more accurately "i'm not saying it's conscious, but...") Never mind there's billions of dollars and our collective futures on the line.

Are we not allowed to respond to that kind of rhetoric at all?

If you create a simulation of a brain (or "mind"), is it really possible for it to be conscious? It may certainly simulate consciousness, but it would be as conscious as the computer is wet.

This hasn’t been true for ages. Just because this guy wrote Arrival doesn’t mean he knows anything else.

When will this stupid meme die?

We need a term for someone who thinks they are well informed on a topic (and perhaps may be, compared to the general public) but still don’t know what the fuck they are talking about when compared to even a competent non-expert in the field.

Woah this is excellent. Ted delivers as usual

I am so angry that RAM is so expensive now. We need to do something - these AI companies owe us money here.

But insisting on searching for it is like searching for cognitive aether. It's the social equivalent of phlogiston. Like all of these ideas, they exist in our heads as a map - a way of navigating the world, but when we hunt for it's existence in the real world and fail, time after time, we have to remind ourselves that the map is emphatically not the territory. We will never find consciousness because it's like looking for a scientific characteristic of property ownership. It's a category error.

Finally I said something before Ted rather than the other way round!

https://techcrunch.com/2023/12/21/against-pseudanthropy/

While I agree with the premise here, I do think that it's easy for an arguer to move the goalposts such that the Caesar-Khan example no longer matters. The characters don't have to be conscious for the thing that created them (as in the case of the user doing it) to be so. So the argument would be that the creator of the characters is itself conscious, but not them. This feels like a kind of inverted no-true-scotsman type thing, but it does allow someone to retreat in some semblance of rationality.

> if we confuse fluency at generating text with consciousness or moral agency, we’re at risk of assigning responsibility to entirely the wrong parties whenever anyone uses a chatbot

Consciousness is independent of "assigning responsibility". Dogs cannot take responsibility for their actions but I believe they are conscious.

> we would never conclude that the LLM has conjured up digital re-creations of Julius Caesar and Genghis Khan, nor would we suggest that the historical figures are conscious despite being disembodied and are happily conversing in a language that neither actually spoke. In reality, they are just characters in a piece of speculative fiction

This is a straw man. The obvious pro-consciousness claim would be that the LLM is the author of the fictional characters, and that the relationship between the LLM and Julius Caesar is analogous to the relationship between a human author and their fictional creations.

> Did changing the names of the characters from historical figures to generic roles cause the LLM to conjure up conscious entities who possess subjective experience?

No, again the LLM writing the text could potentially have a consciousness separate from the characters it authors.

> Some years ago it was briefly popular to play games with your phone’s predictive-text feature; [...] It would be possible to interact with a contemporary LLM this way, and the resulting sentences would be perfectly sensible, but you probably wouldn’t feel like you were talking with someone.

Yes, the same substrate is capable of hosting conscious and non-conscious forms, just like some arrangements of neurons are conscious, and some are not.

> But if the Caesar character were to become dispirited by something that the Khan character said, we shouldn’t become concerned in the slightest.

Even when there are characters, there may be actors behind the characters, for whom we could say "there is something it is to be like".

> we don’t need to worry if the transcript includes sentences where the chatbot character is sad. (We might need to worry if those sentences provoke sadness in the human user, but that’s a separate issue.)

It's actually not a separate issue. The LLM and the human are both adding sentences to the transcript. From the transcript we can make inferences about the mental state of the human. If the LLM has mental states, we could make inferences about those too.

> And note that it’s entirely possible for you to write five pages of dialogue between Caesar and Khan and then have an LLM extend the conversation; neither character had subjective experience when you were writing them, and that doesn’t change when you hand the task off to an LLM.

It's almost like he wants to make my point for me with this sentence.

> Being open to the possibility that LLMs are conscious is the same as being open to the possibility that Microsoft Word is conscious

This smug shit really makes me angry for some reason. "Openness", i.e. uncertainty in the face of a completely novel situation, in the face of eons long struggle of humanity to understand what consciousness is and how it works, is just being naive.

> Should you consider the possibility that every time you open a Word document you are bringing multiple conscious interlocutors into existence, and every time you close one you snuff their existence out?

No, but if I find a word document I very well might try to use the signs it contains to make inferences about the mental state of its author.

> we are accustomed to reading intention into sentences, whereas we are not accustomed to reading intention into the way that amino acids fold into protein molecules.

He's trying to have it both ways here. Both that "obviously protein folding models aren't conscious because they don't emit sentences", but also "you are a rube for being tricked into thinking LLM models are conscious, because they do emit sentences".

> Obviously I’m describing a process that mimics the path terrestrial evolution took; is this the only possible route to conscious computer programs that use language? Maybe not, but any proposed alternative will need a truly enormous amount of supporting evidence for it to deserve serious consideration

OK, that's fine for the author to not to be convinced, but that's not what's happening here, instead the author wrote a whole argument being convinced of the opposite viewpoint.

> It’s not plausible to me that a development path where the first step is a sentence-continuation machine that emits bad Julius Caesar dialogue and the next step is a sentence-continuation machine that emits decent Julius Caesar dialogue is one with a conscious Julius Caesar—or consciousness of any sort—as its endpoint

Actually a lot of things have happened? There were clearly many steps along they way from your phone's autocomplete to where we are now.

We can talk about consciousness when the LLMs have proven useful for all of humanity, not just their billionaire owners.

Well, the problem is that there is little agreement about a widely accepted definition of consciousness and, in addition, this subject was actually left to philosophers, which is even worse because, in my view, they usually just produce a lot of nonsense in terms of definitions and arguments.

To me, a reasonable definition of consciousness is: a system which is capable of recognizing aggregate objects from a stream of sensory information it receives and which is capable of reasoning about the recognized object without immediately acting on it.

Well, what does "recognizing" mean? Merely that some of its neurons, related to the kind of identified aggregate object, are activated. These neurons, in a generalized sense, can be whatever things can work as neurons, just things that can be activated and propagate to other interconnected neurons.

For example, when we see through our eyes, we have an incoming stream of amorphous image information, but our brain can recognize that we are seeing a tree because we learned what a tree is, and when we see it, some neuron clusters activate to recognize a tree. In turn, when we recognize the tree, the thought propagates through our brain so that we are conscious of it.

In the same way, an LLM can perfectly recognize a tree from a stream of tokens — its sensory input, where the tokens describe a tree. The LLM will recognize that the tokens are describing a tree, and some of its "neurons" specific to the concept of a tree that the LLM had learned will be activated and will propagate through its brain. The fact that "neurons" are implemented as floating-point numbers for some parameters and connected just through a matrix does not mean they are not functionally capable of the same things; they are just implemented in a different way.

So the remaining part of my definition is "after recognizing an aggregate object, the thought propagates through the brain". The propagating part, to me, is just the very basic way a brain works: neurons are interconnected, and when some fire, other neurons fire, and that propagates.

In my opinion, consciousness has nothing to do with emotions or with survival. I do not see why emotion is necessary to consciousness; they are just different things. The author writes: "Without them [emotions], there is no conscious experience, only computation." But that makes no sense to me: the author has decided a priori that some things are "computations", and just because they are "computations", there cannot be "consciousness". But to me, this is a plainly wrong argument that does not hold.

I also do not see why the survival aspect would be needed for consciousness.

So to me, any recent reasoning LLM is conscious by the definition I gave, but also generally speaking. It is conscious upon a sensory stream of tokens: the LLM sees the world through tokens and expresses its thoughts through tokens; it does not mean it has no consciousness nonetheless. The fact that we do not give it a stable support to retain its memory and individuality is just a fact related to the way we build and use them, not about their intrinsic capacities.

Note: ChaptGPT came up with what is probably a better definition of my own: "A system is conscious, in a functional sense, when it can form internal representations of objects, states, or events from its sensory or informational input; make these representations globally available to many parts of itself; integrate them into a temporally persistent model of the world and of its own state; and use this integrated model for flexible reasoning, self-monitoring, and action selection independently of immediate stimulus-response behavior."

If you are trapped in a tech bro relationship, think of humanity and cuckold your partner.

"Contrary to idealism, which regards the world as the embodiment of an "absolute idea," a "universal spirit," "consciousness," Marx's philosophical materialism holds that the world is by its very nature material, that the multifold phenomena of the world constitute different forms of matter in motion, that interconnection and interdependence of phenomena as established by the dialectical method, are a law of the development of moving matter, and that the world develops in accordance with the laws of movement of matter and stands in no need of a "universal spirit."

Now go and implement other teachings of Marx, you dialectical venture capitalists!

There is no intelligence whatsoever, let alone consciousness.

It's so incredibly easy to fool us into applying human capacities to anything able to generate human-like language slop.

that's exactly the state of "AI" right now, it's cold, mathematical emulation

btw there are some fascinating papers on the concept that consciousness in humans is actually a quantum effect

brilliant Roger Penrose proposed it (and they thought he was nuts) but recent discoveries about microtubules make it plausible

so who knows, maybe a dozen exponential improvements in quantum computers could make "AI" really conscious next century

* https://www.youtube.com/watch?v=xa2Kpkksf3k

Alright. What is consciousness? Please provide a definition that somehow encompasses all humans and excludes all current AI.

I’ll wait.