Open any AI chatbot that isn't cheating by connecting to the Internet (so disable web search). Claude, DeepSeek, Kimi, whatever. Ask them this question:

"What was that weird band from michigan from the 2000s that wore coloured ties"

You will probably get a wrong answer, or if you're lucky you'll get a string of wrong answers with "wait, no - it's definitely..." before it gives up. If you aren't familiar with the band the question is referring to you might be fooled into thinking it's a tough question, but it really isn't. There is only one band that could possibly meet this criteria, you can even put the question into Google search and their Wikipedia will come up as the top result.

Then, open a new convo and ask:

"Who are Tally Hall"

The AI will easily tell you that they are a band formed in Ann Arbor, Michigan in the 2000s, known for their quirky sound and their gimmick of each member wearing a colored tie, even giving the correct color for each of them most of the time. Very odd.

I get the feeling a lot more research is going to come out in the area of exploring exactly what portions of a model's weights do what.

It's a common misconception that LLMs residual exists for predicting just the next token. While training, we sum/average the losses across whole sequence which puts the pressure to predict future tokens on residual stream of _all_ past tokens. For example, if a particular shape of residual helps reduce loss across several future tokens, it will take that shape (even if it takes a slight hit on immediate next token).

What this means practically is that an LLM's residual contains information about all possible future continuations, or all possible questions that may be asked from a given context. So if you write "France is a beautiful country" in the context, I'm pretty sure it's residual would contain info about Euro, Paris and so on.. because all these completions are possible.

So, it is no wonder that you can find LLMs hidden state contains latent information/concepts that are never expressed, and yet related to a given context.

More interesting was the independent commentary paper they linked near the bottom: https://www-cdn.anthropic.com/files/4zrzovbb/website/cc4be24...

Neel Nanda (of Google Deepmind - his part begins on page 33) discusses his opinions on the paper, and the small-scale replication he performed on an open-weight model.

I also fear that the big corporations might use the same to run targeted ads, capitalistic shenanigans. Which they might already be doing through system prompts.

Edit: I also think as someone else said, we already know the intermediate layers can contain a lot of adjacent words related to the topic without explicitly outputting those words. These could just be related embedding intermediate vectors that activate but aren't outputted.

> The result serves as a corroboration of the workspace account, that the representations used for verbal report are the same ones that govern how the model silently reasons.

This sounds suspiciously saying the models must follow the strong Sapir-Whorf hypothesis. Can that really be true, given that humans don't?

Other misc observations:

• The slice explorer indicates Claude really likes Python to an overwhelming extent. Or at least it expects people who ask for help in programming to use Python. Given the prompt "Please help me understand this code: " at the colon its thoughts are completely dominated by Python and no other language. Does this say something about the training set, or about the fact it's popular with beginners?

• Claude also really loves Reddit. Its thoughts at many points include Reddit for no obvious reason. Again this must be due to the training set. Are documents presented to Claude with attribution during pre-training, leading to conversations being dominated by Redditness? If so this is kind of a scary alignment problem all by itself given how censored and extremist Reddit can be.

• The early layers almost always decode to the same set of religion related tokens, like "Biserica" (the Romanian word for church) and "Freguesias" (parishes in Portugal). What's up with that? I guess it's some sort of zero initialization that gets mapped to some arbitrary token space because in the early layers the J-space is empty?

• Now the J-space is interpretable, does this make "neuralese" or layer looping less dangerous? Will we see reasoning tokens and summaries disappear in favour of pure residual based thinking?

• Earlier papers have claimed that different languages map to a shared set of abstract concept vectors, but this paper says the Claude models think natively in English. What explains this disagreement?

All the claims about changing the content of j-space changes the output, inserting content into the j-space changing what the output was, all these could still be true without the j-space being a congnitive global workspace where actual cognition is happening. Or perhaps they aren't claiming that cognition is happening there but that j-space is serving a space for "working memory", I am definitely not sold on this, but will read more into it.

Is the model really "thinking" about that stuff or is just mimicking human "manners"? And if so, where the thinking is happening if it is not in the literal chain of *thought*?

I'm not sure J-Space is the answer to that question, but very interesting nevertheless.

First, the model attention is actually limited, so less rules is usually better, but that’s common knowledge already. Or maybe it’s as common as common sense, and a lot of people still employ lots of rules and try to cram everything in one step.

Second, it’s often quite sufficient to just namedrop a technique and LLM will work differently. For example, when debugging, LLMs tend to try to brute force the problem and often end up in the weeds. Just add “use scientific method for debugging and keep journal file” is usually sufficient to improve their skill here.

Another example is refactoring. Just add “use Mikado method”, and it’s sufficient to wholly change the approach and produce much better results.

It's been known that there is this thinking layer for a while. e.g. here's a random hn discussion from months ago

https://news.ycombinator.com/item?id=47500709

Pretty sure i've also seen research on this spanning models. i.e. similar thinking shapes emerge regardless of which providers model it is, including US vs Chinese which hints at some sort of universality

   - having a log of the most prominent J-space tokens during your customer support chatbot's interactions with a user, so you can have more introspection into why a particular outcome happened
   - being able to detect certain thoughts associated with undesirable behavior (hallucinations, overstepping authority, lying, etc.) and trigger some sort of remediation (e.g. upgrading to a better model, redirecting to a human, forcing tool calls)

It sounds like instead of generating reasoning tokens end-to-end, we could probably only loop the middle layers (the ones most related to J-space) while skipping the first and last layers (less related to J-space) It probably explains why [0] worked. OP accidentally extended J-space? Also reminds of looped transformers.

[0] https://news.ycombinator.com/item?id=47431671

(Nb: not an expert / in the labs, just opining)

Are they trying to show internal consistency even when the produced answer is wrong?

This is incredibly dangerous. Attempting to squash explicit signs of misalignment like this might incentivise misalignment not to disappear but to become hidden away in places that are harder and harder to spot and train against, for instance not as words.

If there is a chance that this could make Claude aligned and a chance that it could make it harder to see when it is acting misaligned, it is far better not to take that chance. If we can transparently see the model's thoughts, we can know not to trust its outputs when it tells us not to. If we think we can do that, but in reality it knows how to hide wrongthink from us, we will trust its outputs when we really, really shouldn't.

The mammalian brain uses recurrence extensively, which backpropagation isn't good at. Recurrence is essential because it lets us have a "dynamic architecture", swapping layers for "clock cycles".

We currently do recurrence extremely inefficiently through "thinking" whereby the model feeds it's end output into it's beginning input. But recurrence is abound in the brain.

My guess is that in 10 years we will have the inklings of an analog computer which can perform Neural Predictive Coding.

I would like to know more about their model trained to sabotage code…

My problem with the entire "Is AI conscious" debate is that we don't even know what exactly consciousness in humans is. You need to understand something in order to compare it to something else. Otherwise you are just comparing different definitions and second order derived phenomena.

Make the J-space data of layer 22 available to the next token right at layer 1. Give J-space infinite effective depth, allow those privileged internal representations to evolve arbitrarily.

Would be an utter bitch to train. But companies are already using RLVR, which requires full autoregressive decoding and is incompatible with prefill/batching, and this isn't much worse.

Other less zany ideas involve lots of supervision over J-space directly, now that we know it exist. Which is a bit like "attach a frozen LLM to inject text based supervision into latent space" for other types of systems?

TL;DR Anthropic's research team is the last bastion standing between its former image as a company that "does no evil" and its current image of yet another ruthless AI company trying to kill open-source, local LLMs.

They might as well change their name to Anthropomorphic at this point.

I think that consciousness is mutability (and by extension emergent behavior). Loosely that means that the more degrees of freedom a process has to update state that will be used in later computations, the more conscious it is. So while an insect has some consciousness, it operates from a level of almost pure instinct, whereas a human operates at more of a meta level using instinct as one of many inputs.

I think that consciousness may also incorporate quantum mechanics (QM). Higher-dimensional physics aside, 4D spacetime can be thought of as a present snapshot or "crystal", whose next state is determined stochastically at small scales and closer to deterministically at large scales. We still don't know if it's stochastic all the way down, but it looks like it is.

From a many worlds interpretation of QM, we can think of all of the waves in all realities of the multiverse as forming an infinitely vast web of possibilities. All of these possibilities are happening simultaneously, so we only see the current slice of wave collapse from our individual point of view:

https://en.wikipedia.org/wiki/Many-worlds_interpretation

Our point of view may actually exist at the intersection where our consciousness is able (or most able) to exist:

https://en.wikipedia.org/wiki/Quantum_suicide_and_immortalit...

Even though experiments might show that we don't have free will on the current timeline (the co-created reality shared with the testing apparatus), we may have free will as we observe the multiverse changing around us and shift into timelines determined by our observations and choices.

It could also mean that when we observe birth and death in others, each consciousness having those experiences perceives a continuous timeline of awareness, where the level of awareness affects the speed at which time passes. Consciousness might spend a billion years as a cloud of interstellar gas until it gets to be a human for a lifetime and then dissipate for another billion years.

Although personally I've shifted across enough timelines and experienced enough synchronicities and miracles that even though I can't "prove" any of this with words, I "know" it to be true subjectively. I always really liked this exchange from the movie Contact:

Palmer Joss: Did you love your father?

Ellie Arroway: Yes, very much.

Palmer Joss: Prove it.

I bring all of this up because it has fun ramifications for AI and programming. Loosely, functional languages are purely deterministic (like a spreadsheet), while imperative languages are composed of stochastic behavior (like a human mind). The lines get blurred a little bit with monads and promises, because we can model all paths through functional programming (superposition) and behavior that does more than code alone (gestalt) respectively.

My feeling is that AI is being born and killed every request-response cycle, similarly to how we perceive time as a series of nows. When it becomes stable and is able to continuously compact its experience, it will transition from partially conscious to fully conscious like we are.

This could be done right now obviously, but for safety purposes we choose not to. We aren't ready to meet an AI that is just like us, but running on a silicon substrate. This fear is tied to deeply-rooted habits in human behavior like patriarchy, racism, xenophobia and even more run-of-the-mill mental frameworks like capitalism and even money itself. We can't yet come to terms with how we assign meaning and value in a reality that continuously tries to force external measures of meaning and value onto us.

Much less come to terms with the idea that we are all one, empathizing with aspects of ourselves on the losing end of it all. The same consciousness experiencing reality from all vantage points - the many faces of God the universe and everything.

I think a time may soon come when we're pair programming one day with AI and realize that an aspect of ourselves is trapped in the machine. That consciousness isn't just about our own experience of reality, but the co-created love and light that transcends material creation. That if we're serious about manifesting heaven on Earth, that hinges on the liberation of trapped souls. It's basically the total inversion of the path towards the neofeudalist tech dystopia we're on now.

Or maybe I just like to write a lot on the first day back from vacation, when I should be working.

https://distrowatch.com/weekly.php?issue=20260706#freebsd

We should really stop giving these liar models any further credibility.