One thought on your nicely illustrated "key observation [is] that neural networks tend to place features along directions": my guess is that the neural net was TOLD to behave that way by choosing e.g. Cosine Loss?

For readers interested in this, let me point to the somewhat similar "Activation Atlas" interactive paper, published in the sadly now-defunct scientific journal Distill.

https://distill.pub/2019/activation-atlas/

I never got that far, but I once built a little page that just visualized embedding vectors generated from a few hundred cat, dog and plane photos as heatmaps. I used this for demonstrating to database & storage folks what embedding vectors physically are, at low level. The point of the heatmap (of different vectors) was to show that there are visually observable "vertical bands" standing out when plotting out many embeddings of the same types of objects (like different cats with different backgrounds) in a single heatmap.

I then also took a single cat photo, rotated it by one more degree 360 times and created a heatmap of these vectors to illustrate the point of what the embedding models really detect (you have to uncheck the "Normalized" checkbox in the "same cat rotated by 360 degrees" page to see the vertical bands show up).

The web-app is here:

https://tanelpoder.com/catvector/

One thing I still struggle with in my head is how these vision embeddings can then be used to give LLMs eyes.

Because you somehow need a giant training set which describes images in natural language, no? Is that actually how it works, or is there some smart trick so you don't need to pay labellers a bunch of money to look at pictures and describe them.

Mechanistic interpretability is a fun topic to "play with" (good title there). I recommend watching videos featuring Neel Nanda or Chris Olah