FRESH

Hacker News

Matrix Orthogonalization Improves Memory in Recurrent Models

79 points by at2005

by imurray

1 subcomments

Here is a pytorch optimizer that can maintain a matrix as orthogonal throughout optimization:
https://github.com/adrianjav/pogo — POGO: A Proximal One-step Geometric Orthoptimizer
https://arxiv.org/abs/2602.14656 — An Embarrassingly Simple Way to Optimize Orthogonal Matrices at Scale; Adrián Javaloy, Antonio Vergari

by BirbSingularity

3 subcomments

I can't help but think of orthogonal frequency-division multiplexing and it's use in encoding data on multiple carrier frequencies, and it makes me wonder what other parallels we will discover between digital transmission technology for cross-domain stuff like this.

by hasley

2 subcomments

I suspect with "orthogonalization" they mean to find vectors that form an orthogonal bases (same subspace) for the vectors in the source matrix.
I wonder what would be the result if they used a matrix that is orthogonal and closest to the source matrix. Usually one uses the Frobenius norm (root of the sum of all squared matrix entries). Maybe, one could even try another norm that gives a sparser matrix.

by phkahler

3 subcomments

If it can be made orthogonal, can you go a step further and diagonalize it? The storage and performance improvement from that would be huge.

by harveyrook

1 subcomments

Now I’m wondering what is the eigenspace of an LLM? If I take a set of LLM’s with the same number of parameters, then what are the eigenvectors? Do they have different personalities?

by mv_d5339e31

0 subcomment

[dead]