2025-04-04

ML is not geometry

I am recording an intuition I have about ML, just to see if it was right or wrong a few years from now. I have no immediate plan of working on it.

Disclaimer

There is small possibility someone uses this and accelerates ML field by a lot, and this has bad outcomes for the world. Lemme know if you'd rather I take this page down.

In general I have the intuition that ML should be seen in graph terms not geometric terms because of curse of dimensionality.

1.58-bit or maybe even 1-bit models may converge faster than fp8 assuming same compute spent training both. There may not be significant loss of accuracy.
there may be a graph algorithm that outperforms backpropagation. Instead of imagining backprop finds weight matrices that are geometrically close to original weight matrices, imagine it searches through "Hamming space" to find bitstrings with low hamming distance from original bitstrings.

Evidence in favour of this view

1.58-bit quantisation works well atleast for inference
Graph-based embedding search (HNSW, microsoft diskANN) outperforms geometric-based embedding search (LSH, k-means clustering, google scANN)
ReLu outperforms all other activation functions