2025-04-04
ML is not geometry
Disclaimer
- Quick Note
- This post might contain knowledge of dual use capabilities
- There is small possibility someone uses this and accelerates ML field by a lot, and this has bad outcomes for the world. Lemme know if you'd rather I take this page down.
I am recording an intuition I have about ML, just to see if it was right or wrong a few years from now. I have no immediate plan of working on it.
In general I have the intuition that ML should be seen in graph terms not geometric terms because of curse of dimensionality.
- 1.58-bit or maybe even 1-bit models may converge faster than fp8 assuming same compute spent training both. There may not be significant loss of accuracy.
- there may be a graph algorithm that outperforms backpropagation. Instead of imagining backprop finds weight matrices that are geometrically close to original weight matrices, imagine it searches through "Hamming space" to find bitstrings with low hamming distance from original bitstrings.
Evidence in favour of this view
- 1.58-bit quantisation works well atleast for inference
- Graph-based embedding search (HNSW, microsoft diskANN) outperforms geometric-based embedding search (LSH, k-means clustering, google scANN)
- ReLu outperforms all other activation functions
Subscribe
To subscribe, enter "Subscribe " followed by Signal username, email address or whatsapp phone number. You will receive atmost one update per month.
Alternatively, you can subscribe via RSS, or a third-party service like blogtrottr
Comment
Enter comment