Browse parent directory

Date: April 4, 2025

ML Is Not Geometry

I have a hunch about ML. I want to see if it is right in a few years. I do not plan to work on it now.

Note: This idea might help speed up ML a lot. That could lead to bad things for the world. Please tell me if you want me to take this post down.

Main Thought

• We might view ML in graph form, not in a shape-based (geometric) form. This is due to the "curse of many dims."
• 1.58-bit or 1-bit models might train faster than fp8, if we spend the same compute.
• The drop in accuracy might not be big.
• There could be a graph-based way to beat backprop. We might think of weight shifts in bit steps (Hamming space) rather than in smooth steps (geometry).

Signs That This Might Be True

1. 1.58-bit quant works well, at least for inference.
2. Graph-based search tools (like HNSW or diskANN) do better than shape-based ones (like LSH or k-means or scANN).
3. ReLU is better than all other activation functions so far.