Home | Search

2026-02-27

Update

• Epoch AI says distributed training on household GPUs will soon be competitive with large centralised training runs

I am aware this is not a problem for today, but for the next few years.

• I am aware that the current codebases for distributed training runs are shit.
• I am aware as of today it is maybe not that hard to just go find the person coordinating the training run via social media, and then go arrest them.
• However, I expect in the near future, it will possible to anonymously start a training run, anonymously rent out your GPUs to the swarm, and anonymously coordinate payments in cryptocurrency. In theory, you can do all this over Tor such that even nation state actors find it difficult to detect, no matter how many systems they have backdoored. There will unfortunately always be some cypherpunk ideology-influenced people who will try to build stuff like this.
• Therefore, studying actual engineering limits seems useful.

2026-02-24

I want data on theoretical limits of distributed training runs

Disclaimer

• Quick Note
• target audience - my mind simulation of atleast some technical AI safety/aligmment researchers I have met

I would love to read an article on distributed versus centralised training runs. Most of the publicly available data is on the sizes of centralised compute clusters, and sizes of models trained on centralised compute clusters. However, if we try to regulate those, some researchers will attempt to do the same training runs on distributed compute clusters.

I want to understand what are the limitations in terms of fundamental physics on how much slower a distributed training run must be compared to a centralised training run. There are also limitations like the fact that these codebases are less mature, but eventually I can see researchers overcoming those. Actual physical limitations like how network bandwidth is worse than memory bandwidth in terms of ping or synchronisation or similar, would be useful for me to read about.

Subscribe

Enter email or phone number to subscribe. You will receive atmost one update per month

Comment

Enter comment