-
Suppose we ended up with a world where even its most powerful organisations cannot keep secrets. Information about values and about capabilities of the orgs will be public.
- Values becoming public means the public can decide if the org represents their true values, and if not, coordinate to shut down the org or build another org to replace it.
- This is control via a more direct democracy than what we currently have, and not control via market.
- Capabilities becoming public means all weapons capabilities will also be open source. This document analyses implications of open-source weaponry.
-
Examples of technologies that will become open source.
- nuclear weapons and ICBMs, underlying supply chains for uranium/plutonium enrichment, missile guidance systems etc
- bioweapons and underlying supply chains for biotech
- AI and GPUs and underlying supply chains
- (hypothetically) ASI, BCIs, human genetic engg, human connectome simulations
- (hypothetically) gene drives, solar geoengineering, etc
- (hypothetically) any offence-beats-defence weaponry, including such weaponry deployable by small groups
-
As of 2025, I am okay with all these technologies being open sourced as long as this is a byproduct of the organisation's other information (such as values and decision-making processes) also leaking.
-
Why is open sourcing weaponry not fatally unsafe?
- Open source nuclear weapons
- As of 2025-05 only 9 governments have nuclear weapons. Many other governments would like to have them but don't. My guess is that the bigger causal factor by far for this is that they're pre-emptively threatened by (some subset of) these 9 governments, not that they lack the technical knowledge and infrastructure to build nuclear weapons.
- As of 2025-05, my guess is open sourcing technical knowledge to produce nuclear weapons is not likely to significantly increase the number of governments in the world that have nuclear weapons.
- Open source bioweapons
- As of 2025-05, most bioweapons require an industrial scale production process to manufacture. This keeps them out of the reach of smaller organisations, and governments are by default aware of and responsible for the biotech supply chains in their countries.
- DNA synthesis machines require an industrial process to manufacture as of 2025-05. Genetically engineered bioweapons can likely be more deadly than bioweapons without it, and these are reliant on DNA synthesis machines.
- Older bioweapons such as anthrax. My guess is as of 2025-05, scaling up these processes to produce sufficient volume again requires an industrial scale process.
- Assuming an industrial scale process is required to manufacture bioweapons, the situation is similar to nuclear weapons.
- My guess is that it is difficult for a government with the required technical knowledge to prevent other governments from also gaining the same knowledge.
- What can again work is international coordination. Either some governments can threaten other governments to stop bioweapons production and R&D, or all governments on Earth can coordinate together to halt bioweapons production and R&D.
- (Hypothetical) It is possible for a small group to secretly use the infrastructure overseen by a government to manufacture bioweapons, against the interests of the government.
- If we had lots of public information on what the infrastructure is being used to build at all times, it would be easier to identify such a group and stop them.
- (Hypothetical) It is possible that one day bioweapons can be manufactured by a small group of people.
- As of 2025-05, I would prefer halting bioweapons R&D worldwide before this scenario is reached.
- Even if this scenario is reached, as of 2025-05 I would weakly prefer a world with open source capabilities and lots of public information on everyone building them, over a world with closed source capabilities held by a small number of organisations (whose values and decision-making processes are also secret).
- I'm assuming a majority of people will still be able to shut down any small organisation trying to build bioweapons, once they had lots of public information about them.
- (Hypothetical) Open source ASI model weights
- As of 2025-05 I would prefer halting ASI development worldwide. If there is a worldwide agreed halt, that is compatible with my suggestion for a world with less secrets. Leaked information about AI companies may help with coordinating this ban.
- As of 2025-05 I am not optimistic that a small time period of closed source development in one organisation is necessary or sufficient to solve the ASI alignment problem. This time period is likely less than 5 years, before the capabilities get leaked or replicated in another org anyway.
- (Yudkowsky was at some point in favour of closed source ASI development. If you defer to Yudkowsky on this, maybe consider that even Yudkowsky does not believe uninterpretable deep-learning-based ASIs will likely be aligned within 5 years of research at one organisation.)
- If ASI gets developed (and we don't go extinct due to AI misalignment soon after), I would strongly prefer a multipolar world where we had multiple orgs with independent militaries developing ASI, and we had information about the values and capabilities of each org developing ASI.
- ASI aligned to a small group of people could enable stable dictatorship by a small organisation, and leaking information about the values and decision-making processes of this organisation could help avert this outcome.
Enter email or phone number to subscribe. You will receive atmost one update per month