2024-12-31# My AI timelinesDISCLAIMER - I'm not a deep learning expert. I unders
tand theory of LLMs, RNNs, CNNs, etc. but I don't have experience training large
models or curating large datasets or doing original DL research. - Please consi
der getting the opinion of many people other than me. Many of the signatories on
[this list](https://www.safe.ai/work/statement-on-ai-risk) have made individual
statements, search for their podcast interviews, blogposts, research publicatio
ns and so on. Lesswrong is also a good forum. - I strongly recommend you form yo
ur own independent opinion on this subject, and not blindly copy the views of ot
hers, no matter how smart or trustworthy they seem. This field would make a lot
more sense if more people formed independent opinions. (And yes forming a good o
pinion takes a lot of time and effort, and you have to decide whether the time i
nvestment is worth it for you personally.)This document is mainly aimed at lessw
rong (LW) rationalists / effective altruists / adjacent people, since a lot of m
y work is culturally downstream of theirs, and a lot of my potential research co
llaborators exist in their communities. This doc will make less sense if you hav
en't encountered their writings before.Most of this doc is guesswork rather than
models I have a lot of confidence in. Small amounts of evidence could upend my
entire view of these topics.**If you have evidence that my view is wrong, please
tell me**. Not having to spend any more time thinking about AI will improve my
quality of life. I am being completely serious when I say I might thank you till
the day I die, if you persuade me either way. I can also pay you atleast $1000
for convincing me, although we'll have to discuss the details if you really wann
a be paid.The last time I read papers on this topic was early-2023, it's possibl
e I'm not up-to-speed on any of the latest research. Feel free to send me anythi
ng that's relevant.I have ~15% probability humanity will invent artificial super
intelligence (ASI) by 2030. - If it happens, this will be the most significant e
vent in ~10,000 years of human history on many metrics of what counts as signifi
cant. (If the intelligence gap is sufficiently large it might even be the most i
mportant event in the ~14 billion year history of the universe. My probability f
or this is smaller.) - This will more-likely-than-not use transformers + data an
d compute scaling. - I define artificial superintelligence to be AI superior to
the best humans (not median human) at basically every task humans can do on lapt
op with internet access. - See my usual disclaimer on predictions shaping realit
y. If you are involved in building ASI, please consider *not* doing that or atle
ast talking to me once about it. I don't want ASI to be built by 2030 unless man
y aspects of society change first.Conditional on ASI being invented by 2030, I e
xpect ~30% probability it will kill everyone on Earth soon after. In total that'
s ~5% probability of humanity killed by ASI by 2030.I have chosen not to work on
this problem myself. This is downstream of ~15% not being large enough and the
problem not capturing my curiosity enough. I am not a utilitarian who blindly pi
cks the biggest problem to solve, although I do like picking bigger problems ove
r smaller ones. If you are working on safety of AGI/ASI, I think you are doing i
mportant work and I applaud you for the same.## Reasons for my beliefs*Mildly re
levant rant on consensus-building:* Both these probabilities seem to involve dea
ling with what I'd call deep priors. "Assume I take a random photo of Mars and a
random pixel from that photo, what is the probability its RGB value has higher
Green than Blue?" Humans tend to agree better on prior probabilities when there'
s enough useful data around the problem to form a model of it. Humans tend to ag
ree less on prior probabilities when they're given a problem with very little ob
viously useful data, and need to rely on a lifetime of almost-useless-but-not-co
mpletely-useless data instead. The "deepest" prior in this ontology is the [univ
ersal prior](https://en.wikipedia.org/wiki/Solomonoff%27s_theory_of_inductive_in
ference).~15% is lower than what many people in EA/LW communities assign, becaus
e I reject a lot of the specific models they use to forecast higher likelihood o
f ASI. - Total amount of neurons in human brain has nothing to do with compute r
equired to build ASI, as evolution produced human brains and evolution used a lo
t more compute than an individual brain. - Nobody has a proposal that's IMO like
ly to work at replicating biological evolution in-silico. How to capture initial
conditions? How much compute do you require to simulate the environment? I have
n't seen convincing arguments for why these details aren't important. - I have a
prior that most STEM research that promises bold outcomes fails. This is the de
fault not the exception. To study this prior you have to go back in time to each
of the failed research agendas of the past 100 years, and notice what it would
have **felt like** if you were born then and were hyped up by a research agenda
you didn't know would succeed or fail. - Update (credits: someone on LW): I have
seen some basic arguments from neuroscience saying neurons (or the genetic info
rmation that produced the neurons) don't by themselves encode much of whatever i
t is that produces human intelligence, most of it is learned after birth. I'm op
en to arguments of this type but I'll probably have to study neuroscience to eva
luate them.[Discontinuous progress in human history by Katja Grace](https://www.
lesswrong.com/posts/CeZXDmp8Z363XaM6b/discontinuous-progress-in-history-an-updat
e) is the closest thing I could find to work that tries evaluating the prior pro
bability of any reserach agenda succeeding with radical outcomes. I have not spe
nt a lot of time searching for such research though. Convincing me either way ma
y require publishing or pointing me to a lot more work of this type.Alternativel
y you have to provide me a gears-level model that explains why the LLM scaling l
aws empirically hold.~15% is higher than what many AI researchers assign, becaus
e I reject a lot of the specific reasons they give for why LLM scaling cannot po
ssibly achieve ASI - I have read some arguments for why specific unsolved proble
ms are *hard* when compared with already solved problems, and why there's no spe
cific reason LLM will crack them. - However most of the problems LLMs have alrea
dy cracked also got cracked without anyone having an actual model for why LLMs w
ill or won't crack them. LLM scaling has consistently proven itself (to me) to b
e black magic; both its supporters and its detractors fail to accurately predict
which problems it will or won't crack. Predicting loss curve doesn't tell which
problems will or won't be cracked. - Some people cite architectural limitations
of LLM as a bottleneck. For instance LLM has a theoretical upperbound on number
of calculations per forward pass, and a certain ratio of "memory access" to "co
mputation" steps. But solving a real world problem can use multiple forward pass
es + non-LLM scaffolding + non-self-attention layers. For example, you can often
use layers from two different architectures (let's say self-attention layers an
d CNN layers) in a single model, train this model and get good performance. - So
me people say we will run out of data required as per empirical scaling law but
my guess is this is more likely than not solveable. I'm very unsure on this. (An
d I lack sufficient expertise to evaluate this argument tbh.) I'm guessing you c
an teach a model simple reasoning using a smaller dataset, use this model to fil
l holes in a bigger dataset, and then teach the model more difficult reasoning u
sing this data.~30% probability for extinction conditional on ASI invention by 2
030 is because I am more optimistic about boxing an ASI than some LW rationalist
s. I do believe misalignment happens by default with high probability. - The int
elligence difference between Einstein or Churchill or Hitler, and a hyperpersuas
ive AI is large, relative to the intelligence jumps shown by scaling so far. I u
nderstand there's no natural scale for intelligence. I am open to the idea a hyp
erpersuasive AI can exist in theory. (In this context, a hyperpersuasive AI is a
n AI that can gain complete control over your mind with near 100% probability si
mply by talking to you.) - I am assuming a high level of competence among the pe
ople boxing the ASI. I have low probability of ASI coming completely by surprise
. A lab building ASI in this decade will almost certainly employ many people who
take the prospect of ASI seriously. - I am assuming that pausing ASI research w
ill be politically possible, even obvious, once we have built a boxed ASI with e
mpirical evidence of a lack of safety. I think I have better intuitions on polit
ics than the median LW poster. - I have some confusion on how the AI will reason
about the hardware-software abstraction. At what point does an AI translate "ma
ximise XYZ assembly variable" into "maximise the number of electrons in position
s in semiconductor underlying XYZ"? My guess is whether an AI wants to "break ou
t" of its machine depends on how it reasons about this. I accept that an ASI cou
ld understand that its variables are made up of positions of electrons in a semi
condutor, I'm just unsure what it'll do once it knows this.