2025-01-10# Embedding searchI'm writing a post on why I'm so excited by embeddin
g search.It may have massive upsides and downsides. Also the ingredients for it
seem mostly ready. So it'll be difficult to un-invent even if the downsides are
greater. It seems better to adapt to a world where better embedding search is pr
esent.A lot of distinct problems in life seem like they could benefit from bette
r embedding search. - Finding a life partner - Finding funding or a job - Findin
g books, articles, etc relevant to your research - Establishing trust. Getting d
iverse information sources about a person helps build trust faster.Some dual use
implications: - Finding people who are trying to stay hidden because they are a
fraid of or competing with you or someone else. (This is dual use.) - Finding se
crets leaked to the internet (or any private database) by powerful organisations
. (This is dual use.) - In general, even things like finding funding or becomin
g better at research, etc accelerates everyone irrespective of what their goals
are. Attention markets on the internet right now are quite power-law distributed
. - If you want to find people of a specific niche, often you have to first get
attention of the mass public, and then filter out the subset of this that is int
erested in your niche. This is especially true if there doesn't exist an already
well-established community with organisers and funders. - Attention markets are
power law-distributed because there's atleast 1 million people competing for at
tention, but any random user has not more than 10 or 20 internet sources they re
gularly follow. (Sure, you might follow 1000 instagram pages, but you're not pay
ing attention to all of them.) If you can't get into someone's top 20 follows, y
ou're at a massive disadvantage when trying to get mass attention. - There aren'
t clear rules for how to give or receive attention. In practice this leads to an
arms race, one side invents ever new techniques to spam all the platforms, and
the other side raises the entry barrier to allow anything in.Embedding search mi
ght (??) also help fix the attention market. For example, I could list 5000 peop
le I'm interested in, and the other person could also list 5000 people they're i
nterested in, and if there's overlap then we connect. At no point do either of u
s need to send a DM without knowing a good probability estimate of whether this
is useful or spam for the other person.Elites use filters - The more you wield a
scarce resource such as attention or money, the more people are competing to ge
t in your network or influence your thinking. And the more you need to be delibe
rate about what filters you use to let people in or out. - A lot of problems in
society are causally downstream of principal-agent problems. Elites don't know w
ho to trust and get fooled into doing massively suboptimal things, basically all
the time. - Building better filtering mechanisms for elites seems useful.