WEBVTT

00:00.799 --> 00:05.399
Ah, hello everyone. (sighs) This is a test
recording,

00:05.539 --> 00:10.500
uh, for the talk I'm going to give in the Ooty
Retreat, uh,

00:10.559 --> 00:13.779
in June 2025. My name is Samuel Shadrack.

00:14.420 --> 00:16.979
It's 3:00 PM right now, so the

00:17.840 --> 00:22.639
talk will go till 5:00 PM, you know, assuming this
test recording works out.

00:24.500 --> 00:29.479
I'm going to introduce machine learning and
introduce large language models at

00:29.500 --> 00:34.199
a technical level. Uh, (door thuds) I'm going to
be covering a lot of material,

00:34.959 --> 00:36.800
so if you're completely new to this,

00:37.479 --> 00:40.920
it's possible you will not (person coughs)
understand all of it, but you'll understand

00:40.979 --> 00:41.759
most of it.

00:42.439 --> 00:47.080
The point is that you get an overview, not that
you understand every single detail, you

00:47.099 --> 00:51.700
know, just within two hours. That is going to
require, you know, more homework exercises,

00:51.739 --> 00:54.519
more practice, more, you know, writing your own
code,

00:55.439 --> 00:59.039
you know, deriving the formulas yourself. Like,
that will take a bit more time.

00:59.139 --> 01:00.599
It can't be done in two hours.

01:01.439 --> 01:05.239
But once you have this overview, then you will be
better placed to, you know, do more of

01:05.259 --> 01:08.439
that learning even on your own or following in an
online course.

01:10.899 --> 01:13.819
I'm assuming a background of

01:15.259 --> 01:17.419
basically, like, class 12,

01:18.739 --> 01:22.799
you know, science background, so I'm assuming you
know calculus, I'm assuming you know

01:22.799 --> 01:24.159
differential equations.

01:25.699 --> 01:29.739
Uh, not much of differential equation, but yeah,
you should understand basic calculus,

01:29.739 --> 01:31.659
you should understand matrix multiplication.

01:32.639 --> 01:37.459
Uh, we are going to use concepts of calculus and
matrices in today's

01:37.679 --> 01:38.159
talk.

01:41.639 --> 01:45.799
Okay, so let's begin. First, let's start with the
problem.

01:46.639 --> 01:50.899
So this is a very famous problem in machine
learning called MNIST.

01:51.179 --> 01:51.519
Uh,

01:52.219 --> 01:56.999
MNIST stands for Modified NIST. So NIST is the
standards body in the

01:57.079 --> 02:00.899
US, and also many countries have their own st-
standards bodies,

02:02.959 --> 02:06.659
uh, that define, you know, various units,

02:07.479 --> 02:12.419
like, you know, what is a meter, what's a
kilometer? You know, how big should a map be?

02:12.439 --> 02:15.479
What is one degree on a map? You know, these types
of things.

02:16.279 --> 02:17.899
So, they have defined a standard

02:18.539 --> 02:23.279
dataset, which is for optical character
recognition.

02:24.079 --> 02:28.839
So basically, you have these photos of people who
have written various letters...

02:28.839 --> 02:33.379
not letters, sorry, numbers. So people have
written, you know, three or two or seven or

02:33.399 --> 02:36.039
one. Like, any number they've written on a piece
of paper.

02:36.059 --> 02:37.839
We have taken a photo of that piece of paper.

02:38.779 --> 02:39.219
Uh,

02:40.219 --> 02:44.499
we have converted it more specifically into a 28
by 28 pixel image.

02:44.559 --> 02:48.359
It's grayscale, so it's not, you know, RGB. It's
not red, green and blue.

02:48.359 --> 02:48.759
It's just

02:50.099 --> 02:54.359
either black or white or gray. So the gray value
can be between zero and one.

02:54.819 --> 02:57.939
So, you know, 0.8 value means, you know, that
pixel is

02:58.979 --> 03:02.419
very close to black, and maybe a 0.2 means it's
very close to white.

03:03.359 --> 03:05.939
Now we have, you know, 60,000 such photographs.

03:07.299 --> 03:09.439
We already know which classes they're put into,

03:10.459 --> 03:15.259
and now we want to write a program that will take
an image as input and tell you, you

03:15.319 --> 03:15.539
know,

03:16.159 --> 03:18.839
what, uh, number is this. "Is this zero? Is this
one?

03:18.879 --> 03:21.679
Is this two?" And, like, we already know the
answers, but

03:22.499 --> 03:24.899
we don't want the program to memorize the answers.

03:26.079 --> 03:30.039
Uh, you know, o- one way to solve this problem
would be just, you know, we already know

03:30.039 --> 03:34.259
the answers, why not just have like a dictionary
in the (laughs) program that says, you

03:34.279 --> 03:38.719
know, "If you notice the first image, say this. If
you notice the second image, do this.

03:38.959 --> 03:41.299
If else, if else, if else," you know, 60,000
times.

03:41.999 --> 03:45.779
But we are not going to do that, and there's a
reason why we are not going to do that.

03:46.599 --> 03:51.199
The reason we are not going to do that is because
we also want this program to work

03:51.419 --> 03:53.759
on, uh, images which we have not seen.

03:54.159 --> 03:59.039
So apart from these 60,000 photos, there's another
10,000 photos which we

03:59.059 --> 04:01.179
have not seen. Like, we do not know what they are.

04:02.639 --> 04:05.779
I mean, they're actually there on the internet,
but right now we are not allowed to look

04:05.799 --> 04:06.879
at them. That's the rule.

04:07.899 --> 04:11.699
And we want our program to also perform well on
these 10,000 images,

04:12.399 --> 04:14.979
and those are also, again, you know, photos of
zero, one, two, three, four.

04:15.959 --> 04:16.359
So,

04:17.879 --> 04:22.479
we can't hard code it. We need some actual, you
know, way of solving this problem so that

04:22.519 --> 04:24.299
it will work on the photos we have never seen.

04:26.119 --> 04:27.619
I'm going to give everyone, like,

04:28.619 --> 04:30.919
maybe two minutes. Just try thinking

04:32.019 --> 04:35.139
how would you solve this problem and maybe you can

04:36.339 --> 04:37.919
share your approach. Like,

04:38.799 --> 04:42.639
I hope the problem is clear to everyone. Has
everyone understood what the problem is?

04:48.739 --> 04:49.759
Okay. So,

04:50.619 --> 04:51.579
here is a

04:53.819 --> 04:58.459
neural network based solution. So, in machine
learning, we have this concept called

04:58.519 --> 04:59.859
neural network, which is

05:00.719 --> 05:04.099
basically a special type of program. It's
ultimately a program.

05:04.099 --> 05:06.019
It takes input and it gives you output,

05:08.259 --> 05:11.699
but it is a bit different from regular programs
(door creaks) or the programs you write

05:11.719 --> 05:15.439
in, you know, C or Python or whatever. Like, we
will also write this in Python, but we...

05:15.479 --> 05:17.479
the way we write this program is a little bit
different.

05:18.439 --> 05:18.819
So,

05:19.679 --> 05:20.659
if you see these,

05:21.579 --> 05:23.919
this line here, this is basically our program.

05:24.859 --> 05:27.139
Can you see? Y equals ReLU of,

05:28.239 --> 05:28.619
uh,

05:29.479 --> 05:32.779
this into this. So this at symbol is matrix
multiplication.

05:33.039 --> 05:33.399
Uh,

05:34.339 --> 05:35.059
it's just, you know,

05:35.759 --> 05:39.519
like A at B just means, you know, matrix A
multiplied by matrix B.

05:40.299 --> 05:42.159
(door creaks) So we take this matrix,

05:42.819 --> 05:46.539
B mul- matrix multiplied with W2 and we apply this
ReLU function on it.

05:48.359 --> 05:50.039
So x here is the input.

05:50.939 --> 05:55.719
So the input is this, you know, 60,000
photographs. We have converted this into a matrix.

05:57.299 --> 05:57.799
Uh,

05:58.439 --> 06:01.779
we will multiply this with a weight matrix called
W1.

06:01.779 --> 06:06.359
I will talk about quickly what all of these are,
but...Yeah, x is our

06:06.379 --> 06:06.959
input.

06:07.640 --> 06:11.099
We will multiply the weight matrix. We will apply
this ReLu function.

06:11.319 --> 06:15.059
We will multiply that with w2. Then we apply this
ReLu function again.

06:15.839 --> 06:20.639
And y is our final answer. Like for each of these
60,000 images, you know, which,

06:20.820 --> 06:22.299
uh, class are they in?

06:23.819 --> 06:24.939
Okay. And

06:26.280 --> 06:29.759
now I will talk about, you know, what is ReLu,
what is w1, what's w2, what are all these

06:29.780 --> 06:30.380
things. So

06:31.319 --> 06:33.659
may, actually, no, maybe I'll first even just
start with the input.

06:33.659 --> 06:35.540
So x is the input, right?

06:37.179 --> 06:40.059
(cat meowing) So x has dimensions n cross d.

06:40.539 --> 06:42.539
If you've studied matrices, you know, right?

06:42.539 --> 06:42.879
Like a

06:43.739 --> 06:47.179
matrix has two dimensions, like, you know, its
length and its breadth.

06:48.019 --> 06:52.039
So n is the number of images. In this case, you
know, n is 60,000.

06:52.859 --> 06:57.679
So x is a 60,000 times 784 matrix. So there are
60,000

06:57.699 --> 06:58.699
images and-

06:58.699 --> 06:58.780
(sniffs)

06:58.780 --> 07:00.619
... each image has 784 pixels.

07:02.099 --> 07:03.699
Uh, yeah, one important thing here,

07:04.819 --> 07:07.999
we are not storing the images in a 28 cross 28.

07:07.999 --> 07:11.119
We're just storing it as, you know, single line of
784 numbers.

07:12.799 --> 07:15.939
Uh, this is a very common thing that's done in
machine learning.

07:16.899 --> 07:18.459
I'm not going to talk about why.

07:18.459 --> 07:18.499
(clicks tongue)

07:18.499 --> 07:20.599
But yeah, that's just a thing that's done.

07:20.679 --> 07:20.839
So

07:22.079 --> 07:25.379
we... Instead of having, you know,
dun-dun-dun-dun-dun, dun-dun-dun-dun-dun,

07:25.419 --> 07:28.419
dun-dun-dun-dun-dun, like we have, you know, 28
cross 28, we're just going to

07:29.319 --> 07:33.739
put everything in this one line. So 28 numbers,
then the next 28 numbers, the next 28

07:33.759 --> 07:38.059
numbers, next 28 numbers and so on. You know, you
have 70, 784 of them in line.

07:38.939 --> 07:43.699
And in the same way you have, you know, 60,000
such rows. So each row is one image.

07:43.699 --> 07:46.859
There are, you know, 60,000 of them. So this is
our input.

07:48.639 --> 07:51.579
Then we multiply with this weight matrix called
w1.

07:51.619 --> 07:56.199
So w1 is this 784 cross and 800 matrix.

07:57.459 --> 08:02.099
Uh, and what is w1? We will find out what w1 is.
Right now, just assume it's a matrix.

08:02.139 --> 08:04.239
We have somehow found out the value of w1.

08:05.159 --> 08:08.439
Then we multiply it. Then we do this thing called
ReLu.

08:09.339 --> 08:13.319
Now, what is a ReLu? Uh, so here is the definition
of ReLu.

08:13.899 --> 08:18.639
ReLu basically means if you have a matrix for each
cell, if it's a positive cell, just

08:18.699 --> 08:20.899
keep it as it is. If it's a negative cell,

08:21.499 --> 08:23.699
you remove that value and you put zero instead.

08:24.499 --> 08:28.659
So if you had, you know, let's say, three, zero,
minus two, four.

08:29.799 --> 08:32.919
(metal clanks) So now this will become three,
zero, zero, four.

08:33.039 --> 08:35.939
So that negative value just got removed and we put
zero there.

08:36.419 --> 08:40.799
And all the positive values, we just kept them as
it is. So ReLu is nothing special.

08:40.799 --> 08:44.919
It just means remove all the negative values in
that matrix and replace them with zero.

08:47.799 --> 08:49.059
So, yeah, we

08:49.979 --> 08:54.859
multiplied x with this w1 matrix. We made all the
negative values

08:54.939 --> 08:55.459
zero.

08:56.419 --> 08:59.139
Then we multiply this with the w2 matrix.

08:59.199 --> 09:02.059
Now w2's dimensions are 800 cross 4.

09:03.619 --> 09:08.339
Wait, fuck. That's a mistake. It should be 800
cross 10. N- just give me a minute.

09:08.339 --> 09:09.859
I'll just quickly edit that

09:11.539 --> 09:15.619
because there are 10 classes, you know, 0 to 10.
So this has to be 10.

09:15.619 --> 09:16.119
Uh,

09:24.679 --> 09:25.399
hmm.

09:27.899 --> 09:30.659
Yeah, cool. Okay, so we have 800 cross 10.

09:31.319 --> 09:34.579
And finally we get y, which is right now 60,000
cross 10.

09:34.599 --> 09:38.299
Like if you do, you know, the dimensional analysis
of this, you will get the answer, you

09:38.299 --> 09:40.939
know, 60,000 cross 10. Like 60,000

09:41.879 --> 09:46.639
times 784 cross 784 times 800 means 60,000 times
8, 60,000

09:46.659 --> 09:47.419
cross 800.

09:48.239 --> 09:51.859
Then you have 60,000 cross 800 times, you know,
800 cross 10.

09:52.619 --> 09:55.519
That leave you 60,000 cross 10.

09:56.279 --> 09:59.999
So now we have 60,000 images. And now why are
there 10 dimensions here?

10:01.059 --> 10:04.299
Like if... Think about it, there are 60,000 rows.
In each row there are 10 numbers.

10:04.299 --> 10:05.479
Why do we need 10 numbers?

10:06.119 --> 10:10.079
So what you'll get in these 10 numbers is actually
10 probabilities

10:12.019 --> 10:16.279
telling you which number is this. So let's say we
had, you know, number called three.

10:16.379 --> 10:18.739
We convert into, you know, 784 kind of thing.

10:19.959 --> 10:22.619
And like this there are many images. At the end we
will get...

10:23.119 --> 10:25.199
For each image we're getting ten probabilities.

10:26.619 --> 10:30.599
Now these are actually log probabilities. They're
not even probabilities.

10:31.299 --> 10:33.779
So here is an example of what we might get at the
end.

10:34.459 --> 10:36.119
And this is just for one image right now.

10:36.119 --> 10:39.499
Actually, there'll be 60,000 of these, but for one
image...

10:41.659 --> 10:41.859
and

10:43.039 --> 10:47.099
actually there will be ten of this. Uh, fuck, this
is annoying.

10:48.179 --> 10:50.059
I should edit that as well. Yeah.

10:53.779 --> 10:58.099
(clicks tongue) I'm just going to make things easy
for now and

10:59.679 --> 11:01.939
go with LN0, you know, six times.

11:05.899 --> 11:08.099
(keyboard clicking) Two, three, four, five, six.

11:09.599 --> 11:10.839
Minus inf.

11:19.599 --> 11:22.119
(keyboard clicking) One, two, three, four, five,
six.

11:23.679 --> 11:26.919
Zero, zero, zero, zero, zero, zero, zero, zero.

11:28.659 --> 11:31.399
Yeah, I think that's about right.

11:33.779 --> 11:37.139
Up- (door clanks) load this and here we go.

11:40.879 --> 11:41.479
Ooh.

11:44.039 --> 11:44.779
Yeah, cool.

11:45.419 --> 11:50.119
So our y is actually... Here is what y looks like.
So we have some values in here.

11:51.079 --> 11:55.579
These are all run... numbers we get, you know,
after doing the two matrix multiplications

11:55.579 --> 11:57.979
and the two ReLus. This is what we, you know, get
at the end.

12:00.839 --> 12:05.419
I can close all this. Yeah. So we got minus 1.89,
minus 0.28, minus

12:05.479 --> 12:06.819
1.89, minus three,

12:07.459 --> 12:11.223
minus infinity.And infinity just means, you know,
very large numbers.

12:11.243 --> 12:14.703
This could be in a minus, you know, 10 to the
power of 32 or

12:15.343 --> 12:19.083
whatever, you know, register we are using here for
float numbers, so this is, like, the

12:19.103 --> 12:22.384
largest number we can put in that register. So
it's as good as infinity.

12:24.463 --> 12:27.264
And y dash is what our actual answer is.

12:27.264 --> 12:30.323
So remember, we know what the actual answer is, so
here is y dash.

12:32.603 --> 12:36.383
So now we're going to take y, which is our
predicted answer,

12:37.603 --> 12:40.763
and by the way, this y is actually log
probability.

12:40.763 --> 12:42.763
So what this thing actually means is

12:43.743 --> 12:46.543
we are predicting, you know, 15% probability it's
a zero,

12:47.203 --> 12:51.983
75% probability it's a one, a 15% probability it's
a two,

12:52.243 --> 12:57.103
a 5% probability it's a three, and you know, 0%
probability it's anything else.

12:57.263 --> 12:59.843
Like, this is what this actually means, like, you
know,

13:00.543 --> 13:04.783
ln of 0.15 is, you know, -1.89.

13:06.763 --> 13:07.763
So yeah,

13:08.543 --> 13:12.103
this is the meaning of the thing but actually when
we run this program, what we're going

13:12.103 --> 13:13.563
to get is we're going to get this thing.

13:15.103 --> 13:18.703
Uh, assuming we're using good weight matrices, W1,
W2.

13:18.763 --> 13:23.223
We have not yet told you where did we get W1 and
W2 from, but if you're using a good

13:23.243 --> 13:27.203
value for W1 and a W2, we're, we might get
something like this.

13:28.843 --> 13:30.163
Or we might get something different.

13:30.163 --> 13:31.483
It's just this thing is saying, you know,

13:32.243 --> 13:37.043
this character is probably a one because, you
know, it's saying there's, there's a 75%

13:37.083 --> 13:38.003
probability it's a one.

13:39.523 --> 13:40.123
And now,

13:41.463 --> 13:43.803
yes, here's the predicted, here's the actual
thing.

13:44.723 --> 13:49.283
Now there is a question of, yeah, how do we find
good W1 and W2?

13:50.503 --> 13:52.223
So now here is an important question.

13:52.923 --> 13:53.623
And again, I'll,

13:54.343 --> 13:54.763
I know I've

13:55.383 --> 13:58.963
given you a lot of information. I want you to just
think for five minutes, you know.

14:00.763 --> 14:04.343
Here's the program. We want to find the W1 and the
W2.

14:04.683 --> 14:06.863
These are the dimensions of W1 and W2,

14:07.803 --> 14:08.183
and

14:09.003 --> 14:11.423
we want to ensure at the end,

14:12.923 --> 14:17.363
these predicted values should match close with the
actual

14:17.383 --> 14:21.983
values. So like if we manage to get exactly, you
know, the exact actual values, you know,

14:22.923 --> 14:24.443
here, ln0, ln1.

14:25.043 --> 14:27.743
What, what is ln1? Ln1 is zero, right? Yeah.

14:29.503 --> 14:34.063
So if we could get, you know, minus infinity,
zero, minus infinity, minus infinity, minus

14:34.103 --> 14:36.543
infinity, minus infinity, minus infinity and so
on, like,

14:37.183 --> 14:38.863
that is our ideal answer.

14:42.163 --> 14:45.163
And you know, otherwise anything else is a bad
answer.

14:53.403 --> 14:56.183
Also, we have this measure of how bad is our
answer, like

14:56.943 --> 15:00.003
our prediction is not a hundred percent accurate.
There is some inaccuracy here.

15:00.003 --> 15:03.363
So how inaccurate is it? So that thing is called
loss.

15:04.383 --> 15:04.723
So thi-

15:05.763 --> 15:09.263
this which tells you how far away is your
prediction from the truth.

15:09.303 --> 15:13.483
So this is defined typically, there are multiple
ways to define it, but common way of

15:13.483 --> 15:14.103
knowing it is

15:15.083 --> 15:17.663
negative sum of y dot y dash.

15:18.523 --> 15:21.143
So these are our predicted log probabilities.

15:21.583 --> 15:23.783
This is, you know, our actual ground truth.

15:25.163 --> 15:28.783
We are going to dot product this. So dot product
means what? Just this one we're taking.

15:28.783 --> 15:29.303
So this,

15:30.423 --> 15:31.443
we just took this.

15:34.363 --> 15:35.043
And

15:36.063 --> 15:37.483
yeah, that's our loss. So

15:38.163 --> 15:40.743
what probability did we give, you know, the
correct answer.

15:40.863 --> 15:44.283
In this case, we are just keeping it simple. We're
just keeping that as a loss.

15:44.703 --> 15:47.143
In practice here, there are other loss functions
also.

15:47.343 --> 15:47.643
But

15:48.783 --> 15:51.883
yeah, for now it's just the log probability of the
correct answer.

15:53.063 --> 15:54.623
That is our loss.

15:55.683 --> 15:59.203
So we want this to be less. And remember this, we
have now for one image.

15:59.563 --> 16:02.283
We will add this value up for now 60,000 images.

16:02.283 --> 16:04.003
We will get, you know, 60,000 such values.

16:04.003 --> 16:04.923
We will add them all up

16:05.583 --> 16:08.103
and that tells us, you know, how inaccurate our
thing is.

16:08.903 --> 16:12.723
So yeah, I, again, I hope this whole problem setup
is

16:13.603 --> 16:17.823
clear. Now we need to figure out a good W1 and a
good W2.

16:17.863 --> 16:20.703
Like how will we search for a good W1 and a good
W2?

16:21.823 --> 16:23.043
That's the question here.

16:28.063 --> 16:29.363
Okay. So

16:33.603 --> 16:36.723
how, uh, how many options do we have for W2?

16:36.743 --> 16:40.243
Like, we need to search for a good W2, but like
what's the search space?

16:40.263 --> 16:41.643
The search space is this, you know,

16:43.183 --> 16:47.803
784 cross 800 search space. 784 cross

16:47.823 --> 16:49.003
800. That's

16:50.063 --> 16:52.463
627,000 values, right?

16:54.243 --> 16:55.923
That is also wrong.

16:58.123 --> 17:02.643
627,000 values and W2 has

17:03.963 --> 17:08.563
8,000 values.

17:13.243 --> 17:14.063
2 to the

17:14.063 --> 17:21.763
power...

17:24.483 --> 17:24.883
Yeah.

17:27.003 --> 17:29.003
Huh. Lot of mistakes. Yeah.

17:32.723 --> 17:35.523
Yeah. So how many values do we have? We have...

17:36.783 --> 17:38.283
Here we have around...

17:41.223 --> 17:43.703
This is not changing. Cool.

17:44.403 --> 17:48.883
784 times 800. We have around six to seven...

17:49.643 --> 17:52.323
Around 600,000 values basically. And

17:56.283 --> 17:59.063
W2 has, you know, around 8,000

17:59.063 --> 18:11.939
values.That's

18:12.079 --> 18:12.799
60 s-

18:13.840 --> 18:14.239
sorry,

18:15.659 --> 18:19.819
784 into 800 into 800 into 10.

18:21.540 --> 18:24.620
(keyboard clicking) This is...

18:25.419 --> 18:28.179
That was in... Yeah, this is around 5 billion, I
guess.

18:29.620 --> 18:34.260
2 to the power 5... That means 75 into 10 to the
power of 9. (keyboard clicking) Cool.

18:34.279 --> 18:35.380
So we have 2 to the power of 627,000 into 8,000,
that's around 5 billion values.

18:35.380 --> 18:35.380
Uh, so, we have... Even if in each cell we can
only set, you know, 0 or 1, and actually

18:35.380 --> 18:35.380
we can set values in between also. We can set, you
know, 0.3, we can set 0.5.

18:35.380 --> 18:35.380
We can set, you know, 50,000. We can set, you
know, -87.5. You know, we can set anything.

18:35.380 --> 18:35.380
But even if we could set only 0 or 1, we have, you
know, 2 to the power of 5 billion

18:35.380 --> 18:35.380
possibilities. And in practice because we can set
many values, we have actually a lot

18:35.380 --> 18:35.380
more possibilities than that. So, we definitely
cannot do any sort of brute force.

18:35.380 --> 18:35.380
We cannot even do, like, a very smart sort of
brute force.

18:35.380 --> 18:35.380
Like, even if you reduce this, you know, 2 to the
power of 5 billion into, you know, 2 to

18:35.380 --> 18:35.380
the power of 1 million possibilities, like, maybe
you find some symmetry and you say,

18:35.380 --> 18:35.380
"Okay, these cells are similar to these cells, so
we don't have to actually search them

18:35.380 --> 18:35.380
all together. Let's just search these ones and
we'll copy the values in there." Like,

18:35.380 --> 18:38.620
often when we have brute force problems, this is
how people, you know, make it efficient,

18:38.620 --> 18:42.339
right? They try to reduce the search space.

18:42.439 --> 18:43.779
But even that's not really

18:43.800 --> 18:48.980
going

18:49.019 --> 18:52.719
to work here because our search space is just so
crazily large.

18:52.719 --> 18:55.320
So we can't really do it. We have to find
something better.

18:55.339 --> 18:59.699
And also,

18:59.699 --> 19:03.819
actually, if you look at this

19:04.199 --> 19:07.379
problem more, you will understand there's also
not, like, an exact answer.

19:07.379 --> 19:11.899
Like, you can't

19:11.899 --> 19:14.339
just look at this formula and calculate what they
look.

19:14.719 --> 19:15.759
Here is the formula for calculating,

19:16.219 --> 19:25.999
you

19:25.999 --> 19:27.459
know, the optimal

19:28.139 --> 19:30.380
W1 and W2. Like, to optimize loss, you know, here
is the optimal W1, here's the optimal

19:30.380 --> 19:30.380
W2. There is no direct formula to do this either.

19:30.380 --> 19:30.659
Like, just by looking at it you won't understand
this.

19:30.680 --> 19:34.419
If you spend maybe a month (laughs) just

19:34.419 --> 19:37.319
thinking about this problem, you'll realize, like,
yeah, there's no way you're getting an

19:37.319 --> 19:37.360
exact formula for this. So what does that leave us
with?

19:37.360 --> 19:37.360
Remember, we need to optimize loss given a W1 and
W2.

19:37.360 --> 19:37.360
So, we are going to bring in a concept from
calculus.

19:37.360 --> 19:37.360
Uh, in calculus, when you have to optimize one
variable with respect to other variables,

19:37.360 --> 19:37.360
what do we do, typically? Yeah. So we take, you
know, d loss by dw1 equals to 0.

19:37.360 --> 19:37.419
Like you remember, right? Maxima, minima. Yeah.

19:37.419 --> 19:37.960
So, that's kind of what we're going to do. Uh,
there are a couple of complications.

19:37.960 --> 19:37.960
The first complication is w1 is not a number, it's
a matrix.

19:37.960 --> 19:37.960
Because it's a matrix, you know, that complicates
things a little bit. The...

19:37.960 --> 19:37.960
Because, you know, if you just do dL by dw1 and
d...

19:37.960 --> 19:37.960
If w1 was a scale of value, then, you know, we can
just do dL by dw1.

19:37.960 --> 19:37.960
W1 is a matrix, so if you do dL by dw1, now that
is going to be another matrix.

19:37.960 --> 19:37.960
And you will say, "Okay, well, wh- why can't we
just put that matrix equal to 0?" Like,

19:37.960 --> 19:37.960
that mat-... dL by dw1 is a matrix, we'll
calculate it.

19:37.960 --> 19:37.960
We'll say, you know, that matrix equal to 0
matrix.

19:37.960 --> 19:37.960
And similarly,

19:39.499 --> 19:44.319
you know, dL by dw2 will be

19:45.759 --> 19:50.279
one more matrix. We will get that matrix, we'll
equate that to 0

19:50.279 --> 19:51.939
matrix. We will solve this entire system and we'll
get the answer.

19:51.939 --> 19:53.119
And, like, first of all you can try to do that.

19:53.119 --> 19:56.599
It's still going to be a very large system to
solve.

19:56.599 --> 19:58.119
Uh, if someone wants, they can try this out. I
have not tried it. You can try it out.

19:58.119 --> 19:58.159
But, y- like, you can imagine, right, you're going
to be getting...

19:58.159 --> 20:03.079
Like, again, you're going to get, you know, a
system of five billion equations.

20:04.359 --> 20:04.639
Like,

20:05.279 --> 20:07.399
five

20:07.399 --> 22:59.379
billion

23:01.199 --> 23:05.279
linear equations, and you're going to try to solve
them, I think, and...

23:05.279 --> 23:08.879
Well, okay, you can put it, you know, in a linear
algebra form, maybe you'll get it

23:08.879 --> 23:11.159
solved, but, yeah, that's again hard.

23:12.419 --> 23:14.959
And that's even more not going to work when you
make...

23:15.599 --> 23:18.159
like, when you add more matrices and you make this
thing bigger.

23:18.239 --> 23:19.819
Like, instead of, you know,

23:20.839 --> 23:25.499
w1 relu, w2 relu, what if I then did, you know, w3
relu, w4

23:25.519 --> 23:30.099
relu, w5 relu? Like, you know, multiply this with
another matrix, take a relu,

23:30.439 --> 23:34.159
multiply that with one more matrix, take a relu,
multiply that with one more matrix, take

23:34.179 --> 23:34.659
a relu.

23:35.519 --> 23:39.479
And in practice, actually, for more difficult
problems, we need to do that.

23:40.679 --> 23:44.459
Like, right now we have picked an easy problem, so
this gets solved pretty well just

23:44.479 --> 23:48.919
with, you know, two layers, like two matrix
multiplications and relus.

23:48.919 --> 23:49.219
But

23:49.919 --> 23:50.239
for

23:50.899 --> 23:52.939
harder problems we might take more. So,

23:53.799 --> 23:58.619
you know, for transformers, like, you know, your
latest AI use, like, GPT 3.5,

23:58.659 --> 24:02.099
GPT 4 or whatever, you might take, you know, like,
80 layers or even more.

24:02.819 --> 24:06.419
So there will be, you know, 80 matrix
multiplications running one after the other.

24:07.159 --> 24:09.999
And then you'll be like, "Well, how do we find the
optimal

24:10.919 --> 24:15.839
matrix?" Like, you're not going to be able to
solve, you know, d loss by dw1

24:15.879 --> 24:20.747
equal 0, like, analytically.What you can do...

24:23.327 --> 24:26.227
Is this thing called gradient descent, which is

24:27.848 --> 24:28.967
basically...

24:31.468 --> 24:32.447
Yeah. Basically,

24:33.107 --> 24:34.768
move along the direction

24:35.528 --> 24:39.608
decided by the slope. So, wherever you are right
now, so I mean, I can make...

24:39.608 --> 24:41.707
This might be easier to draw with, you know, a
figure.

24:41.827 --> 24:42.108
But

24:43.268 --> 24:44.887
whatever your loss value is right now,

24:45.928 --> 24:48.247
find out dL by dW in there

24:49.088 --> 24:51.947
and whichever direction it's pointing to, just
move in that direction

24:52.747 --> 24:56.268
because that is... If you move along that
direction, that's where the loss will reduce

24:56.347 --> 24:57.127
the fastest.

25:03.367 --> 25:06.548
So, this is the important trick in gradient
descent.

25:09.627 --> 25:14.167
So, we are going to find dL by dW1 and dL by dW2.

25:14.787 --> 25:19.467
We'll move in that direction and we'll find ano-
new W1 and W2 that are slightly

25:19.527 --> 25:20.847
better than the previous ones.

25:22.167 --> 25:25.587
And then we can repeat this process. You know,
once we have W1 and W2,

25:26.267 --> 25:30.227
we'll find the gradient, we'll move a little bit
to find a slightly better one.

25:30.687 --> 25:34.707
We'll find the gradient again. We'll find another
W1 and W2 that are even better than

25:34.727 --> 25:37.587
that. We'll repeat this, repeat this, repeat this.

25:37.587 --> 25:40.167
You know, after enough times repeating this
process,

25:41.447 --> 25:44.167
we'll get a really good W1 and W2.

25:45.107 --> 25:49.247
So for this, we need to find dL by dW1 and dL by
dW2.

25:50.587 --> 25:55.447
This is... Now actually finding these gradients is
kind of straightforward.

25:55.447 --> 25:57.107
It's also kind of boring. Uh,

25:58.287 --> 26:01.407
if you have, you know, half an hour to

26:02.667 --> 26:07.107
sit with a pen and paper and compute these
gradients, it can totally be done.

26:08.787 --> 26:11.127
So I mean, dL by dW2 is basically

26:11.727 --> 26:14.967
dL by dW2 IJ for each of these IJ.

26:16.227 --> 26:21.067
So it's like, you know, we had matrix times matrix
ReLU matrix ReLU.

26:22.007 --> 26:22.987
So now D

26:23.607 --> 26:27.187
and O... So this is Y. And then we did, you know,
dot product with the actual answers, we

26:27.207 --> 26:27.907
took a sum.

26:28.647 --> 26:30.627
So now this D loss by...

26:31.767 --> 26:33.247
We have to do with each of these values.

26:33.247 --> 26:36.127
If this value changes, you know, how much does the
loss change?

26:36.127 --> 26:38.347
If this value changes, how much does the loss
change?

26:38.947 --> 26:40.947
And so on for all the values in this matrix.

26:41.587 --> 26:44.827
That's dL by dW1. Same way we look at this matrix.

26:45.667 --> 26:48.107
We'll be like, if this value changed, how much
should the loss change?

26:48.107 --> 26:50.047
If this value changed, how much should the loss
change?

26:50.687 --> 26:51.367
And so on.

26:52.027 --> 26:56.267
We'll get one more matrix of those changes. That's
dL by dW2.

26:56.767 --> 26:59.547
And yeah, those are just your gradients.

26:59.547 --> 26:59.907
Like,

27:01.607 --> 27:03.467
if you change each, if you change this...

27:03.467 --> 27:08.127
Like for each of these values in W1, if you
changed it by some epsilon, it will change

27:08.167 --> 27:09.847
by, you know, this much times epsilon.

27:09.847 --> 27:14.367
Like, you change it by epsilon, you change it by
maybe 0.03 epsilon.

27:14.367 --> 27:16.967
So 0.03 is, you know, your gradient.

27:28.767 --> 27:33.007
Yeah. I mean, I can maybe find a nicer way of
writing this.

27:33.067 --> 27:34.127
Just give me a minute. I will

27:35.027 --> 27:37.107
write this in a kind of nice way.

27:38.467 --> 27:39.907
Like, uh...

27:52.227 --> 27:54.867
The loss is equal to...

28:02.427 --> 28:03.707
Into, you know,

28:05.727 --> 28:06.947
something.

28:19.667 --> 28:22.347
Then we took, you know, a ReLU of the entire
thing.

28:33.907 --> 28:38.727
Let me just use a... This bracket to
differentiate, to

28:38.767 --> 28:40.387
keep it, you know, looking separate.

28:41.727 --> 28:44.967
And then we're gonna multiply that with yet
another matrix.

28:52.607 --> 28:53.127
And

28:54.787 --> 28:56.907
then we took, you know, yet another ReLU.

29:00.987 --> 29:03.927
So I'm using another bracket for that.

29:11.247 --> 29:15.067
Then what did we do? We multiplied this with the
actual answer, correct?

29:18.747 --> 29:19.187
Yeah.

29:26.007 --> 29:28.767
So this we multiplied with

29:30.487 --> 29:33.207
w dot y dash.

29:35.567 --> 29:39.847
And then what did we do? We took a negative sum of
the entire thing, right? Yeah.

29:41.827 --> 29:45.247
So sum we can just, we can just say, you know,
magnitude of this thing.

29:46.847 --> 29:49.787
Sum, sum... Yeah, I'll, I'll just use the
magnitude

29:51.047 --> 29:52.127
notation.

30:03.687 --> 30:04.927
So here we have, you know,

30:06.867 --> 30:08.367
X00.

30:11.407 --> 30:14.747
This goes all the way to, you know, XND,

30:16.847 --> 30:24.023
and-Here

30:24.083 --> 30:25.443
we have w1,

30:27.763 --> 30:30.383
0, 0.

30:32.043 --> 30:34.124
(background noise)

30:36.303 --> 30:37.144
And this

30:38.983 --> 30:39.784
goes to

30:41.064 --> 30:43.463
w1 or w1 dimensions.

30:48.324 --> 30:49.263
W1,

30:51.223 --> 30:52.203
D, D1.

30:53.823 --> 30:57.023
Instead of calling this D1, let's just call it h.

30:57.243 --> 30:59.163
I think h is more convenient.

31:00.563 --> 31:00.944
Yeah.

31:03.563 --> 31:06.244
So we have w1 Dh.

31:15.183 --> 31:19.163
Then similarly we have, you know, w2, 0, 0.

31:21.903 --> 31:23.683
That goes still,

31:25.863 --> 31:26.503
yeah,

31:27.783 --> 31:30.723
w2 Hc.

31:34.224 --> 31:36.023
Yeah. I think that looks good.

31:46.044 --> 31:46.723
Uh,

31:48.283 --> 31:50.583
yeah, just give me a minute. I need to

31:52.824 --> 31:54.223
get that looking.

31:56.684 --> 32:03.563
(background noise)

32:08.523 --> 32:09.284
Yeah.

32:22.743 --> 33:02.123
(background noise)

33:02.123 --> 33:04.824
Where are we? This is not yet updated.

33:13.864 --> 33:16.643
(background noise) Yeah, this looks good.

33:22.023 --> 33:25.343
Yeah. So we just want to find new dL by dx.

33:26.303 --> 33:28.104
No, sorry, Dw1 0, 0.

33:28.923 --> 33:32.223
Yeah. And remember like these are partial
derivatives.

33:33.043 --> 33:33.523
Uh,

33:34.224 --> 33:37.823
I hope everyone remembers, you know, partial
derivatives versus total derivatives.

33:37.983 --> 33:38.324
Uh,

33:39.403 --> 33:43.783
partial derivative basically means when we are
doing dL by Dw1 0, 0, we are assuming

33:43.783 --> 33:47.923
everything else is constant. So now all these
values are constant, you know, everything

33:47.944 --> 33:48.563
in x,

33:49.383 --> 33:53.983
all the other values in w1 itself are constant,
all the values in, you know,

33:54.183 --> 33:57.543
w2 are assumed constant. You know, I true is
assumed constant.

33:57.663 --> 33:57.864
So,

33:58.483 --> 34:02.303
you know, x0, 0 is the... Sorry, w1 0, 0 is the
only thing that's changing.

34:02.783 --> 34:04.423
We change this by, you know, an Epsilon.

34:04.423 --> 34:06.543
(bell ringing) We're just checking by how much
will L change,

34:07.603 --> 34:08.983
by what multiple of Epsilon.

34:10.923 --> 34:11.183
So

34:12.463 --> 34:14.943
now this we can actually solve, you know, using
chain rule.

34:15.743 --> 34:18.464
So we can write it,

34:22.583 --> 34:23.563
you know, this way like...

34:24.323 --> 34:28.023
Wait, that's bad. I can't use H here and H here as
well.

34:28.203 --> 34:30.203
I will have to change this to something else.

34:31.703 --> 34:33.683
Uh, (metal clanking)

34:33.683 --> 34:47.323
(background noise)

34:51.983 --> 34:55.863
let's just make this small h. I think that looks
good.

34:57.744 --> 35:00.303
And yeah, this will become a small h.

35:05.024 --> 35:09.623
You know the notation that we have stepped is
below as well, so I'd rather not change

35:09.684 --> 35:10.363
this.

35:12.843 --> 35:13.703
Mm.

35:16.343 --> 35:19.303
Yeah, I think I'll just have to make this small h
then. Yeah.

35:21.043 --> 35:25.383
Yeah, but that's kind of an annoying way of doing
this.

35:30.183 --> 35:31.723
(background noise) Just give me some other letter.

35:31.783 --> 35:34.363
Uh,

35:38.783 --> 35:41.084
e? Okay.

35:42.003 --> 35:46.024
I, I don't know. This is probably not what's used
in, you know, the standard textbooks or

35:46.024 --> 35:49.244
whatever, but I'm just going to make this thing e,
you know, fuck it.

35:50.883 --> 35:51.423
Uh,

35:54.084 --> 35:56.823
yeah, let me just check, is h sitting anywhere?

35:58.723 --> 36:00.904
This case-sensitive.

36:05.683 --> 36:07.143
Yeah.

36:10.703 --> 36:14.523
(background noise) Cool. Yeah, cool. This is good.

36:18.783 --> 36:23.763
(background noise)

36:27.060 --> 36:30.260
Yeah. So we just made that number of dimensions E.

36:33.139 --> 36:34.659
So E is, you know, 800.

36:37.399 --> 36:39.679
This is why they called, you know, 800 hidden
layers.

36:39.679 --> 36:41.959
Like if you ever go see, you know, the...

36:42.979 --> 36:47.379
Some like standard lecture on this, this is a very
standard problem, you know, two layer

36:47.379 --> 36:49.559
fully connected network trained on MNIST.

36:50.619 --> 36:50.939
Uh,

36:52.119 --> 36:54.039
this thing will be called this number of hidden
layers2.

36:54.039 --> 36:55.639
So there are 800 hidden layers

36:56.260 --> 36:57.839
in our neural network.

36:59.579 --> 37:02.320
So you can solve this. If you solve this, you will
get,

37:03.399 --> 37:05.179
uh, this answer. So

37:06.420 --> 37:10.280
this is gradient W1, you know, dL by dW1 is this
answer, and

37:12.039 --> 37:14.579
uh, dL by dW2 is this answer.

37:17.079 --> 37:18.619
So basically,

37:21.280 --> 37:25.119
this x dot T is x transpose. I'm just clarifying,
you know, the notation.

37:25.739 --> 37:30.279
And maybe I'll just start here. h dot T is, you
know, h transpose, uh, B hat.

37:32.799 --> 37:37.299
Uh, here we are taking minus Y true, so minus Y
true times mask.

37:37.299 --> 37:38.599
Now mask is

37:39.599 --> 37:41.679
all the positive values of a2. ...

37:42.339 --> 37:45.379
a2. Okay. a2 is h times W2.

37:47.259 --> 37:49.579
Can you see? It is a little bit complicated, but
yeah.

37:51.899 --> 37:53.939
Like let's just go back here.

37:55.059 --> 37:57.499
So h times W2. So basically they took this thing,

37:58.139 --> 38:00.479
like this, this much is h times W2 down here.

38:02.879 --> 38:05.519
They checked which of those values are greater
than zero,

38:07.799 --> 38:10.539
and s- some of those are one and some of those are
zero.

38:11.739 --> 38:14.139
So we made that, you know, a mask, mask2.

38:15.239 --> 38:17.759
And by the... Why did we end up with something
weird like this,

38:18.379 --> 38:21.739
where put... the positive values are made one, the
negative values are made zero?

38:21.739 --> 38:22.439
It's basically the

38:23.279 --> 38:26.559
positive values, their gradients are one, and the
negative values, their gradients are

38:26.579 --> 38:29.139
zero. Like, this is the gradient of ReLU.

38:31.339 --> 38:35.659
Like, if you have a function that goes, you know,
ting, ting, so now the gradient of

38:35.679 --> 38:38.259
these values is zero, and, you know, the gradient
of these values is one.

38:38.819 --> 38:41.179
So that's why you ended up with this thing called
a mask.

38:43.139 --> 38:45.479
If you would actually try deriving this, you'll
understand, I think.

38:45.479 --> 38:48.379
I don't have time to, you know, derive the entire
thing right now.

38:50.319 --> 38:51.019
The...

38:51.639 --> 38:51.939
So

38:52.659 --> 38:55.999
you'll, you'll get a mask, you'll do minus Y true
into this mask.

38:56.759 --> 39:00.939
Then you'll do h transpose matrix multiplied with
this entire thing, that gives you your

39:00.979 --> 39:01.699
gradient.

39:02.639 --> 39:04.859
Then now that you have this gradient...

39:08.119 --> 39:12.399
okay. I think that you keep separate, and
separately you also need to do this calculation

39:13.259 --> 39:15.299
for, you know, gradient of W1.

39:16.599 --> 39:20.059
So this derivation, I'm going to leave it as
homework.

39:21.119 --> 39:25.139
Uh, those of who, you who are good at linear
algebra, you can probably get this

39:25.159 --> 39:27.379
derivation done. It's basically, you know,

39:28.019 --> 39:29.919
here is the original thing we started with.

39:30.439 --> 39:34.759
Assume you know, except W100, every, just keep
everything constant.

39:34.839 --> 39:37.719
Then except W101, keep everything constant.

39:37.799 --> 39:42.699
Except W102, keep everything constant, and just
find dL by, you know, that value.

39:45.999 --> 39:49.859
So you'll get these derivations. You know, we have
found the matrices.

39:50.779 --> 39:53.119
So, but this works at one particular point.

39:53.119 --> 39:53.359
So,

39:53.999 --> 39:57.959
you know, access constraint we have assumed in our
input, provide you through the actual

39:57.999 --> 39:59.399
answers is constraint.

40:00.139 --> 40:02.219
W1 and W2 are the things we need to find.

40:02.219 --> 40:06.859
So if we are given a current W1 and W2, we can
find another gradients at that

40:06.919 --> 40:07.759
particular position.

40:09.179 --> 40:09.679
Then

40:10.359 --> 40:12.039
we will update W1 and W2,

40:12.999 --> 40:16.959
like we multiply these ingredients with a small
epsilon, just a tiny amount.

40:16.979 --> 40:19.719
The actual number is something we can decide
later,

40:20.379 --> 40:23.799
how big we want epsilon to be. That is a whole
separate topic.

40:25.099 --> 40:28.839
Uh, there is this thing called an atom optimizer
that is used nowadays.

40:29.019 --> 40:30.619
I will get to that later. Not right now.

40:30.799 --> 40:33.699
But right now, just assume E is some very small
constraint value.

40:33.699 --> 40:36.019
It could be, you know, 0.0001 or whatever.

40:38.219 --> 40:41.719
So we multiply that with our gradient and we
update W1 and W2.

40:41.999 --> 40:46.039
So we get as, uh, a new W1 and W2, which has
slightly less loss.

40:47.899 --> 40:51.079
The L now for this new W1 and W2 is a little bit
less,

40:52.079 --> 40:54.079
or well, sorry, I think L is negative, so

40:55.119 --> 40:56.899
L will go up actually, yeah.

40:58.439 --> 41:01.599
And then we repeat this process millions of times.

41:02.979 --> 41:03.959
Like, we will have to do

41:04.919 --> 41:09.479
these calculations, we'll have to do them millions
of times and keep updating W1 and W2,

41:10.479 --> 41:11.999
and at the end we'll get a good answer.

41:16.699 --> 41:17.239
Uh,

41:18.399 --> 41:18.699
yeah.

41:19.399 --> 41:20.479
This is what

41:21.259 --> 41:23.439
a two layer fully connected network

41:24.059 --> 41:24.639
is.

41:27.199 --> 41:32.059
And how did we manage to run this? Uh, we will
probably need a GPU to run

41:32.099 --> 41:32.439
this.

41:33.279 --> 41:34.759
So the GPU is basically

41:36.079 --> 41:36.259
a

41:37.359 --> 41:40.979
chip that does matrix multiplications very fast.

41:41.279 --> 41:42.479
Like, that's the whole thing.

41:43.179 --> 41:47.179
The GPUs were actually way back, you know, before
deep learning was very popular, GPUs

41:47.179 --> 41:48.519
were actually designed for gaming.

41:49.379 --> 41:54.339
And for gaming you need to render like, you know,
vectors into

41:54.379 --> 41:55.099
graphics.

41:56.079 --> 41:57.539
So your vector might be, you know,

41:58.399 --> 42:01.619
draw a line from 0, 1 to 4, 3.

42:02.679 --> 42:05.619
And we need an actual, you know, pixel matrix.

42:05.639 --> 42:08.859
Like we need all, each of those pixels, you know,
set to 1, 1, 1, 1, 1, 1.

42:10.259 --> 42:12.659
Like, even right now, if you look at this entire
thing,

42:13.699 --> 42:16.259
like what's showing on your monitor, actually,

42:16.999 --> 42:20.519
initially the instructions are like, okay, you
know, draw a rectangle here for this tab.

42:20.519 --> 42:22.999
You know, put some text inside here. Now what the
hell is text?

42:23.019 --> 42:25.239
Now go and get this, you know, font sheet from
here.

42:25.899 --> 42:29.543
There is some, you know, bunch of images for each
of these letters....

42:29.644 --> 42:30.224
you know, draw

42:31.443 --> 42:32.343
lines here,

42:33.263 --> 42:35.703
draw, you know, a rounded rectangle here and, you
know,

42:36.743 --> 42:40.403
this gets a lot more complicated when you are
playing a game, for example.

42:40.443 --> 42:42.783
So when you have a game, you have, you know, a 3D
game engine,

42:43.704 --> 42:47.523
so you need to convert all those lines into, you
know, this ultimate 2D view.

42:48.864 --> 42:53.083
So doing all those conversions also in practice
you end up doing using a bunch of matrix

42:53.124 --> 42:54.583
multiplications, so

42:55.243 --> 42:57.443
that's what GPUs were originally designed for.

42:58.003 --> 43:02.424
Today, you know, GPUs are just being used for AI
and people are even

43:02.463 --> 43:05.563
redesigning GPUs to become better at AI. So, yeah.

43:07.663 --> 43:11.383
So that's why, you know, GPUs were originally good
for matrix multiplication.

43:13.143 --> 43:16.323
And if you look at these, you know, the only
things we have mainly are only

43:16.783 --> 43:20.103
computationally expensive operations here are
matrix multiplication.

43:21.443 --> 43:24.924
We also have this, you know, zero check, is this
positive or negative like across the

43:24.943 --> 43:29.003
matrix? This can also be parallelized and GPUs can
parallelize this as well.

43:29.023 --> 43:33.603
Like, again, remember these are like millions of
values there.

43:35.183 --> 43:36.423
A2 is,

43:37.023 --> 43:38.203
uh, how big is A2?

43:41.983 --> 43:44.183
60,000 cross 10, right? Yeah.

43:45.103 --> 43:48.903
Yeah, so this is 600,000 values. We would rather,
you know, check all of this in

43:48.943 --> 43:50.743
parallel, not check them, you know, one by one.

43:51.023 --> 43:51.323
So,

43:52.463 --> 43:54.203
yeah, all this can be parallelized.

43:54.843 --> 43:58.083
We are not doing any copy-paste anywhere and that
is important.

43:58.223 --> 44:01.643
If we had to, you know, copy-paste, you know, one
million values from here to there, that

44:01.663 --> 44:03.223
would be a problem, it will become slow.

44:04.383 --> 44:07.943
And because we're doing this on every iteration,
so like, we have to copy one million

44:07.983 --> 44:09.803
values, you know, one million times.

44:09.843 --> 44:13.163
So that's like a trillion copy-paste operations
and that's bad.

44:13.263 --> 44:16.003
So, if we had copy-paste that would be a problem.

44:17.023 --> 44:18.183
We don't have that either.

44:18.803 --> 44:21.803
We just need to in place, we need to update, you
know, w1 and w2.

44:24.583 --> 44:27.243
So, yeah, this will run quickly on a GPU.

44:29.203 --> 44:29.743
And,

44:30.783 --> 44:33.683
yeah, okay, there's a note, I have simplified some
steps here.

44:34.603 --> 44:38.563
So one thing which I've simplified is, in practice
we are not going to be...

44:39.023 --> 44:42.843
typically, we are not going to be processing our
entire 60,000 images all at once.

44:43.443 --> 44:45.683
We will typically split it into batches.

44:46.223 --> 44:50.403
We will do, like the gradient descent we'll be
doing, we'll be taking different batches

44:50.423 --> 44:51.683
in different iterations.

44:52.303 --> 44:57.243
And this you can actually maybe go read online how
batch SGD works but, basically we su-

44:57.283 --> 45:00.543
spread the 60,000 into batches of let's say 100
images each.

45:00.983 --> 45:04.783
We compute the gradient not over the entire 60,000
images, but over 100 images.

45:05.463 --> 45:06.483
So our x,

45:07.523 --> 45:11.563
when we do this anti-gradient calculation, we will
take x as, you know, just

45:12.003 --> 45:16.923
100 cross, uh, 784 instead of, you know, 60,000
cross 784.

45:17.563 --> 45:20.183
We'll compute the gradients, we'll update w1 and
w2.

45:20.863 --> 45:23.963
Now we will take a different batch, and th- here's
where things get weird,

45:24.783 --> 45:27.643
uh, and then we will compute the gradient on that.

45:28.623 --> 45:31.723
But now you will ask, "Uh, isn't that a whole
separate problem?" You know?

45:32.643 --> 45:36.543
Like the optimal w1 and w2 for this batch and
optimal w1 and w2 for that batch are

45:36.543 --> 45:37.123
different like,

45:37.743 --> 45:40.903
this is an entire separate gradient descent
problem, that's now a separate gradient

45:40.923 --> 45:44.363
descent problem but, we'll take one gradient from
here, we'll take one gradient from

45:44.403 --> 45:47.743
there, we'll make a third batch, we'll take
another gradient from there.

45:48.583 --> 45:51.483
Then we will use all of these gradients, we'll add
up all of them, you know.

45:52.063 --> 45:55.543
Epsilon plus this, epsilon plus this, epsilon plus
this, and we'll keep doing all of

45:55.603 --> 45:55.903
this.

45:57.263 --> 46:01.403
So this is called, you know, batch SGD. SD is
stochastic gradient descent.

46:02.043 --> 46:02.963
And here there's a

46:04.283 --> 46:08.343
bit of randomness that will come in because we are
not taking all the images at once,

46:08.343 --> 46:09.763
we're like just randomly selecting

46:10.403 --> 46:13.503
different batches and taking gradients from all of
them and adding it up.

46:14.183 --> 46:17.963
There's a bit of randomness that comes in, but
this will also give a good enough

46:17.963 --> 46:18.543
solution.

46:19.423 --> 46:24.163
Yeah, and by the way, the only reason we didn't
take all of them is, again, 60,000 images

46:24.163 --> 46:27.543
might not fit in your GPU advance, and that's the
only reason, like, you...

46:28.023 --> 46:32.923
If you can fit in 60,000 G- like images into the
GPU you might be honestly better off

46:32.923 --> 46:37.183
just doing the normal gradient descent, like, the
regular way of doing it what I've

46:37.183 --> 46:37.963
described, but

46:38.863 --> 46:39.763
if you had even more

46:41.203 --> 46:45.263
data like instead of 60,000 images you had, you
know, like a billion images,

46:46.183 --> 46:50.743
and in modern datasets you have literally, you
know, billions of images, billions of, you

46:50.763 --> 46:52.863
know, text documents, so,

46:53.783 --> 46:56.723
you can't fit all of that into GPU, so, yeah.

46:57.923 --> 47:00.203
Then we use an optimizer called ADAM.

47:01.483 --> 47:06.143
ADAM is basically just a way of changing these
epsilon values, so, we

47:06.203 --> 47:10.883
start with big epsilon values like we first
descend f- fast and then we reduce the

47:10.883 --> 47:14.363
epsilon values, so Adam is just, how do you do
this?

47:14.363 --> 47:14.583
And

47:15.723 --> 47:19.603
the main reason to do this is just to again save
cost like

47:20.483 --> 47:23.363
how do we do this anti-gradient descent more
quickly?

47:24.323 --> 47:27.203
And last what? Cross-entropy loss,

47:28.083 --> 47:31.243
yeah I think I've already defined this using
cross-entropy loss only.

47:32.803 --> 47:37.403
But yeah cross-entropy loss is basically this
thing about the log probabilities like we

47:37.403 --> 47:40.563
didn't use actual probabilities we used log
probabilities like that,

47:41.263 --> 47:44.903
the loss that comes out at the end of this is
called cross-entropy loss.

47:47.263 --> 47:47.643
Yeah.

47:48.383 --> 47:49.123
So now,

47:49.783 --> 47:51.143
here's some homework.

47:52.123 --> 47:55.043
So the first homework is I have already told you,
you know you need to

47:56.943 --> 48:00.083
derive this backprop formula yourself.

48:01.783 --> 48:06.283
This thing just actually derive this or at least
go online search for a

48:06.303 --> 48:10.803
derivation you know look at it like you can ask O3
to do this stuff for you like

48:11.163 --> 48:15.503
actually O3 computed this you know you can ask O3
give me more steps you can be like you

48:15.503 --> 48:18.283
know O3 I do not understand this you know explain
this or whatever.

48:19.423 --> 48:22.283
Like I can literally show you what I'm talking
about. Uh...

48:26.623 --> 48:28.223
Explain this

48:29.883 --> 48:30.583
in about

48:31.283 --> 48:31.743
10, 10

48:33.003 --> 48:37.455
to 15 minutes.Like I ment-

48:37.775 --> 48:39.695
mentioned a point earlier, you know, like...

48:41.076 --> 48:41.235
Wait,

48:41.855 --> 48:43.636
let's, uh, copy this.

48:46.036 --> 48:46.855
(keyboard clicking) And put

48:48.416 --> 48:49.735
this in here.

48:55.076 --> 48:57.395
(keyboard clicking) Why are we doing

48:59.595 --> 49:01.755
A2 greater than zero?

49:04.395 --> 49:08.176
(keyboard clicking) Like can you see how we did A2
greater than zero?

49:08.195 --> 49:10.235
I'm just asking O3, you know, why are we doing
that?

49:16.795 --> 49:17.715
(breathing)

49:19.135 --> 49:22.536
(chair creaking) So, you can ask for re- these
types of, you know, specific questions.

49:24.815 --> 49:29.516
Yes. What does it say? Okay, it just... yeah, this
is just defining the problem.

49:32.175 --> 49:34.075
Okay, there are... chain rule.

49:36.015 --> 49:38.775
Yeah, that's an lengthy-ass derivation.

49:39.095 --> 49:39.496
Uh,

49:40.376 --> 49:41.715
yeah, here is our explanation.

49:42.595 --> 49:44.476
Why is A2 greater than zero?

49:45.296 --> 49:50.135
Yes, a ReLu is this function, you know, zero if
negative, you know, Z is

49:50.155 --> 49:51.155
positive. And

49:51.916 --> 49:53.336
its derivative is zero and one.

49:54.355 --> 49:56.155
That's why we did this.

49:57.955 --> 50:01.515
Yeah, I mean, you can sit and read all this if
you, for instance want to understand, you

50:01.515 --> 50:02.995
know, how to do the gradient of ReLu.

50:04.795 --> 50:06.816
So you can... yeah, that's your first homework.

50:07.436 --> 50:10.956
The second homework which is more advanced, but
again, you will find this online.

50:11.315 --> 50:13.976
You don't have f-... You can do this from scratch
if

50:14.995 --> 50:19.015
you really want to understand this or, you know,
you're a bit of a masochist, uh, but

50:19.635 --> 50:24.475
you will find this assignment existing online too,
which is, you know, actually train

50:24.475 --> 50:27.095
this fully connected network with, you know, 800
hidden,

50:28.135 --> 50:29.595
uh, dimensions.

50:31.195 --> 50:32.856
Use Pytorch. Uh,

50:33.715 --> 50:37.955
you will get a 1.6% error rate, which means in out
of the 60,000,

50:38.895 --> 50:43.276
uh, examples, not... (keyboard clicking) Sorry, on
the remaining 10,000, right?

50:43.276 --> 50:44.015
(keyboard clicking) So on the remaining

50:44.736 --> 50:46.435
10,000 you'll get,

50:48.635 --> 50:52.276
(keyboard clicking) uh, what am I doing? Yeah,
98.4% correct.

50:52.276 --> 50:52.495
So

50:53.475 --> 50:58.136
9,840 examples you'll get correct. And about 160
you'll get wrong

50:58.375 --> 51:01.315
on the remaining 10,000 examples which you've not
seen till now.

51:02.915 --> 51:05.536
(chair creaking) So that's pretty good.

51:05.595 --> 51:08.595
(child talking in background) So yeah, that's what

51:09.355 --> 51:13.835
you will do. (paper rustling) Yeah, and also I
want, if you're doing this

51:13.835 --> 51:16.655
assignment, like you're actually sitting and
writing this in Pytorch,

51:17.355 --> 51:20.515
uh, I would recommend you actually write down
these gradients.

51:20.515 --> 51:22.835
Otherwise Pytorch has this feature called
autograd,

51:23.636 --> 51:27.496
which means Pytorch can compute these gradient
formulas on its own and use them.

51:28.395 --> 51:28.755
Like,

51:29.455 --> 51:34.355
this is part of the reason why NVIDIA is on track
to become a trillion

51:34.395 --> 51:35.195
dollar company,

51:36.155 --> 51:37.496
uh, is, you know,

51:39.416 --> 51:42.075
Pytorch does these types of calculations on their
own.

51:42.075 --> 51:45.275
So ML engineers don't need to sit and, you know,
break their head on how to do

51:46.575 --> 51:47.455
gradients,

51:48.496 --> 51:50.335
like how to calculate this gradient formulas.

51:50.336 --> 51:54.115
This is not the only reason NVIDIA is so rich,
but, uh, you know, it's part of the

51:54.115 --> 51:54.475
reason.

51:55.095 --> 51:55.996
(paper rustling) Uh, so yeah,

51:56.615 --> 51:58.755
I would recommend for the homework assignment, you
know,

51:58.795 --> 52:03.756
(child playing in background) uh, yeah, use these
actual formulas which are written

52:03.775 --> 52:04.936
down here. Don't, you know,

52:05.555 --> 52:08.315
ask Pytorch to i- autograd to, you know, just do
it.

52:11.655 --> 52:12.035
Cool.

52:13.575 --> 52:15.135
Any questions? I think

52:16.315 --> 52:19.695
this- (bell ringing) what we've done here, this is
actually the most important thing in

52:19.735 --> 52:20.535
the lesson today.

52:21.235 --> 52:24.295
The remaining one hour is a bit less important
(laughs) than this one hour.

52:25.355 --> 52:29.595
If you have just understood this very well, you
have understood, uh, like a lot about how

52:29.615 --> 52:30.695
deep learning works.

52:34.935 --> 52:35.455
Cool.

52:36.975 --> 52:38.975
Now (coughs) comes the next section.

52:39.595 --> 52:39.895
Yeah.

52:40.655 --> 52:41.895
So here is a question.

52:43.115 --> 52:47.235
Why did we do all this, you know? Why did we use
two layers? Why did we use a ReLu?

52:47.775 --> 52:49.955
Why did we define our loss function this way?

52:50.155 --> 52:53.095
Like we could have defined the loss function
different ways, you know, actually if we

52:53.115 --> 52:53.735
wanted to.

52:55.055 --> 52:55.335
Like

52:56.095 --> 52:58.975
here I took, you know, this value and ignored all
the other values.

52:59.115 --> 53:01.675
And also I said this should be a log probability.

53:01.695 --> 53:04.295
Like why did we decide all this, you know?

53:05.795 --> 53:10.635
And in practice, when you go into proper deep
learning, you will get even more questions.

53:11.935 --> 53:13.835
Like you will learn things, you know...

53:14.575 --> 53:16.275
There will be a lot of these types of tricks.

53:16.275 --> 53:19.415
Like okay, use RMS norm here, use Softmax here,

53:20.355 --> 53:25.215
define attention layers in this way, use these
many layers, like we use two layers, you

53:25.215 --> 53:26.535
know. Why two layers? Like...

53:27.135 --> 53:29.315
And this in practice becomes a lot more layers.

53:29.975 --> 53:34.675
You know, what's a residual layer? Like, we are
going to encounter a lot of tricks

53:35.295 --> 53:38.955
and whenever a question comes in your mind, you
know, why are we doing this trick?

53:39.415 --> 53:41.595
Always by default, here is the answer.

53:43.035 --> 53:45.315
Um, I'm going to just open the answer right now
for you.

53:53.915 --> 53:54.015
(person coughing)

53:55.655 --> 53:55.715
(water running)

53:55.715 --> 53:56.595
Should we bow?

53:57.235 --> 53:59.795
Yeah, he's a king. Seems like I'm always thanking
you for something.

54:00.335 --> 54:02.015
(clears throat) What are you doing?

54:02.595 --> 54:04.655
Uh, we, we don't do that here.

54:06.015 --> 54:06.635
Yeah.

54:07.295 --> 54:08.875
So that's the answer.

54:09.755 --> 54:11.795
We do not ask these types of questions here.

54:12.675 --> 54:15.575
Do not ever ask me, "Why did we do this?" and "Why
did we not do that?"

54:17.335 --> 54:18.295
The answer is

54:19.075 --> 54:21.515
some guy tried it a few years back, it worked out.

54:21.855 --> 54:25.155
Some guy tried a little bunch of similar things,
those did not work and this worked

54:25.275 --> 54:28.975
because who knows? And that's the answer. That's
always the answer.

54:29.115 --> 54:31.215
You're not going to get no- a better answer than
that.

54:32.135 --> 54:35.435
If you think you have a better answer than that,
often you're actually mistaken.

54:37.355 --> 54:41.075
And that comes here. Like lots of people will give
intuitions for why, you know, this

54:41.095 --> 54:45.735
trick works and that does not work.But often,
these intuitions are created

54:45.736 --> 54:47.695
after we have actually tried the experiment.

54:47.696 --> 54:47.955
Like,

54:49.216 --> 54:50.395
we tried an experiment,

54:51.015 --> 54:52.876
some particular trick worked, you know.

54:53.536 --> 54:56.296
Let's say this tool ReLU worked very well.

54:57.335 --> 55:00.376
Instead of ReLU, we could have refined a
different, you know, activation function.

55:01.175 --> 55:03.555
But ReLU worked very well and now, let's...

55:03.555 --> 55:08.135
We will now invent some, you know, intuition for
why ReLU is a good idea and why, I don't

55:08.135 --> 55:09.735
know, why ReLU is a bad idea or whatever.

55:11.995 --> 55:15.975
Uh, yeah. ReLU is the real thing. Uh, I can show
you.

55:16.995 --> 55:18.815
Like, there are other loss functions.

55:25.055 --> 55:28.235
Yeah. So this is what ReLU looks like. Here's the
formula for ReLU.

55:29.075 --> 55:33.595
ReLU is just, you know, negative X, make it zero,
positive X, make it, you know, X.

55:35.175 --> 55:37.335
And here is a bit more ... Yeah.

55:40.636 --> 55:42.135
So, yeah, people just

55:44.516 --> 55:46.575
create their intuitions afterwards.

55:47.535 --> 55:51.855
Also, yeah, especially modern day ML, because
there's only a few labs that are doing, you

55:51.855 --> 55:53.215
know, the latest research,

55:54.055 --> 55:57.135
uh, only people in those labs are going to have
these intuitions anyway.

55:57.395 --> 55:57.815
Like,

55:58.536 --> 56:00.355
they are the ones who are running hundreds of
runs.

56:00.376 --> 56:04.355
Like, to get an intuition, you need to try some
things and see that some of them work and

56:04.355 --> 56:07.195
others did not work. Like, it's a very empirical,

56:08.315 --> 56:11.575
uh, field. It's not a theoretical or research kind
of field.

56:11.915 --> 56:15.135
It's just pure, tried this, it worked, tried that,
it did not work.

56:16.335 --> 56:19.996
And right now, the people trying things are only,
like only if you have, you know, a

56:20.015 --> 56:24.055
company that has raised, you know, millions of
dollars in funding, you can try things

56:24.075 --> 56:24.915
anymore. So,

56:25.695 --> 56:27.595
they are the only ones who have these intuitions
anymore.

56:28.555 --> 56:31.655
Yeah. So if you ask me, you know, why all these
older things?

56:31.655 --> 56:31.855
So,

56:32.515 --> 56:36.755
here are, you know, older architectures, here are,
you know, older activation functions,

56:36.776 --> 56:41.775
older loss functions, older optimizers, you know,
older regularization

56:41.835 --> 56:46.075
tactics. You know, a lot of these things, you will
read about these in, you know, five

56:46.115 --> 56:49.515
year old tutorials. You will not read them, about
them in today's tutorials.

56:49.616 --> 56:50.015
And

56:50.795 --> 56:53.756
the only reason is just, you know, we found
something else that works better.

56:53.795 --> 56:55.056
That's the only reason.

56:56.636 --> 56:57.035
Cool.

56:58.036 --> 56:59.095
Now, we are coming

56:59.716 --> 57:00.056
to

57:01.376 --> 57:02.015
modern day

57:02.835 --> 57:04.615
machine learning, deep learning, you know.

57:05.335 --> 57:06.895
We are coming to transformers.

57:09.395 --> 57:13.755
So we have, you know, ChatGPT or Grok-3, or you
know, Claude.

57:13.755 --> 57:16.636
Any model you are using today is basically a
transformer.

57:19.455 --> 57:23.115
Uh, here is the lecture from which I first studied
transformers.

57:23.115 --> 57:24.895
I still think it's an amazing lecture.

57:30.895 --> 57:33.895
It's by this guy called Justin Johnson.

57:34.175 --> 57:39.055
It's four years old, but I think it's still
relevant, even despite being a four year old

57:39.815 --> 57:43.455
video. You know, four year old videos would
otherwise be considered ancient in

57:44.315 --> 57:47.815
deep learning speak. So if you have a video that's
four years old and it's still relevant

57:47.815 --> 57:48.475
today, you know,

57:49.275 --> 57:50.395
that means it's a good

57:51.035 --> 57:51.515
video.

57:53.715 --> 57:57.135
So yeah, here's lecture 13. This will make more
sense if you actually see some of the

57:57.155 --> 58:00.995
previous lectures. It is going to make less sense
if you just jump into it directly.

58:01.035 --> 58:04.955
But yeah, if you want to understand transformers
more in depth, that's the lecture for

58:04.975 --> 58:05.195
you.

58:06.375 --> 58:09.975
And here is an actual transformer we are going to
look at.

58:14.175 --> 58:19.115
This is LLaMA-4. It was trained by Facebook, uh,
as the model

58:19.175 --> 58:19.615
card.

58:21.755 --> 58:22.175
Yeah.

58:23.115 --> 58:26.135
Llama-4 Scout, LLaMA-4 Maverick.

58:28.235 --> 58:29.535
Hmm. So,

58:31.875 --> 58:36.795
yeah. Scout has 109 billion parameters, Maverick
has 400

58:36.815 --> 58:40.895
billion parameters. I'll explain later on, you
know, what is parameters.

58:40.915 --> 58:41.855
Right now, I'm just saying,

58:42.635 --> 58:45.015
like, this is basically the size of the model.

58:45.095 --> 58:46.915
Like, it's the size of the weight matrices.

58:46.915 --> 58:49.315
You know, we talked about, right, like W1, W2, so.

58:49.615 --> 58:51.375
Transformers also have weight matrices.

58:51.375 --> 58:51.875
This is the

58:52.555 --> 58:56.275
sum of, you know, all of the weight matrices, in
total, how big are these matrices.

58:57.295 --> 59:01.315
So there are a total of 400 billion numbers inside
all these weight tr- matrices,

59:01.535 --> 59:02.175
basically.

59:05.055 --> 59:09.495
And yeah, this was released just two months back,
in April 2025.

59:14.075 --> 59:15.515
I think I'm going to have water.

59:29.335 --> 59:34.015
Okay, cool. Uh, this was trained on five million
GPU hours.

59:34.815 --> 59:39.175
It basically means you took a GPU, you ran it for,
like, 500 years.

59:39.715 --> 59:40.735
500 years, right?

59:41.675 --> 59:42.375
Five

59:43.895 --> 59:45.395
million hours

59:46.835 --> 59:49.495
is... Okay, yeah, 570 hours. So,

59:50.435 --> 59:54.095
you take, like, 570 GPUs and you run them together
for a year

59:55.975 --> 01:00:00.875
or, or, you know, you can take more. You can take,
like, uh, like, what, 7000 GPUs and

01:00:00.895 --> 01:00:02.135
you can run them for a month.

01:00:04.815 --> 01:00:05.735
Uh, like,

01:00:06.655 --> 01:00:08.655
a GPU looks like this, by the way.

01:00:13.915 --> 01:00:15.855
Yeah. Here is what it...

01:00:18.075 --> 01:00:19.855
GPU looks like.

01:00:20.795 --> 01:00:22.455
It's just a card like this.

01:00:23.455 --> 01:00:26.615
And the cooling is, I think not... Yeah, this
includes the cooling.

01:00:27.815 --> 01:00:31.035
Like, separately you need, you know, these are the
GPUs inside and you need the cooling

01:00:31.035 --> 01:00:32.015
unit separately.

01:00:36.435 --> 01:00:37.635
So, yeah.

01:00:39.955 --> 01:00:41.975
So now I'm going to actually look at the code.

01:00:42.035 --> 01:00:44.375
The code is not that too difficult to understand.

01:00:45.575 --> 01:00:49.315
If you already have studied transformers, you
might actually understand a fair amount of

01:00:49.315 --> 01:00:52.791
the code.Yeah.

01:00:54.091 --> 01:00:58.591
So again, I don't have time to go through the
entire thing, but yeah.

01:00:59.351 --> 01:01:03.152
See here, this is called attention. This thing
called attention is the most important

01:01:03.192 --> 01:01:05.311
part of a transformer.

01:01:06.531 --> 01:01:08.832
Uh, here we defined forward.

01:01:09.752 --> 01:01:11.311
This is a forward pass, so

01:01:12.812 --> 01:01:16.711
you know, what we do to our input to get output,
that is called a forward pass.

01:01:21.211 --> 01:01:23.831
Here is forward for the transformer block.

01:01:24.111 --> 01:01:26.432
Again, I will explain all this soon.

01:01:30.251 --> 01:01:32.291
Yeah, I think this is where our actual...

01:01:36.031 --> 01:01:37.791
Yeah. Okay, so this is LLaMA 4.

01:01:39.291 --> 01:01:41.131
And where is LLaMA 4 forward?

01:01:42.652 --> 01:01:46.492
Okay, LLaMA 4 inference mode. This is where we
generate... Ah. Yeah, transformer input.

01:01:46.492 --> 01:01:46.492
Okay. Yeah. Where is this line called? 93.

01:01:46.492 --> 01:01:46.711
Yeah, okay, model equals transformer model args.

01:01:46.711 --> 01:01:47.071
So here's the input we give to the transformer and
we... No, no, no.

01:01:47.091 --> 01:01:47.131
Here's the

01:01:47.131 --> 01:01:55.772
hyperparameters

01:01:55.791 --> 01:01:56.452
and I

01:02:20.052 --> 01:02:21.351
won't explain that right now.

01:02:24.532 --> 01:02:28.451
Yeah, I think this script is less useful. It's
this one that's useful.

01:02:31.212 --> 01:02:34.131
Yeah, return transformer output.

01:02:35.831 --> 01:02:36.751
Trans...

01:02:37.372 --> 01:02:40.391
Oh, yeah, yeah, yeah. This is useful. Very useful.

01:02:40.532 --> 01:02:40.811
(sniffs)

01:02:41.431 --> 01:02:44.591
So this is where we start. We start with some
tokens.

01:02:44.591 --> 01:02:48.631
Okay, so we got self model input.

01:02:49.331 --> 01:02:53.071
Tokens equals model input or tokens. So here is
already a first step that has happened.

01:02:54.491 --> 01:02:55.171
Ah.

01:02:56.231 --> 01:02:56.771
Where are we?

01:02:59.031 --> 01:03:03.131
Yep, so here is a bunch of steps that happen in a
transformer.

01:03:03.711 --> 01:03:07.131
The first step is called tokenize. We use a
tokenizer.

01:03:08.231 --> 01:03:12.531
And tokenizer is pretty straightforward. Basically
means we...

01:03:14.931 --> 01:03:18.911
Actually first, sorry, I should maybe start with,
what the hell does a transformer do?

01:03:18.931 --> 01:03:23.431
So a transformer takes a sequence of tokens as
input and it gives you log

01:03:23.451 --> 01:03:25.331
probabilities about what the next token should be.

01:03:26.351 --> 01:03:26.651
So

01:03:27.571 --> 01:03:29.651
tokens are just words, you know, like

01:03:31.511 --> 01:03:36.351
I'll say, "My name is Samuel. What is your name?"
So my is a token, name is a

01:03:36.391 --> 01:03:38.911
token, is is a token, Samuel is a token.

01:03:39.751 --> 01:03:41.531
There's a sequence of tokens and

01:03:42.291 --> 01:03:45.451
a transformer will be given, you know, a sequence
of tokens, you know.

01:03:45.451 --> 01:03:48.031
"My name is Samuel. What is?"

01:03:49.211 --> 01:03:52.951
Predict the next token. And it will give you, you
know, log probabilities of like

01:03:54.391 --> 01:03:59.331
what is a probability, what is your probability,
what is my probability,

01:03:59.351 --> 01:04:03.611
what is their probability. Like you will get, you
know, a sequence of numbers giving the

01:04:03.651 --> 01:04:06.151
probabilities for every possible next token.

01:04:07.411 --> 01:04:10.351
So input is some tokens, output is some tokens.

01:04:12.631 --> 01:04:12.891
So

01:04:14.611 --> 01:04:17.791
yeah, first we convert our words into tokens.

01:04:18.271 --> 01:04:23.151
The tokenizer just says instead of my there is a
dictionary of maybe some 100,000

01:04:23.211 --> 01:04:26.631
popular words in English and they're all numbered,
you know, zero, one, two, three, four

01:04:26.691 --> 01:04:30.331
till 100,000. Instead of my, maybe my is, you
know, word number

01:04:31.451 --> 01:04:35.871
4,058. So instead of my we'll put, you know, 4058.

01:04:37.131 --> 01:04:39.291
Okay, my name, maybe name is, I don't know,

01:04:39.911 --> 01:04:43.711
50732 word or it's the 50,732nd word.

01:04:43.711 --> 01:04:44.311
So instead of

01:04:45.091 --> 01:04:47.511
this we'll put, you know, 50732 or whatever.

01:04:48.531 --> 01:04:51.071
Yeah, actually we won't put 50732. We will

01:04:52.371 --> 01:04:57.191
fill 100,000 zeros and on the 50732 index instead
of zero we'll put a

01:04:57.251 --> 01:04:59.611
one. That's actually how we will fill the matrix.

01:05:00.811 --> 01:05:03.431
But yeah, remember we need matrices. Our entire...

01:05:03.471 --> 01:05:07.451
All this work has to wor- do- get done on
matrices. We can't do work with words.

01:05:07.451 --> 01:05:10.851
We have to start with some X. X has to be some
matrix input.

01:05:11.531 --> 01:05:14.131
So we'll convert our words into, you know, a
matrix like this.

01:05:14.131 --> 01:05:16.351
So that's just what tokenization is.

01:05:17.891 --> 01:05:21.711
Then next step, we'll have this thing called a
positional embedding.

01:05:22.211 --> 01:05:23.551
Where's the positional embedding?

01:05:25.571 --> 01:05:27.691
Yeah, start position.

01:05:29.391 --> 01:05:32.771
Okay, so we got a position. What did we do with
the position?

01:05:35.151 --> 01:05:38.111
We converted that into frequency.

01:05:42.231 --> 01:05:46.011
What did we do with the frequencies? Okay, we took
the frequency here and

01:05:48.391 --> 01:05:50.151
we defined layer.

01:05:51.291 --> 01:05:54.331
Like, you know, if you have to sit down and
understand this code, yes, it will take time.

01:05:54.511 --> 01:05:59.491
I'm going to do... Take a shortcut. I'm going to
copy this entire code and I'm going

01:05:59.551 --> 01:06:01.391
to ask our AI to tell us

01:06:02.211 --> 01:06:03.731
what does everything here mean.

01:06:05.911 --> 01:06:06.991
Explain

01:06:07.831 --> 01:06:11.571
all the steps in simple English.

01:06:17.151 --> 01:06:17.611
Bam.

01:06:20.691 --> 01:06:25.431
Yeah, uh, O3 is a great tool for learning anything
by the way.

01:06:25.531 --> 01:06:28.891
Like O3 sometimes makes mistakes, it can
understand it.

01:06:29.091 --> 01:06:31.471
And that can be a problem if you're a beginner.

01:06:32.551 --> 01:06:35.371
But apart from that, O3 is a great way of learning
anything.

01:06:41.691 --> 01:06:42.871
Cool. So we

01:06:43.611 --> 01:06:45.711
have a huge number of steps, you know.

01:06:48.851 --> 01:06:51.851
Utility pieces. Let's skip the utility pieces
right

01:06:51.871 --> 01:06:56.295
now.Yeah, top level transformer.

01:06:56.995 --> 01:07:01.035
It stores hyper parameters, I'm going to ignore
that. It...

01:07:03.596 --> 01:07:08.055
Yeah. Yeah, this is the thing we actually want,
this is the forward pass.

01:07:09.675 --> 01:07:11.955
It converts tokens to embeddings.

01:07:12.815 --> 01:07:15.855
If there's an image, we will also convert that
into embeddings.

01:07:15.915 --> 01:07:19.335
And right now I'm not going to talk about images,
let's just consider text for now.

01:07:19.335 --> 01:07:24.216
But we can also convert an image into embeddings
and then we do that right at the

01:07:24.255 --> 01:07:27.956
start, and then, you know, rest all the steps
there on the same for text or images.

01:07:29.395 --> 01:07:29.796
Yeah, we

01:07:30.735 --> 01:07:35.096
put these positional embeddings, which is
basically along with the token we also kind of

01:07:35.136 --> 01:07:35.535
store

01:07:36.255 --> 01:07:41.155
which position it starts instead of, you know, my
name is Samuel, it becomes like my

01:07:41.255 --> 01:07:43.835
one name two is three, Samuel four.

01:07:44.616 --> 01:07:47.855
So even if you're in the middle of the sentence
somewhere you are at like Samuel four,

01:07:47.915 --> 01:07:49.635
you already know you're at the fourth word.

01:07:50.015 --> 01:07:52.495
Otherwise, when you're somewhere in between, you

01:07:53.096 --> 01:07:55.075
may have forgotten where exactly are you.

01:07:57.595 --> 01:07:59.955
Yeah, then this, this attention mask,

01:08:00.755 --> 01:08:02.255
I'm going to skip that.

01:08:06.935 --> 01:08:10.855
Yeah, I will, I will have to ask this a bunch,
bunch of more questions to actually

01:08:10.875 --> 01:08:15.095
simplify this. Like, you can ask or refer more
questions, you know, what the hell is

01:08:15.115 --> 01:08:18.735
this? You know, what is this? You can keep asking,
it will eventually simplify it, but it

01:08:18.735 --> 01:08:19.635
will take a lot of time.

01:08:20.315 --> 01:08:23.015
I have actually already simplified this stuff, and

01:08:24.755 --> 01:08:25.955
I'm going to follow that.

01:08:26.775 --> 01:08:28.295
So here is the simplified version.

01:08:29.035 --> 01:08:32.955
See, we tokenized, we put positional embeddings,
we got image embeddings.

01:08:33.595 --> 01:08:38.375
Then we have N of these, so this entire thing
actually repeats, you know, maybe 32

01:08:38.415 --> 01:08:43.115
times or 64 times or 80 times. So let's say we
repeat this 80 times.

01:08:43.855 --> 01:08:48.575
We do something called an RMS norm, we do
something called multi-headed attention, which

01:08:48.575 --> 01:08:50.755
is the most important part of this entire model.

01:08:51.635 --> 01:08:54.755
This is basically with our input, we, you know,
computed something, we computed

01:08:54.795 --> 01:08:56.735
something, we computed something, we computed
something.

01:08:57.375 --> 01:09:01.215
We did this entire thing 80 times, then we did a
projection and we got an answer.

01:09:01.875 --> 01:09:05.835
So we did something called a multi-headed
attention, we did add residual, we did RMS

01:09:05.855 --> 01:09:06.315
norm,

01:09:06.935 --> 01:09:09.955
we did a feed-forward layer, we did add residual.

01:09:10.795 --> 01:09:12.215
So this is the entire thing we did,

01:09:13.055 --> 01:09:17.715
and I don't have time to explain all of these in
detail, so I'm just going to explain the

01:09:17.715 --> 01:09:19.155
steps which are really important.

01:09:20.575 --> 01:09:22.795
Steps which are important and the steps which are
simple.

01:09:23.795 --> 01:09:28.615
So the steps which are simple are like tokenizing
is simple, embedding is simple, RMS

01:09:28.675 --> 01:09:29.535
norm is simple,

01:09:30.355 --> 01:09:33.395
residual is simple. So I'll just explain some of
these which are, like, very simple,

01:09:34.275 --> 01:09:37.615
and the thing which is important. So the important
thing is this projection

01:09:38.315 --> 01:09:40.535
and this, uh, dot product attention.

01:09:42.675 --> 01:09:46.095
So I'm, I'm going to explain some of these more
and some of these less.

01:09:48.455 --> 01:09:52.795
So anyways, keeping the big picture, we started
with the sequence of tokens,

01:09:53.675 --> 01:09:56.575
we did this giant amount of calculation on it,

01:09:57.435 --> 01:10:02.115
we got output, you know, log probabilities for
what the next token should be,

01:10:03.195 --> 01:10:07.175
and this entire thing is described in this, you
know, 500 lines of code here.

01:10:07.175 --> 01:10:11.335
If you go through this, like most of these steps
are all, all the code is just sitting

01:10:11.335 --> 01:10:15.255
here. Some of it, a little bit might be setting in
ffn.pie for example.

01:10:15.995 --> 01:10:18.195
So here this, you know, another 50 lines of code
but,

01:10:18.835 --> 01:10:22.535
yeah, the vast majority of the code for this
entire process is just sitting in this one

01:10:22.595 --> 01:10:23.895
script. If you actually

01:10:24.755 --> 01:10:29.315
sit and read this properly you will understand
everything I've talked about here is

01:10:29.315 --> 01:10:30.235
actually sitting in there.

01:10:31.155 --> 01:10:35.075
And okay, a bunch of these steps are all actually
optional, like rotary embedding,

01:10:35.095 --> 01:10:39.835
optional, headwise RMS norm, optional, temperature
tuning for long context, this is

01:10:39.895 --> 01:10:43.115
optional. So you can actually, you can even kind
of ignore these for now.

01:10:43.175 --> 01:10:44.155
These are not important.

01:10:46.975 --> 01:10:51.035
Cool. So we started with the sequence of tokens,
we did all sorts of

01:10:51.375 --> 01:10:55.115
computation on it, we got an output, now what do
we do with this output?

01:10:57.215 --> 01:10:58.575
With these log probabilities?

01:11:05.215 --> 01:11:08.975
Remember right now, actually these log
probabilities are completely inaccurate, they're

01:11:08.995 --> 01:11:09.715
like garbage,

01:11:11.115 --> 01:11:13.035
because we don't know what our weight matrices
are.

01:11:13.555 --> 01:11:16.495
So like when we do these calculations we will need
weight matrices.

01:11:16.495 --> 01:11:16.795
So like

01:11:17.495 --> 01:11:21.615
we will need a weight matrix here for embeddings,
we will need a weight matrix for the

01:11:21.635 --> 01:11:22.455
projection,

01:11:23.295 --> 01:11:25.975
we will need a weight matrix for this projection,

01:11:27.855 --> 01:11:31.435
we will need... Actually we'll need three weight
matrices here,

01:11:32.855 --> 01:11:35.395
we will need weight matrices for the feed-forward
layer.

01:11:35.415 --> 01:11:38.235
So there's a lot of weight matrices here which
right now we don't know,

01:11:38.855 --> 01:11:41.715
and we're just going to initialize them with some
junk values.

01:11:42.795 --> 01:11:44.515
And yep, we are

01:11:45.235 --> 01:11:47.695
going to do gradient descent remember? Yeah.

01:11:48.295 --> 01:11:52.395
We are going to define some loss at the end and we
are going to do dL by dW.

01:11:53.695 --> 01:11:55.355
So first of all, what is loss?

01:11:57.435 --> 01:12:01.775
Loss is how much our predicted answers differ from
the actual answers.

01:12:02.415 --> 01:12:05.135
So we will get some log probabilities of the next
token,

01:12:05.955 --> 01:12:09.975
and we know the actual next token. So right now
with that data we are assuming we

01:12:09.975 --> 01:12:11.315
actually know the next token.

01:12:12.175 --> 01:12:16.395
So now this, given this and this, we can mul- dot
product these two, we will get a

01:12:16.415 --> 01:12:20.635
cross-entropy loss. This is just a value telling
us, you know, how bad is our prediction,

01:12:20.655 --> 01:12:23.955
which right now when we start off it's pretty,
pretty bad.

01:12:25.855 --> 01:12:27.835
Then we compute, you know, gradients,

01:12:29.635 --> 01:12:32.575
you know, dL by dW, and there are a lot of Ws
here.

01:12:32.595 --> 01:12:37.075
There's, you know, this projection W, there's, you
know, the dot product W, there is a...

01:12:37.815 --> 01:12:39.475
No, sorry, here there's no W, there's,

01:12:40.455 --> 01:12:43.115
yeah, this, this W, there's, you know, tokenizing
W.

01:12:43.115 --> 01:12:46.215
But yeah, the most important weight matrices are
actually this one, the one in the

01:12:46.215 --> 01:12:48.875
projection step, and multi-headed attention.

01:12:49.035 --> 01:12:53.155
When I describe multi-headed attention I'll talk
to you about what this projection step

01:12:53.175 --> 01:12:58.024
actually is.So,

01:12:58.104 --> 01:13:00.983
anyways, there is a weight matrix that is required
for this.

01:13:00.983 --> 01:13:01.383
We will do

01:13:02.344 --> 01:13:07.223
d loss by that W. We will then get, we will then
update that

01:13:07.344 --> 01:13:07.703
W.

01:13:08.963 --> 01:13:11.803
We will set a good epsilon. Remember epsilon?

01:13:12.043 --> 01:13:12.603
Uh,

01:13:13.723 --> 01:13:17.764
epsilon is basically once we know the gradient,
how much do we nudge our weight matrices,

01:13:17.764 --> 01:13:20.403
right? Do we nudge our weights a little bit or do
we nudge them a lot?

01:13:21.024 --> 01:13:24.023
So, we will use Adam with weight decay to

01:13:26.023 --> 01:13:28.003
decide how much to update our weights.

01:13:29.243 --> 01:13:33.623
And then, we will spend hundreds of billions of
dollars, uh,

01:13:34.424 --> 01:13:39.243
running this gradient descent, just updating W,
update W, update W, and you spend

01:13:39.264 --> 01:13:40.923
hundreds of billions of dollars this way.

01:13:42.083 --> 01:13:42.663
Uh,

01:13:43.743 --> 01:13:47.703
yeah, this might distract if I bring this up right
now. But actually, I'll just...

01:13:55.683 --> 01:13:58.103
Is my note not working? What? Hmm.

01:13:59.563 --> 01:14:02.643
Yeah, my note is fine. It's slow.

01:14:03.283 --> 01:14:03.703
Okay.

01:14:04.323 --> 01:14:07.503
(instrumental music plays)

01:14:08.743 --> 01:14:13.323
Stargate, put that name down in your books, 'cause
I think you're gonna hear a lot about

01:14:13.343 --> 01:14:18.163
it in the future. A new American company that will
invest 500 billion dollars

01:14:18.263 --> 01:14:20.423
at least in AI infrastructure.

01:14:21.143 --> 01:14:23.163
The data centers are actually under construction.

01:14:23.363 --> 01:14:25.623
The first of them are under construction in Texas.

01:14:26.163 --> 01:14:28.463
The Abilene location, which is our first location.

01:14:29.283 --> 01:14:33.003
Abilene is a town on the western central plains of
Texas.

01:14:33.363 --> 01:14:36.323
We kinda scratch and fight for everything good
that comes our way.

01:14:36.363 --> 01:14:40.603
I had people reaching out and they said, "When is
the President going to come see you?"

01:14:40.923 --> 01:14:42.863
And I said, "Well, he hasn't texted me yet, so."

01:14:43.003 --> 01:14:44.263
(laughs)

01:14:44.263 --> 01:14:46.623
I'm thrilled we get to do this in the United
States of America.

01:14:46.903 --> 01:14:49.523
I think this will be the most important project of
this era.

01:14:51.963 --> 01:14:56.883
Welcome to Abilene, Texas. This is the site of
Project

01:14:56.963 --> 01:15:01.883
Stargate, a very mysterious, much talked about,
much speculated about project,

01:15:01.983 --> 01:15:02.783
that brings together-

01:15:03.223 --> 01:15:06.483
Yeah, so by the way, this is a documentary on the

01:15:07.963 --> 01:15:12.463
largest computer data center being built in the
world, as of, you know, June

01:15:12.523 --> 01:15:14.223
2025. Uh,

01:15:16.623 --> 01:15:19.643
you know, one year later this data might be
outdated, but as of today it's the biggest

01:15:19.643 --> 01:15:21.343
one, uh, under construction.

01:15:21.983 --> 01:15:26.583
For OpenAI they've committed like $500 billion
over, I think, four years.

01:15:26.923 --> 01:15:29.303
Like, the first year investment has gone through
for sure.

01:15:29.483 --> 01:15:31.103
The remaining years seems uncertain.

01:15:32.543 --> 01:15:35.803
Uh, I only bring this up because the point is, you
know,

01:15:36.863 --> 01:15:41.723
we're spending all this money to ultimately update
the weight matrix here,

01:15:41.883 --> 01:15:44.103
basically. And remember, there are 80 of them.

01:15:44.203 --> 01:15:46.463
Like, they could be 80, they could be more than
100.

01:15:46.483 --> 01:15:46.783
Like,

01:15:47.883 --> 01:15:51.483
we do this, you know, 100 times so there are 100
weight matrices.

01:15:51.543 --> 01:15:54.903
We are going to keep updating those weight
matrices 100 times.

01:15:56.543 --> 01:15:58.283
That's what the data center is for.

01:16:00.003 --> 01:16:04.703
So yeah, now let me talk about what are these
weight matrices that we are going to

01:16:04.723 --> 01:16:05.243
update

01:16:06.403 --> 01:16:08.503
by spending billions of dollars on them.

01:16:10.143 --> 01:16:10.583
Yeah.

01:16:12.723 --> 01:16:13.663
So, this is

01:16:14.323 --> 01:16:15.543
the attention block.

01:16:16.763 --> 01:16:19.983
So, the most important part is that x is our
input. You remember, right?

01:16:19.983 --> 01:16:23.483
We started with images here, we're starting with
text. So x is just an input.

01:16:24.563 --> 01:16:28.943
It has a huge number of examples, and each example
has, you know, an array of

01:16:28.983 --> 01:16:29.603
numbers.

01:16:30.823 --> 01:16:32.563
So, these are the tokens essentially.

01:16:34.063 --> 01:16:37.663
We're going, our weight matrices are WQ, WK and
WV.

01:16:38.503 --> 01:16:42.663
We multiply x with each of these, we'll get, you
know, three matrices, QKV.

01:16:43.683 --> 01:16:46.923
Then we will do this thing and we will get y,
which is our output.

01:16:48.803 --> 01:16:51.983
And this is the most important step in a
transformer.

01:16:52.123 --> 01:16:56.103
That's why I'm going to explain it a lot in
detail, because it's the most important.

01:16:57.523 --> 01:17:01.283
So, I guess this is straightforward, you
understood what's a prediction.

01:17:02.323 --> 01:17:05.523
Now we are doing Q multiplied by K transpose.

01:17:05.643 --> 01:17:08.223
So K transpose, we added a mask.

01:17:09.623 --> 01:17:10.763
A mask

01:17:11.403 --> 01:17:15.543
is a matrix that has some values which are
hardcoded to zero, and

01:17:17.243 --> 01:17:18.903
some values are hardcoded to one.

01:17:19.923 --> 01:17:23.303
Actually, no, ma- mask has, a mask has some value
zero, some values one.

01:17:23.323 --> 01:17:24.823
So when you add a zero, nothing happens.

01:17:24.823 --> 01:17:28.823
But when you add a one, this entire thing gets,
like, locked into one.

01:17:30.543 --> 01:17:30.903
I'll,

01:17:31.523 --> 01:17:34.343
I'll explain to you what softmax is, then it will
make sense.

01:17:35.023 --> 01:17:36.903
So softmax is basically,

01:17:37.983 --> 01:17:41.803
if you have a matrix, you want to do softmax on
this matrix, it's like, for the first,

01:17:42.223 --> 01:17:44.023
you will first do e to the power all the values.

01:17:44.023 --> 01:17:46.943
So just all the values that are there, just
replace them with e to the power of that

01:17:46.943 --> 01:17:49.823
value. So you started with, you know, one, two,
three, four.

01:17:51.043 --> 01:17:55.743
Okay no, one, two, three, minus one. So e to the
power one, e to the power two, e to the

01:17:55.763 --> 01:17:57.243
power three, e to the power minus one.

01:17:57.283 --> 01:18:00.103
So we just replaced all the values with e to the
power of the same value.

01:18:01.983 --> 01:18:05.203
Then we will divide each value with the sum of
those rows.

01:18:05.203 --> 01:18:05.443
So,

01:18:07.003 --> 01:18:08.503
instead of one we now have

01:18:09.103 --> 01:18:13.303
e to the power one upon e to the, divided by e to
the power one plus e to the power two.

01:18:14.203 --> 01:18:16.323
Like let's say our values in that row were one and
two.

01:18:17.663 --> 01:18:21.063
Now that two will become, you know, e to the power
two divided by e to the power one plus

01:18:21.063 --> 01:18:21.763
e to the power two.

01:18:22.523 --> 01:18:24.683
On the next row we had what, three and minus one,
right?

01:18:24.683 --> 01:18:28.483
So three will become e to the power three divided
by e to the power three plus e to the

01:18:28.483 --> 01:18:29.203
power minus one.

01:18:29.923 --> 01:18:31.283
Minus one will become, you know...

01:18:32.423 --> 01:18:37.123
Yeah, maybe I can just give an example here that
seems pretty easy to do, I should maybe

01:18:37.243 --> 01:18:39.063
just actually give an example here

01:18:40.103 --> 01:18:41.303
of softmax.

01:18:49.023 --> 01:18:57.663
Softmax...

01:18:58.443 --> 01:19:00.223
One, two, three, four, five, six, seven,

01:19:00.223 --> 01:19:04.592
eight.This

01:19:04.611 --> 01:19:05.231
one...

01:19:08.931 --> 01:19:10.371
Equals...

01:19:23.711 --> 01:19:24.872
Square.

01:19:27.071 --> 01:19:28.651
By equaling square.

01:19:29.531 --> 01:19:32.311
B square by B plus

01:19:33.331 --> 01:19:34.491
A square.

01:19:35.571 --> 01:19:37.471
And cube by...

01:19:39.952 --> 01:19:42.331
Cube plus, you know, one by eight.

01:19:44.092 --> 01:19:46.311
And cube by...

01:19:48.952 --> 01:19:51.931
And cube plus one by, one by eight. (typing sound)
Yeah.

01:19:52.291 --> 01:19:56.092
(hums) (typing sound) Yeah. So here is an example.

01:19:56.092 --> 01:19:56.092
Here, let me just call this an example. (hums)
(typing sound) Yeah. Nevermind.

01:19:56.092 --> 01:19:56.092
So here's an, here's an example. This is what
softmax actually does.

01:19:56.092 --> 01:19:59.472
If you want an intuition for what softmax

01:20:00.231 --> 01:20:01.011
does,

01:21:06.071 --> 01:21:10.751
basically the

01:21:10.791 --> 01:21:14.091
large values in a row, they become close to one,
the smaller values in a row, they become

01:21:14.131 --> 01:21:14.851
close to zero.

01:21:15.791 --> 01:21:18.611
If we had done row normalization, that's what
would have happened anyway.

01:21:18.711 --> 01:21:18.911
Like

01:21:20.231 --> 01:21:22.531
if we had just done, you know, two divided by one
plus two

01:21:23.631 --> 01:21:26.871
(laughs) , basically the large values like two
divided by one plus two is two by three.

01:21:27.031 --> 01:21:30.371
So that's close to one and one divided by one plus
two is, you know,

01:21:31.311 --> 01:21:35.311
one by three that's close to zero. But we, because
we did this E to the power, it's just

01:21:35.351 --> 01:21:39.951
going to be that thing on steroids. So two will
become even more close to one and this

01:21:39.971 --> 01:21:41.571
one will become even more close to zero.

01:21:42.211 --> 01:21:45.331
This three will go even more close to one, and
this will go even more close to zero.

01:21:46.371 --> 01:21:49.091
Like this thing is almost one basically, and

01:21:49.811 --> 01:21:51.531
this thing is almost zero basically.

01:21:52.691 --> 01:21:53.151
I mean I,

01:21:53.871 --> 01:21:57.391
yeah maybe I'll just compute these values and
leave it like why not?

01:21:58.331 --> 01:22:00.411
I guess that's 0.268.

01:22:09.071 --> 01:22:11.991
Square, 0.731.

01:22:13.711 --> 01:22:15.171
0.731.

01:22:24.771 --> 01:22:26.711
0.982.

01:22:39.271 --> 01:22:40.131
Ah.

01:22:45.591 --> 01:22:55.191
(hums)

01:22:57.591 --> 01:22:59.071
0.208.

01:23:01.371 --> 01:23:02.731
208.

01:23:14.631 --> 01:23:15.651
Yeah, so...

01:23:17.311 --> 01:23:18.551
Motherfucker.

01:23:24.631 --> 01:23:29.611
Yes, so the one was already like a small value
went close to zero, two

01:23:29.631 --> 01:23:32.411
was big, it went close to one. Like even within
the row.

01:23:33.091 --> 01:23:35.951
Three went close to one, one minus one went really
close to zero.

01:23:37.011 --> 01:23:38.511
So softmax does this.

01:23:39.411 --> 01:23:42.631
And remember before we do the softmax, we're going
to add a mask.

01:23:43.231 --> 01:23:44.931
So the mask basically says

01:23:46.191 --> 01:23:48.231
some of these values we're just going to add

01:23:49.131 --> 01:23:53.711
something so large that, well, only that gets paid
attention to or something so

01:23:53.791 --> 01:23:56.671
small, you know, that is not going to get paid
attention to at all.

01:23:57.831 --> 01:24:00.911
Basically, we are going to hard set some of these
values to one and some of them to zero,

01:24:01.031 --> 01:24:04.491
depending on what the mask says. I'll talk about
what the mask says.

01:24:04.551 --> 01:24:04.731
But

01:24:06.011 --> 01:24:06.291
yeah,

01:24:07.251 --> 01:24:09.711
so we're going to do soft mask of this entire
thing.

01:24:10.191 --> 01:24:12.311
Oh here we're just dividing a constant value.

01:24:13.111 --> 01:24:17.171
The dimension of q, so if q is, you know, 1000
dimensions we're just going to do square

01:24:17.211 --> 01:24:20.871
root 1000 which is 30, we're just going to divide
all the values by 30.

01:24:22.151 --> 01:24:25.451
It's not that important. But anyways we did
softmax something

01:24:26.151 --> 01:24:29.231
and then the softmax thing we are going to
multiply with V.

01:24:29.351 --> 01:24:31.251
So V is again X at WV.

01:24:33.871 --> 01:24:36.051
So softmax something times V.

01:24:37.731 --> 01:24:42.431
So this is basically telling us, like in this
matrix V, we want to pay attention to some

01:24:42.491 --> 01:24:46.331
parts. Like some parts of V are important and some
parts of V are not important.

01:24:47.091 --> 01:24:51.651
And this entire softmax thing is going to tell us
what parts of V are important and what

01:24:51.671 --> 01:24:52.931
parts of V are not important.

01:24:54.171 --> 01:24:57.971
Oh, and by the way, you should always, always,
always remember

01:24:59.191 --> 01:24:59.691
this.

01:25:00.631 --> 01:25:03.431
Deep learning is alchemy. Deep learning is black
box.

01:25:03.451 --> 01:25:08.435
We don't have mathematical proofs for
anything.When I'm telling you, okay, you know,

01:25:08.455 --> 01:25:12.355
here is all the reasons, these are all intuitions.
We don't have proof for all this.

01:25:13.516 --> 01:25:16.856
There might be some other secret reason why this
thing is working well, which we have no

01:25:16.856 --> 01:25:17.516
idea about.

01:25:18.795 --> 01:25:22.835
Uh, I'm just giving some intuitions of, you know,
why does this particular formula happen

01:25:22.855 --> 01:25:23.695
to look so good.

01:25:25.036 --> 01:25:26.175
You know, this, this thing.

01:25:30.795 --> 01:25:31.876
Yeah. So,

01:25:33.595 --> 01:25:35.915
this basically says... So,

01:25:36.915 --> 01:25:38.116
we are going to

01:25:38.875 --> 01:25:42.695
give a lot of weight to some parts of v and we're
going to ignore other parts of v.

01:25:44.396 --> 01:25:47.156
That's what softmax times means. So now putting
this all together.

01:25:47.895 --> 01:25:50.055
First, we, what, we multiplied with...

01:25:50.955 --> 01:25:54.195
We multiplied Q and K transpose, we added a mask.

01:25:54.716 --> 01:25:57.155
Mask said, "Okay, let's hide something." Some
things are

01:25:58.235 --> 01:26:01.535
as good as zero. We're just not going to consider
information.

01:26:02.295 --> 01:26:04.276
We are going to softmax it, which means

01:26:05.295 --> 01:26:08.595
only some information we pay attention to, some
information we don't pay attention to.

01:26:08.615 --> 01:26:12.455
That's step two. Step three, we are going to do
this on v.

01:26:13.395 --> 01:26:16.795
So, at least some parts of v get paid attention to
and some parts of v don't get paid

01:26:16.815 --> 01:26:17.595
attention to.

01:26:18.215 --> 01:26:20.515
So, when you put this all together, it basically
says

01:26:21.875 --> 01:26:24.355
we are hiding some parts of x, which is our input.

01:26:24.575 --> 01:26:27.155
Like, rem- remember, QKV are not some magic
things.

01:26:27.175 --> 01:26:30.055
QKV are just X multiplied with different weight
matrices.

01:26:33.155 --> 01:26:35.655
So, we are hiding some part of X from ourselves.

01:26:36.915 --> 01:26:38.255
We hide some part of X.

01:26:39.455 --> 01:26:39.855
We

01:26:42.295 --> 01:26:45.855
pay more attention to another part of X. Remember,
this is also X.

01:26:46.375 --> 01:26:48.935
Like, it's kind of like, you know, three
incarnations of X.

01:26:49.735 --> 01:26:50.195
Like...

01:26:52.555 --> 01:26:56.195
Yeah, like this is, you know, first incarnation,
this is the second incarnation, and the

01:26:56.235 --> 01:26:57.375
third incarnation. So the

01:26:58.755 --> 01:27:02.295
third... The first two incarnations of X, along
with the mask,

01:27:03.635 --> 01:27:07.175
or first two incarnations of X with some values
masked out tell you which

01:27:07.795 --> 01:27:11.195
parts of third incarnation of X to, you know, keep
and which to discard.

01:27:11.475 --> 01:27:12.475
It's kind of like this.

01:27:16.615 --> 01:27:17.355
Yeah. And

01:27:19.835 --> 01:27:22.235
then, we get an output.

01:27:24.195 --> 01:27:27.435
And now remember, we don't actually know these
weight matrices either.

01:27:27.435 --> 01:27:29.515
We're going to find them using gradient descent.

01:27:30.995 --> 01:27:35.575
We're going to start with some really bad values
of WQ, WK and

01:27:35.675 --> 01:27:38.915
WV. We are going to compute a loss at the end
after, you know,

01:27:40.035 --> 01:27:41.635
80 rounds of all this crap.

01:27:42.675 --> 01:27:44.775
Uh, we're going to get a loss at the end.

01:27:45.495 --> 01:27:50.155
We're going to compute the dL by dW. Like for all
these weight matrices, we're going to

01:27:50.175 --> 01:27:55.135
get a dL by dWq, dL by dWk, dL by dV. We're going
to update

01:27:55.135 --> 01:27:58.295
these weight matrices so they're like, are a
little bit less bad.

01:27:58.755 --> 01:28:02.775
We're going to do this, you know, trillions of
times until these are actually pretty

01:28:02.835 --> 01:28:03.135
good.

01:28:04.675 --> 01:28:07.295
So ultimately, we want to find really good
matrices here

01:28:08.195 --> 01:28:12.455
that tell us basically in like exactly which
situation, what part of X to pay attention

01:28:12.475 --> 01:28:14.335
to, what part of X to not pay attention to.

01:28:15.535 --> 01:28:17.635
And yeah, remember there are going to be 80 of
these.

01:28:17.635 --> 01:28:17.915
So,

01:28:18.995 --> 01:28:23.895
we are going to do like 80 rounds of this thing
with 80 different sets of, you know,

01:28:24.055 --> 01:28:25.115
weight matrices.

01:28:27.595 --> 01:28:31.115
Sorry, I think that's a lot of things, moving
pieces, but yeah.

01:28:31.115 --> 01:28:33.075
So, in one round we have

01:28:34.455 --> 01:28:36.415
these three weight matrices, which we don't know.

01:28:37.335 --> 01:28:41.255
We did this game about paying attention to some
parts of our input.

01:28:41.875 --> 01:28:45.975
We repeat this with another new set of three
matrices with some other in- with, like,

01:28:46.015 --> 01:28:49.375
that... The first round's output goes as the
second thing's input.

01:28:50.255 --> 01:28:53.215
And there we have another three weight matrices.
We do again this game.

01:28:53.855 --> 01:28:55.855
Then we do it again with another set of weight
matrices.

01:28:55.855 --> 01:28:58.035
Like this, we have, you know, 80 sets of matrices.

01:28:58.535 --> 01:29:00.515
Finally, we get an output right at the end.

01:29:00.855 --> 01:29:01.715
Then we compute

01:29:02.395 --> 01:29:06.635
the gradients on all these 80 sets of matrices. We
update all the 80 sets of matrices.

01:29:06.635 --> 01:29:08.955
Then we come back, we redo this.

01:29:09.915 --> 01:29:11.935
So, that's how this entire thing is working.

01:29:26.355 --> 01:29:26.835
Yeah.

01:29:30.375 --> 01:29:32.915
Yeah. Now, here are some intuitions on

01:29:33.615 --> 01:29:35.755
why does this technique work in the first place?

01:29:35.815 --> 01:29:36.375
Remember,

01:29:37.175 --> 01:29:41.115
we don't actually know. This is guesswork. But our
guesswork is first of all...

01:29:42.195 --> 01:29:44.295
This entire thing can be done parallel.

01:29:45.835 --> 01:29:50.555
Like otherwise, if, if you did not know anything
about Transformers, you just knew,

01:29:52.055 --> 01:29:56.035
uh, this like, okay, look, here is our input, our
input is, you know, billions of tokens

01:29:56.055 --> 01:29:57.515
for which we know the next token.

01:29:58.355 --> 01:30:01.475
And now for some tokens we don't know, we have to
predict the next token.

01:30:02.375 --> 01:30:05.975
Like, you will typically formulate this as like a
sequential problem, you know.

01:30:07.075 --> 01:30:10.635
Like, you know, with weather data is kind of
sequential or, you know, stock market data

01:30:10.635 --> 01:30:12.075
is, you know, sequential. We'll put, you know,

01:30:12.935 --> 01:30:17.775
X1, X2, X3, X4, X5, X6. And now, okay, given each
of these, you know, pred X7s you'll do

01:30:18.175 --> 01:30:20.895
some calculation on X1, you'll do some calculation
on X2.

01:30:21.375 --> 01:30:24.475
X is the first token, then something on the second
token, you'll do some calculation on

01:30:24.495 --> 01:30:28.175
the third token. You will go till the Nth token
and then you'll try to predict, you know,

01:30:28.195 --> 01:30:29.875
what the N plus one token, like...

01:30:30.535 --> 01:30:33.955
Most algorithms, if you'll try on this type of
problem, are sequential.

01:30:37.075 --> 01:30:38.475
So, this could

01:30:39.895 --> 01:30:44.015
be, you know, n-grams. Well, n-grams you can kind
of parallelize too.

01:30:44.095 --> 01:30:45.775
Okay, le- leave aside n-grams. But,

01:30:46.815 --> 01:30:47.595
uh...

01:30:49.855 --> 01:30:50.955
Where were we? Yeah.

01:30:51.935 --> 01:30:55.655
Yeah. So RNNs were a very popular method called
recurrent neural networks.

01:30:56.095 --> 01:31:00.475
Now nobody uses RNNs, but once upon a time, RNNs
were used, and

01:31:01.555 --> 01:31:04.055
they do this type of sequential computation.

01:31:04.075 --> 01:31:04.375
Like,

01:31:05.175 --> 01:31:08.575
you have a weight matrix that does something on
the first metoken, then weight matrix

01:31:08.595 --> 01:31:11.375
does something on the second token, then something
on the third token and so

01:31:11.435 --> 01:31:15.463
on.Ah.

01:31:16.583 --> 01:31:21.443
And the problem with training an RNN, especially a
really big RNN,

01:31:23.583 --> 01:31:24.163
is

01:31:25.243 --> 01:31:27.063
you have to do,

01:31:28.064 --> 01:31:31.043
like you have to update a lot of gradients in
sequence.

01:31:32.084 --> 01:31:36.484
First of all, that's slower, like GPUs can do
computational parallel stuff faster

01:31:37.144 --> 01:31:41.223
compared to doing a lot of sequential stuff and
then do gradients across all of it.

01:31:42.703 --> 01:31:47.463
And there is this vanishing and exploding
gradients problem, which is a common problem if

01:31:47.504 --> 01:31:49.243
you do a lot of machine learning, which is

01:31:49.963 --> 01:31:53.724
sometimes these gradients become very s- close to
zero and sometimes they become very

01:31:53.763 --> 01:31:54.744
close to infinity.

01:31:55.603 --> 01:31:59.604
And when that happens, you need to have some
tricks up your sleeve to deal with it.

01:32:00.063 --> 01:32:03.103
I mean ideally it should not happen, but if it
happens you need to have some tricks to

01:32:03.123 --> 01:32:03.843
deal with it.

01:32:05.183 --> 01:32:05.823
And,

01:32:07.103 --> 01:32:08.103
yeah, one trick

01:32:08.883 --> 01:32:12.383
to make sure this happens less is just don't do a
lot of sequential steps.

01:32:12.383 --> 01:32:15.983
If you do a lot of sequential steps you're more
likely to get gradients that go to zero

01:32:16.003 --> 01:32:16.963
or go to infinity.

01:32:23.543 --> 01:32:24.523
Yeah, so,

01:32:26.323 --> 01:32:27.083
that's why

01:32:28.403 --> 01:32:32.523
this attention we are doing it is good, because we
use this concept of an attention mask

01:32:32.543 --> 01:32:33.223
which just says

01:32:33.823 --> 01:32:36.823
some parts of this we are going to hide now. Which
parts of this we are going to hide?

01:32:37.663 --> 01:32:41.183
That actually we can choose. Like we can choose
what to put in this attention mask.

01:32:41.203 --> 01:32:43.723
Like we can hide the last token and say like
predict this.

01:32:44.083 --> 01:32:46.483
We can hide the first token and say, you know, go
predict this.

01:32:47.083 --> 01:32:50.423
We can hide the fourth token and say, you know,
now let's predict the fourth token.

01:32:51.343 --> 01:32:55.323
And this, when we are training the model we can
keep changing the attention mask as we

01:32:55.343 --> 01:32:55.843
choose.

01:32:57.523 --> 01:33:00.803
But this means now our entire problem is like
parallelized, like

01:33:01.483 --> 01:33:04.063
we had you know ten tokens, we hid the fifth
token.

01:33:04.063 --> 01:33:07.343
We know the first four and the last five and now
we are predicting the fifth token.

01:33:08.323 --> 01:33:11.643
And we will do another gradient descent to predict
good values of the fifth token.

01:33:13.723 --> 01:33:15.323
So this is parallelized and,

01:33:16.963 --> 01:33:21.403
yeah, another good thing about this is we can
dynamically decide what to pay attention

01:33:21.463 --> 01:33:22.043
to, like

01:33:22.643 --> 01:33:25.263
here we have lots of tokens, you know, which of
these are important?

01:33:25.263 --> 01:33:25.903
Like if I say

01:33:26.703 --> 01:33:29.263
my name is Samuel what is

01:33:30.343 --> 01:33:31.203
question mark.

01:33:32.083 --> 01:33:36.863
Now what is might be more important there and
maybe Samuel is not important there.

01:33:36.863 --> 01:33:37.163
Like,

01:33:38.203 --> 01:33:41.143
if it was my name is Alice, your name,

01:33:42.083 --> 01:33:42.423
sorry,

01:33:43.323 --> 01:33:47.023
what is question mark, you know, the answer is
probably the same, you know, that being

01:33:47.043 --> 01:33:48.743
Samuel or Alice makes no difference.

01:33:49.783 --> 01:33:53.943
But what is, that is is very important because it
says okay here is a verb, now after

01:33:53.963 --> 01:33:56.963
this verb you can maybe put a preposition, you can
put a noun.

01:33:57.243 --> 01:33:59.063
You can't put another verb, that's for sure.

01:33:59.923 --> 01:34:01.083
These types of rules

01:34:01.983 --> 01:34:06.363
you can reason about. So you need to know which
parts of this sentence (coughs) ,

01:34:06.823 --> 01:34:09.063
and by the way right now I gave you just six
words, right?

01:34:09.103 --> 01:34:11.403
In practice they could be in the thousands of
words.

01:34:12.503 --> 01:34:12.923
Like,

01:34:14.343 --> 01:34:18.603
like remember O3s here are giving you thousands of
words of output, so it needs to be

01:34:18.623 --> 01:34:20.563
able to do this reasoning on thousands of words.

01:34:20.563 --> 01:34:20.783
So

01:34:21.683 --> 01:34:24.943
it needs to be able to dynamically pay attention
to what is important here.

01:34:25.603 --> 01:34:29.763
Now some older methods like CNNs, convolutional
neural networks were like a really

01:34:29.783 --> 01:34:31.163
popular method. Again,

01:34:32.043 --> 01:34:35.063
extinct now. Nobody uses CNNs anymore, but

01:34:36.403 --> 01:34:40.843
like at most today they, people do like one CNN
layer to convert your image into an

01:34:40.863 --> 01:34:44.423
embedding and then they fit that into your
transformer.

01:34:44.543 --> 01:34:48.123
But yeah, nobody trains like end-to-end, you know,
CNNs anymore.

01:34:48.323 --> 01:34:48.563
(coughs)

01:34:50.523 --> 01:34:50.843
So

01:34:52.323 --> 01:34:56.443
CNNs basically maintain these sliding windows and
like window, window, window, window.

01:34:57.163 --> 01:34:59.263
If you have multiple layers you have multiple
windows

01:35:00.243 --> 01:35:04.443
that decide, okay, now you have to pay attention
to this part of the sequence, then we

01:35:04.443 --> 01:35:07.163
will pay attention to this, then we will take a
smaller window there.

01:35:07.923 --> 01:35:10.423
Like if you study CNNs you will understand what I
am talking about.

01:35:10.923 --> 01:35:14.163
But CNNs kind of fi- hard code this thing what to
pay attention to.

01:35:15.063 --> 01:35:17.703
LSTMs and RNNs, uh,

01:35:18.843 --> 01:35:22.483
maintain one context across the entire thing which
is kind of bad.

01:35:22.703 --> 01:35:23.183
Uh,

01:35:25.263 --> 01:35:26.643
so it's like if you had to do

01:35:27.823 --> 01:35:30.743
my name is Samuel what is question mark.

01:35:31.523 --> 01:35:33.563
Let us say the context said okay

01:35:34.883 --> 01:35:38.303
what is, is paid more attention to and

01:35:39.023 --> 01:35:41.383
name is paid more attention to. Cool.

01:35:41.843 --> 01:35:46.523
Now, okay we predicted the next token, we said
next token is your.

01:35:46.923 --> 01:35:51.163
Now again we want to predict, okay, my name is
Samuel, what is your

01:35:51.643 --> 01:35:56.283
question mark. Now again attention we will say
okay name important, what is

01:35:56.323 --> 01:35:57.723
important, but actually now

01:35:58.383 --> 01:36:01.003
what is is not important. Name is very important
because

01:36:01.843 --> 01:36:05.963
my name is Samuel, what is your question mark.
It's probably going to be name, right?

01:36:06.803 --> 01:36:11.323
So we need to be able to now dynamically change,
okay, now we don't want to pay attention

01:36:11.363 --> 01:36:14.003
to those things, we want to pay attention to a
different thing like

01:36:15.563 --> 01:36:20.043
we need to be able to change this context like
even as we go thr- through each token we

01:36:20.063 --> 01:36:23.463
need to be changing our context of what is
important to pay attention to.

01:36:24.143 --> 01:36:28.783
And LSTMs and RNNs can't do that as well as a
transformer can.

01:36:30.083 --> 01:36:32.863
So yeah, that's another reason transformers are
good.

01:36:39.003 --> 01:36:42.223
Yeah. Here is some more, you know, miscellaneous
stuff.

01:36:42.323 --> 01:36:42.643
(coughs)

01:36:43.563 --> 01:36:44.843
In the transformer

01:36:46.083 --> 01:36:49.483
I guess tokenization again already briefly talked
about

01:36:50.463 --> 01:36:55.143
you convert words into numbers. Positional
embeddings is basically instead of

01:36:55.183 --> 01:36:55.463
saying

01:36:56.123 --> 01:37:01.003
my name is Samuel it becomes my one name two is
three Samuel four, and

01:37:01.023 --> 01:37:03.043
actually we don't store one two three four we

01:37:03.743 --> 01:37:07.783
do like cosine one cosine two cosine three cosine
four or something which is

01:37:09.123 --> 01:37:13.583
yeah a bit unusual but that's the thing again we
tried it, it works you know here's

01:37:14.403 --> 01:37:16.403
one more in your bag of tricks to

01:37:16.583 --> 01:37:20.995
use.The root mean square norm is very

01:37:21.095 --> 01:37:23.116
straightforward. It basically says,

01:37:25.055 --> 01:37:27.095
like we have done a lot of normalization, right?

01:37:27.095 --> 01:37:31.756
We did the normal row normalization, which is
divide each cell by

01:37:32.275 --> 01:37:34.975
some of all cells in the row. We did soft max
which is, you know,

01:37:35.955 --> 01:37:40.155
do the same thing but E to the power, you know, E
to the power cell divided by some of E

01:37:40.155 --> 01:37:41.896
to the power all cells in the row.

01:37:43.095 --> 01:37:46.255
So RMS is yet another... RMS is a divide

01:37:47.495 --> 01:37:51.835
each cell by sum of squares of all the cells in
the row.

01:37:51.895 --> 01:37:54.056
This is just yet another way of normalizing.

01:37:55.715 --> 01:37:57.695
Yeah, we do a lot of normalizing

01:37:58.615 --> 01:38:02.055
partly because of this, you know, vanishing,
exploding gradient problem.

01:38:02.055 --> 01:38:06.575
Like, we don't want gradients to ever go to zero
or infinity, therefore we need to keep

01:38:06.575 --> 01:38:08.015
normalizing stuff all the time.

01:38:11.855 --> 01:38:15.076
Yeah, and this will ensure our values always stay
between zero and one.

01:38:16.275 --> 01:38:18.475
Yeah, then there is this thing called residual
layer.

01:38:18.476 --> 01:38:20.595
Again, maybe I will go back to the big picture.

01:38:21.495 --> 01:38:25.715
The big picture is, yeah, we tokenized, we put the
positions, we put our images into

01:38:25.755 --> 01:38:27.455
something that's not images. So from,

01:38:28.435 --> 01:38:31.175
once you have reached here you can forget the fact
that they're images.

01:38:31.375 --> 01:38:32.095
We just now have

01:38:32.895 --> 01:38:36.075
numbers, you know. We have all these vectors
coming in, that's it.

01:38:36.575 --> 01:38:38.075
So we did, you know, normalizing,

01:38:38.775 --> 01:38:43.455
we did our, you know, projection into X, we made
into QKV and that

01:38:43.535 --> 01:38:45.655
QKV we made back into an output.

01:38:46.275 --> 01:38:47.835
We got some output at the end.

01:38:49.075 --> 01:38:50.995
Uh, this projection is straightforward, we just...

01:38:51.415 --> 01:38:52.955
Yeah, this slide I don't think we need to explain,
it's

01:38:53.915 --> 01:38:55.635
just whatever output we got from here,

01:38:56.875 --> 01:39:00.055
here meaning our soft max times V, we got some
output.

01:39:00.335 --> 01:39:03.975
We got a Y, we multiplied that by, you know, yet
another ma- matrix and we got the

01:39:03.995 --> 01:39:08.075
output. So from here we will multiply one more
weight matrix, we will get the output.

01:39:08.075 --> 01:39:10.035
So here is our multi-headed attention done.

01:39:10.995 --> 01:39:12.575
Now we do a thing called add residual.

01:39:12.575 --> 01:39:17.195
Now add residual basically means imagine from here
we had X, we did multi-headed

01:39:17.215 --> 01:39:21.095
attention, we got Y. Now instead of taking Y, we
will take Y plus X.

01:39:22.515 --> 01:39:25.295
Right, we will take our input, we will add it to
the output and we will keep that.

01:39:25.295 --> 01:39:28.535
And it is a little bit of a weird trick but it
works, we do this in machine learning all

01:39:28.575 --> 01:39:28.995
the time.

01:39:30.035 --> 01:39:32.775
Again just, it works therefore we do it.

01:39:32.775 --> 01:39:33.215
Uh,

01:39:38.775 --> 01:39:39.575
yeah, so

01:39:40.335 --> 01:39:42.695
Y equals FX was done in the previous step.

01:39:43.275 --> 01:39:46.235
So now we will take FX plus X and we will keep
this.

01:39:47.835 --> 01:39:50.335
And why? Our intuition is basically,

01:39:51.235 --> 01:39:55.275
remember we are going to be a- doing this entire
thing on like 80 layers, we are going to

01:39:55.315 --> 01:39:58.235
do this entire 80 times with 80 sets of eight
matrices.

01:39:59.295 --> 01:40:00.795
So if you had just done, you know,

01:40:02.215 --> 01:40:06.515
F of F of F of F of F of F of F of X like 80 times
and then you would do, run

01:40:07.035 --> 01:40:08.995
gradient descent on this to find you know the

01:40:10.275 --> 01:40:12.695
best weight matrices for each of those Fs.

01:40:12.815 --> 01:40:15.055
Like these are all 80 different Fs by the way,
like the

01:40:17.215 --> 01:40:19.915
weight matrices on, inside each of these Fs are
different.

01:40:20.915 --> 01:40:23.555
So if you had to do this you are more like
again...

01:40:23.595 --> 01:40:27.555
Remember sequential is bad for exploding and
vanishing gradients.

01:40:28.335 --> 01:40:29.775
That is a heuristic.

01:40:31.415 --> 01:40:31.915
Uh,

01:40:33.335 --> 01:40:37.895
therefore we are more likely to get exploding or
vanishing gradients however if

01:40:37.935 --> 01:40:38.895
sometimes we

01:40:40.375 --> 01:40:40.715
do,

01:40:41.435 --> 01:40:41.695
like,

01:40:42.315 --> 01:40:47.015
we did F of X plus X, F of this thing plus 3000
thing, F of this thing plus 3000 thing, F

01:40:47.035 --> 01:40:48.375
of this thing plus 3000 thing,

01:40:48.975 --> 01:40:50.135
and then now we...

01:40:50.815 --> 01:40:53.375
And each of these Fs have their own weight
matrices.

01:40:53.395 --> 01:40:56.115
Now we run unigradian descent across this entire
thing

01:40:57.295 --> 01:40:58.535
then, uh,

01:41:03.815 --> 01:41:04.095
yeah,

01:41:06.115 --> 01:41:08.575
then we are more likely to get, uh,

01:41:11.855 --> 01:41:13.715
good, uh, weight matrices

01:41:14.935 --> 01:41:18.575
at the end. Like we are going to get gradients
that are not going to zero or infinity, we

01:41:18.575 --> 01:41:19.575
are going to get...

01:41:23.955 --> 01:41:27.975
Yeah, I guess you got what I said. Okay, then
there is thing called mixture of experts

01:41:27.975 --> 01:41:32.815
which I am not describing now. It's kind of a
recent trick like two years ago nobody did

01:41:32.855 --> 01:41:34.575
this, now everyone is doing it.

01:41:35.475 --> 01:41:39.195
And because it's a recent trick also there's a
possibility it will not get used in the

01:41:39.235 --> 01:41:40.175
future, like

01:41:40.815 --> 01:41:44.295
maybe it will and maybe it will not, it's just I
don't know how to predict that.

01:41:45.095 --> 01:41:48.215
But yeah this is a common trick that has come up
in the last couple of years.

01:41:49.835 --> 01:41:53.335
Uh, feed forward is pretty straightforward, you
can search what feed forward is.

01:41:53.335 --> 01:41:55.655
Feed forward typically nowadays is just,

01:41:56.335 --> 01:41:59.095
just fully connected network in our two layer.
Remember this thing?

01:42:00.315 --> 01:42:04.515
Yeah, so this entire thing is like one tiny tiny
piece in our entire big, giant big

01:42:04.555 --> 01:42:05.235
transformer.

01:42:07.015 --> 01:42:11.755
Yeah, so we put here, you know, we put two weight
matrices, our W1 and W2 they go in here

01:42:12.515 --> 01:42:13.895
then we add a residual again.

01:42:15.075 --> 01:42:18.415
We do this entire thing 80 times, we do one last
projection step.

01:42:18.415 --> 01:42:22.255
Projection just means we multiply it with yet
another matrix and we do a ReLU.

01:42:23.195 --> 01:42:27.095
Oh yeah, I think I forgot to mention we do a lot
of ReLUs like,

01:42:30.235 --> 01:42:34.255
like when we do residual we will probably do a
ReLU, when we do projection we will do a

01:42:34.255 --> 01:42:37.035
ReLU, when we do this projection we will do a ReLU
like,

01:42:38.375 --> 01:42:43.315
like a ReLU just happens so often it's not kind of
worth even mentioning like ReLU is

01:42:43.315 --> 01:42:46.155
that common. We just do ReLUs everywhere.

01:42:51.195 --> 01:42:51.435
Yeah,

01:42:53.175 --> 01:42:58.095
so we have done this lots and lots of computation
and then we are going to do

01:42:58.975 --> 01:43:00.275
gradient descent,

01:43:04.255 --> 01:43:08.795
i- and we are going to find our weight matrices. I
think that's basically the overview.

01:43:10.995 --> 01:43:15.935
There's one final topic that I need to cover or I
want to cover which is the

01:43:15.935 --> 01:43:17.735
hyper-parameters which is like

01:43:18.475 --> 01:43:20.575
how big should this entire thing be, you know.

01:43:21.195 --> 01:43:26.179
How many layers do we pick? How big our weight
matrices do we want to pick?And these

01:43:26.199 --> 01:43:28.039
are just typical values in the past.

01:43:29.900 --> 01:43:33.459
So number of layers is usually between 32 and 128.

01:43:34.199 --> 01:43:36.619
How big are weight matrices is, you know, so

01:43:37.239 --> 01:43:42.020
GPT-2 was the first good transformer that was
trained back in 2019.

01:43:43.579 --> 01:43:45.819
It had 1.5 billion

01:43:46.539 --> 01:43:48.559
parameters. So parameters is just

01:43:49.479 --> 01:43:51.639
how many numbers are in your weight matrix.

01:43:51.760 --> 01:43:52.079
So

01:43:52.920 --> 01:43:54.619
let's go way, way, way back.

01:43:56.139 --> 01:43:56.779
So this

01:43:57.620 --> 01:44:00.200
two-layer network we trained, this has,

01:44:01.239 --> 01:44:01.679
uh,

01:44:02.819 --> 01:44:04.799
so the total number of parameters in this

01:44:06.399 --> 01:44:10.559
network are 784 times 800, that's W1, plus,

01:44:11.280 --> 01:44:13.059
you know, 800 times

01:44:13.920 --> 01:44:17.639
10. Yeah, so we, this is around 600,000
parameters,

01:44:19.299 --> 01:44:24.059
635,000 parameter. So this is a 635,000 parameter
network.

01:44:24.119 --> 01:44:29.019
So GPT-2 is a 1.5 billion parameter network.

01:44:30.279 --> 01:44:34.719
There are going to be 1.5 billion numbers inside
the network,

01:44:35.399 --> 01:44:39.019
and we are going to do gradient descent to find,
you know, gradients for each of these

01:44:39.039 --> 01:44:40.639
1.5 billion numbers,

01:44:41.459 --> 01:44:44.739
and update them a little bit, and update them a
little bit more, update them a little bit

01:44:44.779 --> 01:44:46.159
more. We'll keep doing that.

01:44:47.939 --> 01:44:48.379
So,

01:44:49.899 --> 01:44:50.159
yeah,

01:44:51.379 --> 01:44:56.259
GPT-2 is 1.5 billion parameters. GPT-3 we made a
big jump to 175

01:44:56.279 --> 01:44:57.499
billion parameters.

01:44:58.159 --> 01:45:00.919
GPT-4 is around two trillion parameters.

01:45:00.939 --> 01:45:02.459
You know, our latest models are now,

01:45:03.079 --> 01:45:05.679
they've crossed 10 trillion parameters.

01:45:08.139 --> 01:45:08.399
Yeah.

01:45:09.479 --> 01:45:12.639
Uh, this is a Mixture of Experts thing. I will
skip that for now.

01:45:14.699 --> 01:45:18.319
So if you're using O3, it's most likely based on
GPT-4.

01:45:18.959 --> 01:45:20.999
So O3 is, you know, O3 is this thing.

01:45:20.999 --> 01:45:25.719
How

01:45:25.979 --> 01:45:27.719
to bake chocolate

01:45:28.959 --> 01:45:29.339
cake,

01:45:31.699 --> 01:45:32.799
no egg.

01:45:44.999 --> 01:45:45.879
Yeah. So

01:45:47.359 --> 01:45:50.759
this is, uh, O3, this is based on GPT-4.

01:45:52.119 --> 01:45:54.379
I mean, it's not just straightforward GPT-4.

01:45:54.379 --> 01:45:55.019
We have done some

01:45:55.759 --> 01:45:58.979
shenanigans on top at the end with somebody else.

01:45:58.979 --> 01:46:00.639
We'll have to talk about in a different lecture.

01:46:00.839 --> 01:46:01.099
But

01:46:01.799 --> 01:46:03.079
yeah, it's based on GPT-4.

01:46:04.899 --> 01:46:09.179
It has two trillion parameters which we found
using this gradient descent.

01:46:09.259 --> 01:46:09.559
Uh,

01:46:11.739 --> 01:46:14.379
each parameter is typically stored as four bytes.

01:46:15.579 --> 01:46:19.279
So when we say two trillion parameters, what we
actually mean is eight terabytes.

01:46:20.679 --> 01:46:25.419
You know, each byte, each parameter is four bytes
so 2T into 4 is 8T.

01:46:26.679 --> 01:46:28.219
8 terabytes, uh,

01:46:29.679 --> 01:46:33.479
nowadays there are also tricks being used, like
mixed precision training was used in

01:46:33.519 --> 01:46:37.259
DeepSeek, which is instead of four-byte weights,
we will use, you know, two-byte weights

01:46:37.379 --> 01:46:40.139
somewhere and somewhere we will use four bytes.

01:46:42.079 --> 01:46:43.159
Like for some parts of the

01:46:43.859 --> 01:46:47.619
network we will do gradients on four-byte weights
and some parts of the networks we will

01:46:47.659 --> 01:46:51.659
do gradient descent on two-byte weights, and
apparently that also works.

01:46:53.359 --> 01:46:57.779
So the basic thing is if you do gradient descent
on smaller, like use

01:46:58.359 --> 01:46:59.799
less bytes for the weights,

01:47:00.559 --> 01:47:04.499
you will, you can calculate it faster but also
you'll get less accurate answers.

01:47:05.699 --> 01:47:09.359
So for training, I mean typically people still use
the entire full four bytes.

01:47:09.759 --> 01:47:12.619
For inference, inference is, you know, once you've
trained the model.

01:47:13.119 --> 01:47:17.159
Train means once you've got really good weight
matrices, now you just want to do this on

01:47:17.159 --> 01:47:17.999
some new answer.

01:47:19.719 --> 01:47:22.939
Like you have trained this on our existing data,
and on some new data you need to try it

01:47:22.979 --> 01:47:27.579
out. When you're trying it out on new data, then
people do, you know, two bytes, they

01:47:27.579 --> 01:47:29.719
take one byte weight, you know, four-bit weights.

01:47:29.719 --> 01:47:33.739
We have, even have now like literally less than
two-bit weights.

01:47:33.759 --> 01:47:34.039
Like

01:47:34.859 --> 01:47:37.859
there's this thing called ternary bits which is
like, uh,

01:47:39.819 --> 01:47:40.859
you round a number,

01:47:42.799 --> 01:47:45.439
uh, I forget it, I'm not explaining ternary bits
right now.

01:47:45.439 --> 01:47:48.539
But yeah, we can r- make each weight really,
really small.

01:47:49.719 --> 01:47:54.219
We can make it even like as small as two bits per
weight or even less than two bits, so

01:47:54.219 --> 01:47:55.719
it's 1.58 bits.

01:47:58.759 --> 01:47:59.299
Uh,

01:48:00.379 --> 01:48:01.799
so this is our model size.

01:48:02.639 --> 01:48:04.579
Yeah, and how did we pick these numbers?

01:48:04.579 --> 01:48:09.099
Like who, why did we take the decision of GPT-2
should be this big, GPT-3 should be this

01:48:09.119 --> 01:48:09.479
big?

01:48:10.259 --> 01:48:12.919
It depends on how much data and how much compute
we have.

01:48:13.519 --> 01:48:14.859
So data is our input,

01:48:15.799 --> 01:48:19.559
like we will train this on some, like initially,
you know, we started with 60,000 images.

01:48:19.659 --> 01:48:23.439
So here for text we'll start, you know, some few
million sentences or some trillion

01:48:23.459 --> 01:48:24.219
sentences.

01:48:25.859 --> 01:48:29.499
So how much of that do we have and how much
compute do we have, which is just how many

01:48:29.559 --> 01:48:30.519
GPUs do we have?

01:48:32.299 --> 01:48:37.179
And you can calculate the requirements using this
law called the Chinchilla Scaling

01:48:37.219 --> 01:48:37.499
Law.

01:48:38.559 --> 01:48:42.679
Uh, I'm not explaining Chinchilla Scaling Law
right now, but you can read it on

01:48:42.699 --> 01:48:44.579
Wikipedia, it's pretty straightforward.

01:48:46.399 --> 01:48:46.759
Like,

01:48:48.059 --> 01:48:52.579
C is how much computation you need, n is the
number of parameters,

01:48:53.199 --> 01:48:57.219
b is the size of your dataset, and there's like a
straightforward relationship between

01:48:57.219 --> 01:48:57.779
all of them.

01:48:59.339 --> 01:49:02.919
So given some amount of compute, it just tells
you, look here's how much.

01:49:04.179 --> 01:49:04.499
Like,

01:49:05.919 --> 01:49:09.339
given some amount of compute, given some amount of
data, here's the loss you're going to

01:49:09.379 --> 01:49:10.099
get at the end.

01:49:10.819 --> 01:49:14.539
And using those you can reverse calculate, okay,
well if you want to optimally use that

01:49:14.559 --> 01:49:17.459
compute, you know, how much compute and how much
data should be used at once.

01:49:18.479 --> 01:49:23.059
You will calculate that. And yeah, an important
thing is that usually compute is the

01:49:23.079 --> 01:49:28.027
bottleneck, not data.Like usually we have

01:49:28.207 --> 01:49:29.547
more data than we actually need

01:49:30.347 --> 01:49:32.587
and less GPUs than we actually need

01:49:33.307 --> 01:49:35.147
(laughs) . So more GPUs, bigger model.

01:49:35.988 --> 01:49:39.607
So you know, this first train on a smaller data
center, maybe for an hour.

01:49:39.647 --> 01:49:44.047
This maybe was trained in hundreds of hours. Not
hundreds, sorry, thousands of hours.

01:49:44.967 --> 01:49:48.588
Here, by the time we are reaching here, we are
running into you know, millions of GPU

01:49:48.588 --> 01:49:49.127
hours.

01:49:52.387 --> 01:49:52.727
So

01:49:54.147 --> 01:49:58.987
yeah, it's just to train a bigger model, in short,
you need more compute and more data,

01:49:59.188 --> 01:49:59.528
but

01:50:00.348 --> 01:50:01.988
we have data and we don't have compute.

01:50:01.988 --> 01:50:05.347
So the more compute you have, the bigger model you
will get to train, basically.

01:50:09.147 --> 01:50:09.627
And

01:50:10.688 --> 01:50:12.547
this is just if you want to train it well.

01:50:12.627 --> 01:50:12.967
Like

01:50:13.807 --> 01:50:18.447
remember when we were here, when we defined, we
never said how many times to do a

01:50:18.447 --> 01:50:19.627
gradient update. Like

01:50:20.427 --> 01:50:23.848
we could just update the weight matrices, you
know, 1000 times and be like, look here's

01:50:23.848 --> 01:50:27.347
our performance. It's just this performance won't
be that good.

01:50:27.847 --> 01:50:30.607
If you want the performance to be really good, you
will have to update it, you know,

01:50:30.607 --> 01:50:32.147
millions and millions of times. And

01:50:33.527 --> 01:50:36.407
then there's a question of, you know, how many
GPUs do you have?

01:50:36.407 --> 01:50:39.187
How many years can you run these GPUs for and so
on.

01:50:41.147 --> 01:50:45.867
And yeah, Epoch AI is this nonprofit...

01:50:46.707 --> 01:50:48.827
Are they nonprofit or for profit? I'm not sure.

01:50:48.907 --> 01:50:49.247
Uh,

01:50:50.827 --> 01:50:53.967
let's say it's once in 2028, we'll run out of
data.

01:50:54.047 --> 01:50:54.327
Like

01:50:54.947 --> 01:50:58.127
literally every... all the data on the internet,
you know,

01:50:59.247 --> 01:51:02.867
will have been fed into a model and there's no
more data out there

01:51:04.107 --> 01:51:05.467
and then we'll run out of data.

01:51:06.147 --> 01:51:08.427
And there are some tricks that might bypass that
too.

01:51:09.127 --> 01:51:12.527
I- I- My personal guess is some of those tricks
will work, but,

01:51:13.587 --> 01:51:16.667
uh, yeah, we are going to run out of data in 2028.

01:51:16.707 --> 01:51:17.027
And

01:51:17.807 --> 01:51:20.667
compute, we're, we're never going to run out of
compute.

01:51:24.047 --> 01:51:27.987
That's just a question of, you know, like what
fraction of the world economy do you want

01:51:28.007 --> 01:51:30.187
to invest into, you know, GPUs? And

01:51:31.627 --> 01:51:34.967
GPUs keep getting faster. So you know, every 18
months the-

01:51:35.767 --> 01:51:36.467
you can get

01:51:37.347 --> 01:51:38.627
twice the number of

01:51:39.607 --> 01:51:41.647
matrix multiplications per dollar.

01:51:42.947 --> 01:51:47.067
So the number of dollars being spent is going up
and how many matrix multiplications you

01:51:47.087 --> 01:51:48.827
can get per dollar is also going up.

01:51:49.767 --> 01:51:53.987
And that seems the trajectory at least for, you
know, the next five years, ten years.

01:51:54.107 --> 01:51:54.347
So

01:51:55.087 --> 01:51:57.087
yeah, you can see more on the forecasts

01:51:58.807 --> 01:52:01.467
for how much data and how much compute we will use

01:52:02.687 --> 01:52:06.907
in the next, you know, five years here. Like Epoch
has forecast.

01:52:07.927 --> 01:52:09.767
Ah, my net is fucking slow.

01:52:12.867 --> 01:52:17.467
Mm, yeah, so our compute spending is- we're doing
5x every year.

01:52:17.467 --> 01:52:19.367
So it's like exponentially going up.

01:52:20.487 --> 01:52:22.347
Data runs out in 2028.

01:52:25.687 --> 01:52:27.947
We are spending three times as many dollars

01:52:28.907 --> 01:52:30.147
on training every year.

01:52:31.107 --> 01:52:33.267
So it's first we start with, you know, like

01:52:33.927 --> 01:52:38.447
$10 million, then $30 million, $100 million, $30
million.

01:52:38.647 --> 01:52:40.807
Sorry, $300 million, a billion dollars.

01:52:41.467 --> 01:52:44.667
Right now I think Grok is trained on seven
billion, right? Yeah.

01:52:45.527 --> 01:52:46.307
So Grok,

01:52:47.647 --> 01:52:51.387
trained by Elon Musk, required a data center that
cost $7

01:52:51.507 --> 01:52:52.667
billion.

01:52:54.527 --> 01:52:57.007
Half a billion dollars was spent on Grok itself.

01:52:58.547 --> 01:53:02.727
Uh, but just to acquire that ha-hardware, like
once you have acquired it, what else are

01:53:02.747 --> 01:53:06.107
you going to use it for? It- you're going to just
use it either for training or for

01:53:06.147 --> 01:53:06.947
inference. So

01:53:08.627 --> 01:53:13.107
this is the cost to acquire all that
infrastructure, including the GPUs, including the

01:53:13.147 --> 01:53:15.147
networking, the cooling and so on.

01:53:17.807 --> 01:53:18.427
Uh,

01:53:20.087 --> 01:53:20.487
yeah.

01:53:23.687 --> 01:53:27.107
I think that's pretty much it for my presentation.

01:53:28.867 --> 01:53:29.387
Uh,

01:53:31.507 --> 01:53:34.967
yeah here are some rules of thumb. If you don't
want to actually sit and solve the

01:53:34.987 --> 01:53:38.507
Chinchilla scaling thing directly, here are some
rules of thumb that say

01:53:39.767 --> 01:53:42.487
for these many parameters, here's how much data
you need.

01:53:42.507 --> 01:53:46.687
And for these many parameters and this much data,
here's how much compute you need.

01:53:48.287 --> 01:53:50.927
So yeah, I think that's basically it for my
presentation.