[MUSIC] The second question on stochastic gradient

is how do you pick the step size eta. And this is a significant issue
just like it is with gradient. Both of them, it's kind of annoying, it's a pain to
figure out to
pick that coefficient eta. But it turns out that because of the
oscillations to the stochastic gradient, picking eta can be even more annoying,
much more annoying. So, if we go back to our data set and
when we've been using it, I've shown you this blue curve many times. This was the
best eta that I could find,
the best step size. Now, if I were to pick smaller step sizes. So, smaller ETAs, it
will behave kind of
like stocha elect regular gradient. It will be slower to converge and
you see less oscillations. But, it will eventually get there,
but I mean, much slower. So, we worry about that a bit. On the other hand,
instead of using the best step size, we try to use a larger step
size because we thought, what could make more
progress more quickly? You'll see this crazy oscillations and
the oscillations are much worse than you observe with gradient,
and I showed you gradient. So, you have to be really
careful to pick a too large things can behave
extremely erratically. And in fact, if you picked step size very, very large,
you can end up with behaviors like this. So, this black line here
was an eta that was way too large, that's a technical
term here that we'd like to use. And in this case, the solution is not even close
to anything we got even with etas of oscillations that we have
showed you in the previous slide. It's a huge gap and so, etas too large leads to
really bad
behavior in stochastic gradient. The rule of thumb that we described for
gradient, for picking eta is basically the same as the one for
picking step size for stochastic gradient. The same as for gradient, but
unfortunately it
requires much more trial and error. So, it's even more annoying, so you might
spent a lot of time in the trial and error even though it's a hundred times faster
than converge, it's possible to spend a hundred times more effort trying to find
the right step size, just be prepared. But, we just try several values
exponentially spaced from each other. And try to find somewhere between
an eta that's too small and an eta that is too big, and
then find one that's just right. And, I mentioned this in the gradient
section, but for stochastic gradients, even more important, for those who end
up exploring this further, there's an advanced step where you would make
the step size decrease over iterations. And so, for example, you might have an eta
that depends on what iteration you're in and often you set it to
something like some constant here eta zero divided by
the iteration number t. Iteration number, and
this approach tends to reduce noise and make things behave quite a bit better.
[MUSIC]