Why Neural Net weights need random initialization?

Initializing parameters not to zero but randomly turns out to be very important for training Neural Network. Initializing to zero would not work for gradient decent because:

Consider a Neural Network with multiple hidden layers with two neurons in the first layer, then weights being same for both neurons, both the hidden units are calculating exactly the same function. Also, dz is same for both while back propagation. As a result, outgoing weights also seem to be equal i.e weight matrix of next layer also come out to be zero. So after every iteration we will see that both the hidden units are computing exactly the same function and are symmetric. Hence, there is no point having more than one hidden unit with weights initialized to zero. Hence, we need to initialize the parameters randomly.

Also, weights are initialized with very small values because otherwise the value of z will be high and hence the slope of gradient decent or slope will be quite low leading to slower learning in case of sigmoid or tanh activations. Also b is initialized to zero because it doesn’t have symmetry problem as long as z is randomly initialized.

Related posts:

Lets rewind now to say ten-eleven days before leaving camp.while i was enjoying not knowing anyone and not wanting to,then boom it all happened. And i was here being admired by an anonymous…

You and your partner might have been together for a while, and though the two of you like spending time together, there might be one things that you don’t like that your partner does. Your partner…