Optimizer as the name suggest to optimize something.It is used to optimize and help to attain models efficiency and accuracy of prediction.It is the component in deep learning which closely works with loss function to improve model.Optimizer uses learning rate to increase or decrease weights and bias for model improvement.
How optimizer works?
According to universal approximation theorem anything in this world can be expressed in terms of function . To achieve that we use deep learning.In deep learning till now we have discussed about data, architecture and loss function.As we have seen in the previous post that loss function provide us direction and gradient to improve our model.If you haven't read the post kindly click here.After getting gradients and direction from loss function from differentiation, then we take step to change weights and bias of our model.
Firstly we will defined terms which is going to be used in following post to make you understand optimizer.
GRADIENT:-It is numerical number which is created through differentiation of loss function.
DIRECTION:-The nature of that numerical number which is created through differentiation is the direction in which we have move.Let say we get differentiation as -7.0. (-)this sign indicates us that in which direction we have to go.
STEPS:-How much steps you have taken to reach local minimum of your model.
LOCAL MINIMUM:-It is highest efficiency our model can achieve through this process.