This is a very simple implementation of Deep Learning Module which can be used to train neural networks with small architectures.
- Implemented reverse mode Automatic Differentiation form scratch
- Implemented various optimizers like Momentum based Nesterov Accelerated Gradient Descent
- Optimized the framework by adding in the support for vectors as the atomic unit of parameters.
- Incorporating various other Layers like Convolution, BatchNorm, Attention, etc.
- Introduce other state-of-the-art optimizers like Adam, Adagrad.
- Introduce other loss functions like binary cross-entropy.
Use the framework after doing an editable install of the project as :
pip install -e .
from nnet.nn.val.nn import MLP
from nnet.optimizers.optimizers import SGD
from nnet.losses.losses import mse_loss
import random
random.seed(42)
if __name__=="__main__":
# hyper params initialization
epochs = 20
lr = 0.01
# data initialization
x = [
[3.0, 4.0, 5.0],
[-5.0, -2.0, 8.0],
[-2.0, 2.5, -4.5],
[2.0, -1.5, -4.0]
]
y_true = [1, -1, -1, 1]
# model initialization
model = MLP(3, [4, 4, 1])
optim = SGD(model.parameters(), lr, 0.9, nestrov=True)
# training loop
for epoch in range(epochs):
# forward pass / inference
y_pred = [model(inp) for inp in x]
loss_val = mse_loss(y_pred, y_true)
print(epoch+1, loss_val.value)
# zero grad
optim.zero_grad()
# backward pass
loss_val.backward()
# update params
optim.step()Motivation: Andrej Karpathy's repo