This is project is the implementation of the DeepShift: Towards Multiplication-Less Neural Networks paper, that aims to replace multiplications in a neural networks with bitwise shift (and sign change).

This research project was done at Huawei Technologies.

• Overview
• Important Notes
• Getting Started
• Running the Bitwise Shift CUDA & CPU Kernels
• Results
  • MNIST
    • Train from Scratch
    • Train from Pre-Trained
  • CIFAR10
    • Train from Scratch
    • Train from Pre-Trained
    • Using Fewer Bits
  • ImageNet
    • Train from Scratch
    • Train from Pre-Trained
    • Using Fewer Bits
  • Binary Files of Trained Models
• Code WalkThrough

The main idea of DeepShift is to test the ability to train and infer using bitwise shifts.

We present 2 approaches:

• DeepShift-Q: the parameters are floating point weights just like regular networks, but the weights are rounded to powers of 2 during the forward and backward passes
• DeepShift-PS: the parameters are signs and shift values

• To train from scratch, the learning rate --lr option should be set to 0.01. To train from pre-trained model, it should be set to 0.001 and lr-step-size should be set to 5
• To use DeepShift-PS, the --optimizer must be set to radam in order to obtain good results.

1. Clone the repo:

git clone https://github.com/mostafaelhoushi/DeepShift.git

2. Change directory

cd DeepShift

3. Create virtual environment:

virtualenv venv --prompt="(DeepShift) " --python=/usr/bin/python3.6

4. (Needs to be done every time you run code) Source the environment:

source venv/bin/activate

5. Install required packages and build the spfpm package for fixed point

pip install -r requirements.txt

6. cd into pytorch directroy:

cd pytorch

7. Now you can run the different scripts with different options, e.g., a) Train a DeepShift simple fully-connected model on the MNIST dataset, using the PS apprach:

   python mnist.py --shift-depth 3 --shift-type PS --optimizer radam
   

   b) Train a DeepShift simple convolutional model on the MNIST dataset, using the Q approach:

   python mnist.py --type conv --shift-depth 3 --shift-type Q 
   

   c) Train a DeepShift ResNet20 on the CIFAR10 dataset from scratch:

   python cifar10.py --arch resnet20 --pretrained False --shift-depth 1000 --shift-type Q 
   

   d) Train a DeepShift ResNet18 model on the Imagenet dataset using converted pretrained weights for 5 epochs with learning rate 0.001:

   python imagenet.py <path to imagenet dataset> --arch resnet18 --pretrained True --shift-depth 1000 --shift-type Q --epochs 5 --lr 0.001
   

   e) Train a DeepShift ResNet18 model on the Imagenet dataset from scratch with an initial learning rate of 0.01:

   python imagenet.py <path to imagenet dataset> --arch resnet18 --pretrained False --shift-depth 1000 --shift-type PS --optimizer radam --lr 0.01
   

1. cd into DeepShift/pytorch directroy:

cd DeepShift/pytorch

2. Run the installation script to install our CPU and CUDA kernels that perform matrix multiplication and convolution using bit-wise shifts:

sh install_kernels.sh

3. Now you can run a model with acutal bit-wise shift kernels in CUDA using the --use-kernel True option. Remember that the kernel only works for inference not training, so you need to add the -e option as well:

python imagenet.py --arch resnet18 -e --shift-depth 1000 --pretrained True --use-kernel True

4. To compare the latency with a naive regular convolution kernel that does not include cuDNN's other optimizations:

python imagenet.py --arch resnet18 -e --pretrained True --use-kernel True

Accuracies shown are Top1 / Top5.

TBD

• pytorch: directory containing implementation, tests, and saved models using PyTorch
  • deepshift: directory containing the PyTorch models as well as the CUDA and CPU kernels of LinearShift and Conv2dShift ops
  • unoptimized: directory containing the PyTorch models as well as the CUDA and CPU kernels of the naive implementations of Linear and Conv2d ops
  • mnist.py: example script to train and infer on MNIST dataset using simple models in both their original forms and DeepShift version.
  • cifar10.py: example script to train and infer on CIFAR10 dataset using various models in both their original forms and DeepShift version.
  • imagenet.py: example script to train and infer on Imagenet dataset using various models in both their original forms and DeepShift version.
  • optim: directory containing definition of RAdam and Ranger optimizers. RAdam optimizer is crucial to get DeepShift-PS obtain the accuracies shown here