Improving Efficiency and Accuracy for Training and Inference of Hardware-aware Machine Learning Systems

Deep Neural Networks (DNNs) have been adopted in many systems because of their higher classification accuracy. While progress in achieving large scale, highly accurate DNNs has been made, however, the huge time and energy requirement for both inference and training pose a
challenge to any DNN implementation. The large DNN size causes significant energy and area due to massive memory accesses and computations. In this thesis, we propose a new DNN architecture, LightNN, which replaces the multiplications to one shift or a constrained number of shifts and adds. Thus, LightNN inference uses hardware-efficient shift-based operations, and needs fewer memory
accesses due to the fewer bits to represent each weight. Our experiment using image datasets show that LightNNs can achieve much higher hardware efficiency with a slight accuracy loss. LightNNs constrain all the weights of DNNs to be a limited combination (denoted as k 2 f1;2g)
of powers of 2, and by varying the k one can obtain a group of LightNNs with different levels of accuracy and energy cost. To provide even more design flexibility, the k for each convolutional filter can be optimally chosen instead of being fixed for every filter. In this thesis, we formulate the
selection of k to be differentiable, and describe model training for determining k-based weights on a per-filter basis. Over 46 FPGA-design experiments involving eight configurations and four data sets reveal that lightweight neural networks with a flexible k value (dubbed FLightNNs) fully utilize the hardware resources on Field Programmable Gate Arrays (FPGAs), our experimental results show that FLightNNs can achieve 2 speedup when compared to lightweight NNs with k = 2, with only 0:1% accuracy degradation. Compared to a 4-bit fixed-point quantization, FLightNNs achieve higher accuracy and up to 2 inference speedup, due to their lightweight shift operations. In addition, our experiments also demonstrate that FLightNNs can achieve higher computational energy efficiency for ASIC implementation. Binarized Neural Networks (BNNs) quantize the weights and activations to 1 bit, which has lower
precision than LightNNs and FLightNNs, and therefore can significantly reduce the inference latency and energy consumption in resource-constrained devices due to their pure-logical computation and fewer memory accesses. However, training BNNs is difficult since the activation flow encounters degeneration, saturation, and gradient mismatch problems. Prior work alleviates these issues by increasing activation bits and adding floating-point scaling factors, thereby sacrificing BNN’s energy efficiency. In this thesis, we propose to use distribution loss to explicitly regularize the activation flow, and develop a framework to systematically formulate the loss. Our experiments show that the distribution loss can consistently improve the accuracy of BNNs without losing their energy benefits. Moreover, equipped with the proposed regularization, BNN training is shown to be robust to the selection of hyper-parameters including optimizer and learning rate.