DLW-NAS: Differentiable Light-Weight Neural Architecture Search

In recent years, the method of automatically constructing convolutional neural networks based on neural architecture search has attracted wide attention, and greatly reduces the manual intervention and the cost of manual design of neural networks. However, most neural architecture search methods focus on the performance of the model, but ignore the complexity of the model, which makes it difficult to deploy this method on devices with limited resources. In this paper, a novel differentiable light-weight architecture search method named DLW-NAS is proposed, which aims to search convolutional neural networks (CNNs) with remarkable performance as well as a small amount of parameters and floating point operations (FLOPs). Concretely, in order to limit the parameters and FLOPs from the source of neural architecture search (NAS), we build a light-weight search space containing effective light-weight operations. Moreover, we design a differentiable NAS strategy with computation complexity constraints. In addition, we propose a neural architecture optimization method, which makes the topology of the searched architecture sparse. The experimental results show that DLW-NAS achieves 2.73% error rate on CIFAR-10, which is comparable to the state-of-the-art (SOTA) methods. However, it only needs 2.3M parameters and 334M FLOPs, which reduces that of the SOTA DARTS by 30% and 36% in parameters and FLOPs, respectively. The searched model on CIFAR-100 uses only 2.47M parameters and 376M FLOPs with an error rate of only 17.12%. On ImageNet, compared with the SOTA MobileNet, DLW-NAS achieves 3.3% better top-1 accuracy with much fewer parameters and FLOPs.