Maintained by Difan Deng and Marius Lindauer.
The following list considers papers related to neural architecture search. It is by no means complete. If you miss a paper on the list, please let us know.
Please note that although NAS methods steadily improve, the quality of empirical evaluations in this field are still lagging behind compared to other areas in machine learning, AI and optimization. We would therefore like to share some best practices for empirical evaluations of NAS methods, which we believe will facilitate sustained and measurable progress in the field. If you are interested in a teaser, please read our blog post or directly jump to our checklist.
2021 |
|
| 1018. | Chen, Wuyang; Gong, Xinyu; Wang, Zhangyang Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective Inproceedings ICLR 2021, 2021. @inproceedings{chen2021neural, title = {Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective}, author = {Wuyang Chen and Xinyu Gong and Zhangyang Wang}, url = {https://arxiv.org/abs/2102.11535}, year = {2021}, date = {2021-05-04}, booktitle = {ICLR 2021}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } |
| 1017. | Pan, Zheyi; Ke, Songyu; Yang, Xiaodu; Liang, Yuxuan; Yu, Yong; Zhang, Junbo; Zheng, Yu AutoSTG: Neural Architecture Search forPredictions of Spatio-Temporal Graphs Inproceedings WWW 2021, 2021. @inproceedings{PanWWW2021, title = {AutoSTG: Neural Architecture Search forPredictions of Spatio-Temporal Graphs}, author = {Zheyi Pan and Songyu Ke and Xiaodu Yang and Yuxuan Liang and Yong Yu and Junbo Zhang and Yu Zheng}, url = {http://panzheyi.cc/publication/pan2021autostg/paper.pdf}, year = {2021}, date = {2021-04-19}, booktitle = {WWW 2021}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } |
| 1016. | Liu, Lanlan; Zhang, Yuting; Deng, Jia; Soatto, Stefano Dynamically Grown Generative Adversarial Networks Inproceedings AAAI 2021, 2021. @inproceedings{LiuAAAI2021, title = {Dynamically Grown Generative Adversarial Networks}, author = {Lanlan Liu and Yuting Zhang and Jia Deng and Stefano Soatto}, url = {https://www.aaai.org/AAAI21Papers/AAAI-1376.LiuL.pdf}, year = {2021}, date = {2021-02-02}, booktitle = {AAAI 2021}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } |
| 1015. | Xu, Y; Xie, L; Dai, W; Zhang, X; Chen, X; Qi, G; Xiong, H; Tian, Q Partially-Connected Neural Architecture Search for Reduced Computational Redundancy Journal Article IEEE Transactions on Pattern Analysis & Machine Intelligence, (01), pp. 1-1, 2021, ISSN: 1939-3539. @article{9354953, title = {Partially-Connected Neural Architecture Search for Reduced Computational Redundancy}, author = {Y Xu and L Xie and W Dai and X Zhang and X Chen and G Qi and H Xiong and Q Tian}, url = {https://www.computer.org/csdl/journal/tp/5555/01/09354953/1rgCccYlOaQ}, doi = {10.1109/TPAMI.2021.3059510}, issn = {1939-3539}, year = {2021}, date = {2021-02-01}, journal = {IEEE Transactions on Pattern Analysis & Machine Intelligence}, number = {01}, pages = {1-1}, publisher = {IEEE Computer Society}, address = {Los Alamitos, CA, USA}, abstract = {Differentiable architecture search (DARTS) enables effective neural architecture search (NAS) using gradient descent, but suffers from high memory and computational costs. In this paper, we propose a novel approach, namely Partially-Connected DARTS (PC-DARTS), to achieve efficient and stable neural architecture search by reducing the channel and spatial redundancies of the super-network. In the channel level, partial channel connection is presented to randomly sample a small subset of channels for operation selection to accelerate the search process and suppress the over-fitting of the super-network. Side operation is introduced for bypassing (non-sampled) channels to guarantee the performance of searched architectures under extremely low sampling rates. In the spatial level, input features are down-sampled to eliminate spatial redundancy and enhance the efficiency of the mixed computation for operation selection. Furthermore, edge normalization is developed to maintain the consistency of edge selection based on channel sampling with the architectural parameters for edges. Experimental results demonstrate that the proposed approach achieves higher search speed and training stability than DARTS. PC-DARTS obtains a top-1 error rate of 2.55% on CIFAR-10 with 0.07 GPU-days for architecture search, and a state-of-the-art top-1 error rate of 24.1% on ImageNet (under the mobile setting) within 2.8 GPU-day.}, keywords = {}, pubstate = {published}, tppubtype = {article} } Differentiable architecture search (DARTS) enables effective neural architecture search (NAS) using gradient descent, but suffers from high memory and computational costs. In this paper, we propose a novel approach, namely Partially-Connected DARTS (PC-DARTS), to achieve efficient and stable neural architecture search by reducing the channel and spatial redundancies of the super-network. In the channel level, partial channel connection is presented to randomly sample a small subset of channels for operation selection to accelerate the search process and suppress the over-fitting of the super-network. Side operation is introduced for bypassing (non-sampled) channels to guarantee the performance of searched architectures under extremely low sampling rates. In the spatial level, input features are down-sampled to eliminate spatial redundancy and enhance the efficiency of the mixed computation for operation selection. Furthermore, edge normalization is developed to maintain the consistency of edge selection based on channel sampling with the architectural parameters for edges. Experimental results demonstrate that the proposed approach achieves higher search speed and training stability than DARTS. PC-DARTS obtains a top-1 error rate of 2.55% on CIFAR-10 with 0.07 GPU-days for architecture search, and a state-of-the-art top-1 error rate of 24.1% on ImageNet (under the mobile setting) within 2.8 GPU-day. |
| 1014. | Wu, Junru; Dai, Xiyang; Chen, Dongdong; Chen, Yinpeng; Liu, Mengchen; Yu, Ye; Wang, Zhangyang; Liu, Zicheng; Chen, Mei; Yuan, Lu Weak NAS Predictors Are All You Need Technical Report 2021. @techreport{wu2021weak, title = {Weak NAS Predictors Are All You Need}, author = {Junru Wu and Xiyang Dai and Dongdong Chen and Yinpeng Chen and Mengchen Liu and Ye Yu and Zhangyang Wang and Zicheng Liu and Mei Chen and Lu Yuan}, url = {https://arxiv.org/abs/2102.10490}, year = {2021}, date = {2021-01-01}, keywords = {}, pubstate = {published}, tppubtype = {techreport} } |
| 1013. | Guo, Yong; Zheng, Yin; Tan, Mingkui; Chen, Qi; Li, Zhipeng; Chen, Jian; Zhao, Peilin; Huang, Junzhou Towards Accurate and Compact Architectures via Neural Architecture Transformer Technical Report 2021. @techreport{guo2021accurate, title = {Towards Accurate and Compact Architectures via Neural Architecture Transformer}, author = {Yong Guo and Yin Zheng and Mingkui Tan and Qi Chen and Zhipeng Li and Jian Chen and Peilin Zhao and Junzhou Huang}, url = {https://arxiv.org/abs/2102.10301}, year = {2021}, date = {2021-01-01}, keywords = {}, pubstate = {published}, tppubtype = {techreport} } |
| 1012. | Nguyen, Nam; Chang, Morris J Contrastive Self-supervised Neural Architecture Search Miscellaneous 2021. @misc{nguyen2021contrastive, title = {Contrastive Self-supervised Neural Architecture Search}, author = {Nam Nguyen and Morris J Chang}, url = {https://arxiv.org/abs/2102.10557}, year = {2021}, date = {2021-01-01}, keywords = {}, pubstate = {published}, tppubtype = {misc} } |
| 1011. | Nayman, Niv; Aflalo, Yonathan; Noy, Asaf; Zelnik-Manor, Lihi HardCoRe-NAS: Hard Constrained diffeRentiable Neural Architecture Search Technical Report 2021. @techreport{nayman2021hardcore, title = {HardCoRe-NAS: Hard Constrained diffeRentiable Neural Architecture Search}, author = {Niv Nayman and Yonathan Aflalo and Asaf Noy and Lihi Zelnik-Manor}, url = {https://arxiv.org/abs/2102.11646}, year = {2021}, date = {2021-01-01}, journal = {arXiv preprint arXiv:2102.11646}, keywords = {}, pubstate = {published}, tppubtype = {techreport} } |
| 1010. | Hu, Liangxiao; Liu, Qinglin; Zhang, Jun; Jiang, Feng; Liu, Yang; Zhang, Shengping A-DARTS: attention-guided differentiable architecture search for lung nodule classification Journal Article Journal of Electronic Imaging, 30 (1), pp. 1 – 11, 2021. @article{10.1117/1.JEI.30.1.013012, title = {A-DARTS: attention-guided differentiable architecture search for lung nodule classification}, author = {Liangxiao Hu and Qinglin Liu and Jun Zhang and Feng Jiang and Yang Liu and Shengping Zhang}, url = {https://doi.org/10.1117/1.JEI.30.1.013012}, doi = {10.1117/1.JEI.30.1.013012}, year = {2021}, date = {2021-01-01}, journal = {Journal of Electronic Imaging}, volume = {30}, number = {1}, pages = {1 -- 11}, publisher = {SPIE}, abstract = {Lung cancer has caused the most cancer deaths in the past several years. Benign–malignant lung nodule classification is vital in lung nodule detection, which can help early diagnosis of lung cancer. Most existing works extract the features of chest CT images using the well-designed networks, which require substantial effort of experts. To automate the manual process of network design, we propose an attention-guided differentiable architecture search (A-DARTS) method, which directly searches for the optimal network on chest CT images. In addition, A-DARTS utilizes an attention mechanism to alleviate the effect of the initialization-sensitive nature of the searched network while enhancing the feature presentation ability. Extensive experiments on the Lung Image Database Consortium image collection (LIDC-IDRI) benchmark dataset show that the proposed method achieves a lung nodule classification accuracy of 92.93%, which is superior to the state-of-the-art methods.}, keywords = {}, pubstate = {published}, tppubtype = {article} } Lung cancer has caused the most cancer deaths in the past several years. Benign–malignant lung nodule classification is vital in lung nodule detection, which can help early diagnosis of lung cancer. Most existing works extract the features of chest CT images using the well-designed networks, which require substantial effort of experts. To automate the manual process of network design, we propose an attention-guided differentiable architecture search (A-DARTS) method, which directly searches for the optimal network on chest CT images. In addition, A-DARTS utilizes an attention mechanism to alleviate the effect of the initialization-sensitive nature of the searched network while enhancing the feature presentation ability. Extensive experiments on the Lung Image Database Consortium image collection (LIDC-IDRI) benchmark dataset show that the proposed method achieves a lung nodule classification accuracy of 92.93%, which is superior to the state-of-the-art methods. |
| 1009. | Zhou, Yanqi; Dong, Xuanyi; Akin, Berkin; Tan, Mingxing; Peng, Daiyi; Meng, Tianjian; Yazdanbakhsh, Amir; Huang, Da; Narayanaswami, Ravi; Laudon, James Rethinking Co-design of Neural Architectures and Hardware Accelerators Technical Report , 2021. @techreport{DBLP:journals/corr/abs-2102-08619, title = {Rethinking Co-design of Neural Architectures and Hardware Accelerators}, author = {Yanqi Zhou and Xuanyi Dong and Berkin Akin and Mingxing Tan and Daiyi Peng and Tianjian Meng and Amir Yazdanbakhsh and Da Huang and Ravi Narayanaswami and James Laudon}, url = {https://arxiv.org/abs/2102.08619}, year = {2021}, date = {2021-01-01}, journal = {CoRR}, volume = {abs/2102.08619}, keywords = {}, pubstate = {published}, tppubtype = {techreport} } |
| 1008. | Ru, Binxin; Wan, Xingchen; Dong, Xiaowen; Osborne, Michael Interpretable Neural Architecture Search via Bayesian Optimisation with Weisfeiler-Lehman Kernels Inproceedings International Conference on Learning Representations, 2021. @inproceedings{<LineBreak>ru2021interpretable, title = {Interpretable Neural Architecture Search via Bayesian Optimisation with Weisfeiler-Lehman Kernels}, author = {Binxin Ru and Xingchen Wan and Xiaowen Dong and Michael Osborne}, url = {https://openreview.net/forum?id=j9Rv7qdXjd}, year = {2021}, date = {2021-01-01}, booktitle = {International Conference on Learning Representations}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } |
| 1007. | Roberts, Nicholas Carl; Khodak, Mikhail; Dao, Tri; Li, Liam; Balcan, Nina; Re, Christopher; Talwalkar, Ameet Searching for Convolutions and a More Ambitious NAS Miscellaneous 2021. @misc{<LineBreak>roberts2021searching, title = {Searching for Convolutions and a More Ambitious NAS}, author = {Nicholas Carl Roberts and Mikhail Khodak and Tri Dao and Liam Li and Nina Balcan and Christopher Re and Ameet Talwalkar}, url = {https://openreview.net/forum?id=ascdLuNQY4J}, year = {2021}, date = {2021-01-01}, keywords = {}, pubstate = {published}, tppubtype = {misc} } |
| 1006. | Wang, Dilin; Gong, Chengyue; Li, Meng; Liu, Qiang; Chandra, Vikas AlphaNet: Improved Training of Supernet with Alpha-Divergence Technical Report , 2021. @techreport{DBLP:journals/corr/abs-2102-07954, title = {AlphaNet: Improved Training of Supernet with Alpha-Divergence}, author = {Dilin Wang and Chengyue Gong and Meng Li and Qiang Liu and Vikas Chandra}, url = {https://arxiv.org/abs/2102.07954}, year = {2021}, date = {2021-01-01}, journal = {CoRR}, volume = {abs/2102.07954}, keywords = {}, pubstate = {published}, tppubtype = {techreport} } |
| 1005. | Robbiano, Luca; Rahman, Muhammad Rameez Ur; Galasso, Fabio; Caputo, Barbara; Carlucci, Fabio Maria Adversarial Branch Architecture Search for Unsupervised Domain Adaptation Technical Report , 2021. @techreport{DBLP:journals/corr/abs-2102-06679, title = {Adversarial Branch Architecture Search for Unsupervised Domain Adaptation}, author = {Luca Robbiano and Muhammad Rameez Ur Rahman and Fabio Galasso and Barbara Caputo and Fabio Maria Carlucci}, url = {https://arxiv.org/abs/2102.06679}, year = {2021}, date = {2021-01-01}, journal = {CoRR}, volume = {abs/2102.06679}, keywords = {}, pubstate = {published}, tppubtype = {techreport} } |
| 1004. | Calisto, Maria Baldeon G; Lai-Yuen, Susana K EMONAS: efficient multiobjective neural architecture search framework for 3D medical image segmentation Inproceedings Išgum, Ivana; Landman, Bennett A (Ed.): Medical Imaging 2021: Image Processing, pp. 22 – 34, International Society for Optics and Photonics SPIE, 2021. @inproceedings{10.1117/12.2577088, title = {EMONAS: efficient multiobjective neural architecture search framework for 3D medical image segmentation}, author = {Maria Baldeon G Calisto and Susana K Lai-Yuen}, editor = {Ivana Išgum and Bennett A Landman}, url = {https://doi.org/10.1117/12.2577088}, doi = {10.1117/12.2577088}, year = {2021}, date = {2021-01-01}, booktitle = {Medical Imaging 2021: Image Processing}, volume = {11596}, pages = {22 -- 34}, publisher = {SPIE}, organization = {International Society for Optics and Photonics}, abstract = {Deep learning plays a critical role in medical image segmentation. Nevertheless, manually designing a neural network for a specific segmentation problem is a very difficult and time-consuming task due to the massive hyperparameter search space, long training time and large volumetric data. Therefore, most designed networks are highly complex, task specific and over-parametrized. Recently, multiobjective neural architecture search (NAS) methods have been proposed to automate the design of accurate and efficient segmentation architectures. However, they only search for either the macro- or micro-structure of the architecture, do not use the information produced during the optimization process to increase the efficiency of the search, and do not consider the volumetric nature of medical images. In this work, we propose EMONAS, an Efficient MultiObjective Neural Architecture Search framework for 3D medical image segmentation. EMONAS is composed of a search space that considers both the macro- and micro-structure of the architecture, and a surrogate-assisted multiobjective evolutionary based algorithm that efficiently searches for the best hyperparameters using a Random Forest surrogate and guiding selection probabilities. EMONAS is evaluated on the task of cardiac segmentation from the ACDC MICCAI challenge. The architecture found is ranked within the top 10 submissions in all evaluation metrics, performing better or comparable to other approaches while reducing the search time by more than 50% and having considerably fewer number of parameters.}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } Deep learning plays a critical role in medical image segmentation. Nevertheless, manually designing a neural network for a specific segmentation problem is a very difficult and time-consuming task due to the massive hyperparameter search space, long training time and large volumetric data. Therefore, most designed networks are highly complex, task specific and over-parametrized. Recently, multiobjective neural architecture search (NAS) methods have been proposed to automate the design of accurate and efficient segmentation architectures. However, they only search for either the macro- or micro-structure of the architecture, do not use the information produced during the optimization process to increase the efficiency of the search, and do not consider the volumetric nature of medical images. In this work, we propose EMONAS, an Efficient MultiObjective Neural Architecture Search framework for 3D medical image segmentation. EMONAS is composed of a search space that considers both the macro- and micro-structure of the architecture, and a surrogate-assisted multiobjective evolutionary based algorithm that efficiently searches for the best hyperparameters using a Random Forest surrogate and guiding selection probabilities. EMONAS is evaluated on the task of cardiac segmentation from the ACDC MICCAI challenge. The architecture found is ranked within the top 10 submissions in all evaluation metrics, performing better or comparable to other approaches while reducing the search time by more than 50% and having considerably fewer number of parameters. |
| 1003. | Yan, Shen; Song, Kaiqiang; Liu, Fei; Zhang, Mi CATE: Computation-aware Neural Architecture Encoding with Transformers Technical Report , 2021. @techreport{DBLP:journals/corr/abs-2102-07108, title = {CATE: Computation-aware Neural Architecture Encoding with Transformers}, author = {Shen Yan and Kaiqiang Song and Fei Liu and Mi Zhang}, url = {https://arxiv.org/abs/2102.07108}, year = {2021}, date = {2021-01-01}, journal = {CoRR}, volume = {abs/2102.07108}, keywords = {}, pubstate = {published}, tppubtype = {techreport} } |
| 1002. | Lopes, Vasco; Alirezazadeh, Saeid; í, Lu EPE-NAS: Efficient Performance Estimation Without Training for Neural Architecture Search Technical Report , 2021. @techreport{DBLP:journals/corr/abs-2102-08099, title = {EPE-NAS: Efficient Performance Estimation Without Training for Neural Architecture Search}, author = {Vasco Lopes and Saeid Alirezazadeh and Lu í}, url = {https://arxiv.org/abs/2102.08099}, year = {2021}, date = {2021-01-01}, journal = {CoRR}, volume = {abs/2102.08099}, keywords = {}, pubstate = {published}, tppubtype = {techreport} } |
| 1001. | Turner, Jack; Crowley, Elliot J; O'Boyle, Michael F P Neural Architecture Search as Program Transformation Exploration Technical Report , 2021. @techreport{DBLP:journals/corr/abs-2102-06599, title = {Neural Architecture Search as Program Transformation Exploration}, author = {Jack Turner and Elliot J Crowley and Michael F P O'Boyle}, url = {https://arxiv.org/abs/2102.06599}, year = {2021}, date = {2021-01-01}, journal = {CoRR}, volume = {abs/2102.06599}, keywords = {}, pubstate = {published}, tppubtype = {techreport} } |
| 1000. | Sun, Y; Sun, X; Fang, Y; Yen, G G; Liu, Y A Novel Training Protocol for Performance Predictors of Evolutionary Neural Architecture Search Algorithms Journal Article IEEE Transactions on Evolutionary Computation, pp. 1-1, 2021. @article{9336721, title = {A Novel Training Protocol for Performance Predictors of Evolutionary Neural Architecture Search Algorithms}, author = {Y Sun and X Sun and Y Fang and G G Yen and Y Liu}, url = {https://ieeexplore.ieee.org/document/9336721}, doi = {10.1109/TEVC.2021.3055076}, year = {2021}, date = {2021-01-01}, journal = {IEEE Transactions on Evolutionary Computation}, pages = {1-1}, abstract = {Evolutionary Neural Architecture Search (ENAS) can automatically design the architectures of Deep Neural Networks (DNNs) using evolutionary computation algorithms. However, most ENAS algorithms require intensive computational resource, which is not necessarily available to the users interested. Performance predictors are a type of regression models which can assist to accomplish the search, while without exerting much computational resource. Despite various performance predictors have been designed, they employ the same training protocol to build the regression models: 1) sampling a set of DNNs with performance as the training dataset, 2) training the model with the mean square error criterion, and 3) predicting the performance of DNNs newly generated during the ENAS. In this paper, we point out that the three steps constituting the training protocol are not well though-out through intuitive and illustrative examples. Furthermore, we propose a new training protocol to address these issues, consisting of designing a pairwise ranking indicator to construct the training target, proposing to use the logistic regression to fit the training samples, and developing a differential method to build the training instances. To verify the effectiveness of the proposed training protocol, four widely used regression models in the field of machine learning have been chosen to perform the comparisons on two benchmark datasets. The experimental results of all the comparisons demonstrate that the proposed training protocol can significantly improve the performance prediction accuracy against the traditional training protocols.}, keywords = {}, pubstate = {published}, tppubtype = {article} } Evolutionary Neural Architecture Search (ENAS) can automatically design the architectures of Deep Neural Networks (DNNs) using evolutionary computation algorithms. However, most ENAS algorithms require intensive computational resource, which is not necessarily available to the users interested. Performance predictors are a type of regression models which can assist to accomplish the search, while without exerting much computational resource. Despite various performance predictors have been designed, they employ the same training protocol to build the regression models: 1) sampling a set of DNNs with performance as the training dataset, 2) training the model with the mean square error criterion, and 3) predicting the performance of DNNs newly generated during the ENAS. In this paper, we point out that the three steps constituting the training protocol are not well though-out through intuitive and illustrative examples. Furthermore, we propose a new training protocol to address these issues, consisting of designing a pairwise ranking indicator to construct the training target, proposing to use the logistic regression to fit the training samples, and developing a differential method to build the training instances. To verify the effectiveness of the proposed training protocol, four widely used regression models in the field of machine learning have been chosen to perform the comparisons on two benchmark datasets. The experimental results of all the comparisons demonstrate that the proposed training protocol can significantly improve the performance prediction accuracy against the traditional training protocols. |
| 999. | Su, Xiu; You, Shan; Huang, Tao; Wang, Fei; Qian, Chen; Zhang, Changshui; Xu, Chang Locally Free Weight Sharing for Network Width Search Technical Report 2021. @techreport{su2021locally, title = {Locally Free Weight Sharing for Network Width Search}, author = {Xiu Su and Shan You and Tao Huang and Fei Wang and Chen Qian and Changshui Zhang and Chang Xu}, url = {https://arxiv.org/abs/2102.05258}, year = {2021}, date = {2021-01-01}, keywords = {}, pubstate = {published}, tppubtype = {techreport} } |
| 998. | Liu, Peidong; Zhang, Gengwei; Wang, Bochao; Xu, Hang; Liang, Xiaodan; Jiang, Yong; Li, Zhenguo Loss Function Discovery for Object Detection via Convergence-Simulation Driven Search Technical Report 2021. @techreport{liu2021loss, title = {Loss Function Discovery for Object Detection via Convergence-Simulation Driven Search}, author = {Peidong Liu and Gengwei Zhang and Bochao Wang and Hang Xu and Xiaodan Liang and Yong Jiang and Zhenguo Li}, url = {https://arxiv.org/abs/2102.04700}, year = {2021}, date = {2021-01-01}, keywords = {}, pubstate = {published}, tppubtype = {techreport} } |
| 997. | He, Xin; Zhao, Kaiyong; Chu, Xiaowen AutoML: A Survey of the State-of-the-Art Journal Article Knowledge-Based Systems, 212 , pp. 106622, 2021. @article{he2021automl, title = {AutoML: A Survey of the State-of-the-Art}, author = {Xin He and Kaiyong Zhao and Xiaowen Chu}, url = {https://www.sciencedirect.com/science/article/abs/pii/S0950705120307516}, year = {2021}, date = {2021-01-01}, journal = {Knowledge-Based Systems}, volume = {212}, pages = {106622}, publisher = {Elsevier}, abstract = {Deep learning (DL) techniques have obtained remarkable achievements on various tasks, such as image recognition, object detection, and language modeling. However, building a high-quality DL system for a specific task highly relies on human expertise, hindering its wide application. Meanwhile, automated machine learning (AutoML) is a promising solution for building a DL system without human assistance and is being extensively studied. This paper presents a comprehensive and up-to-date review of the state-of-the-art (SOTA) in AutoML. According to the DL pipeline, we introduce AutoML methods – covering data preparation, feature engineering, hyperparameter optimization, and neural architecture search (NAS) – with a particular focus on NAS, as it is currently a hot sub-topic of AutoML. We summarize the representative NAS algorithms’ performance on the CIFAR-10 and ImageNet datasets and further discuss the following subjects of NAS methods: one/two-stage NAS, one-shot NAS, joint hyperparameter and architecture optimization, and resource-aware NAS. Finally, we discuss some open problems related to the existing AutoML methods for future research.}, keywords = {}, pubstate = {published}, tppubtype = {article} } Deep learning (DL) techniques have obtained remarkable achievements on various tasks, such as image recognition, object detection, and language modeling. However, building a high-quality DL system for a specific task highly relies on human expertise, hindering its wide application. Meanwhile, automated machine learning (AutoML) is a promising solution for building a DL system without human assistance and is being extensively studied. This paper presents a comprehensive and up-to-date review of the state-of-the-art (SOTA) in AutoML. According to the DL pipeline, we introduce AutoML methods – covering data preparation, feature engineering, hyperparameter optimization, and neural architecture search (NAS) – with a particular focus on NAS, as it is currently a hot sub-topic of AutoML. We summarize the representative NAS algorithms’ performance on the CIFAR-10 and ImageNet datasets and further discuss the following subjects of NAS methods: one/two-stage NAS, one-shot NAS, joint hyperparameter and architecture optimization, and resource-aware NAS. Finally, we discuss some open problems related to the existing AutoML methods for future research. |
| 996. | Li, Sheng; Tan, Mingxing; Pang, Ruoming; Li, Andrew; Cheng, Liqun; Le, Quoc; Jouppi, Norman P Searching for Fast Model Families on Datacenter Accelerators Technical Report 2021. @techreport{li2021searching, title = {Searching for Fast Model Families on Datacenter Accelerators}, author = {Sheng Li and Mingxing Tan and Ruoming Pang and Andrew Li and Liqun Cheng and Quoc Le and Norman P Jouppi}, url = {https://arxiv.org/abs/2102.05610}, year = {2021}, date = {2021-01-01}, keywords = {}, pubstate = {published}, tppubtype = {techreport} } |
| 995. | Wang, W; Zhu, L Reliable Network Search Based on Evolutionary Algorithm Inproceedings 2021 International Conference on Computer, Control and Robotics (ICCCR), pp. 279-282, 2021. @inproceedings{9349406, title = {Reliable Network Search Based on Evolutionary Algorithm}, author = {W Wang and L Zhu}, url = {https://ieeexplore.ieee.org/abstract/document/9349406}, doi = {10.1109/ICCCR49711.2021.9349406}, year = {2021}, date = {2021-01-01}, booktitle = {2021 International Conference on Computer, Control and Robotics (ICCCR)}, pages = {279-282}, abstract = {In this paper, we propose a neural architecture compression method based on network search to design a lightweight model for the network compression. The reasonable search method is designed based on evolutionary algorithm and search space for searching the efficient neural architecture, called EANet. The experimental results on several benchmarks datasets show that the performance of the EANet is better and the storage space is smaller. Besides, the light-weight SSD variant detection network based on EANet is applied to a railway intelligent surveillance system, which achieves the practical application based on the CNN model.}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } In this paper, we propose a neural architecture compression method based on network search to design a lightweight model for the network compression. The reasonable search method is designed based on evolutionary algorithm and search space for searching the efficient neural architecture, called EANet. The experimental results on several benchmarks datasets show that the performance of the EANet is better and the storage space is smaller. Besides, the light-weight SSD variant detection network based on EANet is applied to a railway intelligent surveillance system, which achieves the practical application based on the CNN model. |
| 994. | Lou, Xiaoxuan; Guo, Shangwei; Zhang, Tianwei; Zhang, Yinqian; Liu, Yang When NAS Meets Watermarking: Ownership Verification of DNN Models via Cache Side Channels Technical Report , 2021. @techreport{DBLP:journals/corr/abs-2102-03523, title = {When NAS Meets Watermarking: Ownership Verification of DNN Models via Cache Side Channels}, author = {Xiaoxuan Lou and Shangwei Guo and Tianwei Zhang and Yinqian Zhang and Yang Liu}, url = {https://arxiv.org/abs/2102.03523}, year = {2021}, date = {2021-01-01}, journal = {CoRR}, volume = {abs/2102.03523}, keywords = {}, pubstate = {published}, tppubtype = {techreport} } |
| 993. | Fu, Xianya; Li, Wenrui; Chen, Qiurui; Zhang, Lianyi; Yang, Kai; Qing, Duzheng; Wang, Rui NASIL: Neural Network Architecture Searching for Incremental Learning in Image Classification Inproceedings Ning, Li; Chau, Vincent; Lau, Francis (Ed.): Parallel Architectures, Algorithms and Programming, pp. 68–80, Springer Singapore, Singapore, 2021, ISBN: 978-981-16-0010-4. @inproceedings{10.1007/978-981-16-0010-4_7, title = {NASIL: Neural Network Architecture Searching for Incremental Learning in Image Classification}, author = {Xianya Fu and Wenrui Li and Qiurui Chen and Lianyi Zhang and Kai Yang and Duzheng Qing and Rui Wang}, editor = {Li Ning and Vincent Chau and Francis Lau}, url = {https://link.springer.com/chapter/10.1007/978-981-16-0010-4_7}, isbn = {978-981-16-0010-4}, year = {2021}, date = {2021-01-01}, booktitle = {Parallel Architectures, Algorithms and Programming}, pages = {68--80}, publisher = {Springer Singapore}, address = {Singapore}, abstract = {``Catastrophic forgetting'' and scalability of tasks are two major challenges of incremental learning. Both of these issues were related to the insufficient capacity of machine learning model and the insufficiently trained weights as the increasing of tasks. In this paper, we try to figure out the impact of the neural network architecture to the performance of incremental learning in the case of image classification. During the increasing of tasks, we propose to use neural network architecture searching (NAS) to find a structure that fits the new tasks collection better. We build a NAS environment with reinforcement learning as the searching strategy and Long Short-Term Memory network as the controller network. Computation operation and connecting previous nodes are selected for each layer in the search phase. For each time a new group of tasks is added, the neural network architecture is searched and reorganized according to the training data set. To speed up the searching, we design a parameter sharing mechanism, in which the same building blocks in each layer share a group of parameters. We also introduce the quantified-parameter building blocks into the NAS, to identify the best candidate during each round of searching. We test our solution in cifar100 data set, the average accuracy outperforms the current representative solutions (LwEMC, iCaRL, GANIL) by 24.92%, 5.62%, and 3.6%, respectively, the more tasks added, the better our solution performs.}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } ``Catastrophic forgetting'' and scalability of tasks are two major challenges of incremental learning. Both of these issues were related to the insufficient capacity of machine learning model and the insufficiently trained weights as the increasing of tasks. In this paper, we try to figure out the impact of the neural network architecture to the performance of incremental learning in the case of image classification. During the increasing of tasks, we propose to use neural network architecture searching (NAS) to find a structure that fits the new tasks collection better. We build a NAS environment with reinforcement learning as the searching strategy and Long Short-Term Memory network as the controller network. Computation operation and connecting previous nodes are selected for each layer in the search phase. For each time a new group of tasks is added, the neural network architecture is searched and reorganized according to the training data set. To speed up the searching, we design a parameter sharing mechanism, in which the same building blocks in each layer share a group of parameters. We also introduce the quantified-parameter building blocks into the NAS, to identify the best candidate during each round of searching. We test our solution in cifar100 data set, the average accuracy outperforms the current representative solutions (LwEMC, iCaRL, GANIL) by 24.92%, 5.62%, and 3.6%, respectively, the more tasks added, the better our solution performs. |
| 992. | Li, Xiaohan; Xie, Ziyan; Lai, Taotao; Zhao, Fusheng; Xu, Haiyin; Chen, Riqing NAS-WFPN: Neural Architecture Search Weighted Feature Pyramid Networks for Object Detection Inproceedings Wang, Guojun; Chen, Bing; Li, Wei; Pietro, Roberto Di; Yan, Xuefeng; Han, Hao (Ed.): Security, Privacy, and Anonymity in Computation, Communication, and Storage, pp. 384–394, Springer International Publishing, Cham, 2021, ISBN: 978-3-030-68884-4. @inproceedings{10.1007/978-3-030-68884-4_32, title = {NAS-WFPN: Neural Architecture Search Weighted Feature Pyramid Networks for Object Detection}, author = {Xiaohan Li and Ziyan Xie and Taotao Lai and Fusheng Zhao and Haiyin Xu and Riqing Chen}, editor = {Guojun Wang and Bing Chen and Wei Li and Roberto Di Pietro and Xuefeng Yan and Hao Han}, url = {https://link.springer.com/chapter/10.1007/978-3-030-68884-4_32}, isbn = {978-3-030-68884-4}, year = {2021}, date = {2021-01-01}, booktitle = {Security, Privacy, and Anonymity in Computation, Communication, and Storage}, pages = {384--394}, publisher = {Springer International Publishing}, address = {Cham}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } |
| 991. | Luo, Renqian; Tan, Xu; Wang, Rui; Qin, Tao; Li, Jinzhu; Zhao, Sheng; Chen, Enhong; -, Tie LightSpeech: Lightweight and Fast Text to Speech with Neural Architecture Search Technical Report , 2021. @techreport{DBLP:journals/corr/abs-2102-04040, title = {LightSpeech: Lightweight and Fast Text to Speech with Neural Architecture Search}, author = {Renqian Luo and Xu Tan and Rui Wang and Tao Qin and Jinzhu Li and Sheng Zhao and Enhong Chen and Tie -}, url = {https://arxiv.org/abs/2102.04040}, year = {2021}, date = {2021-01-01}, journal = {CoRR}, volume = {abs/2102.04040}, keywords = {}, pubstate = {published}, tppubtype = {techreport} } |
| 990. | Lang, Sebastian; Reggelin, Tobias; Schmidt, Johann; Müller, Marcel; Nahhas, Abdulrahman Expert Systems with Applications, 172 , pp. 114666, 2021, ISSN: 0957-4174. @article{LANG2021114666, title = {NeuroEvolution of augmenting topologies for solving a two-stage hybrid flow shop scheduling problem: A comparison of different solution strategies}, author = {Sebastian Lang and Tobias Reggelin and Johann Schmidt and Marcel Müller and Abdulrahman Nahhas}, url = {https://www.sciencedirect.com/science/article/pii/S095741742100107X}, doi = {https://doi.org/10.1016/j.eswa.2021.114666}, issn = {0957-4174}, year = {2021}, date = {2021-01-01}, journal = {Expert Systems with Applications}, volume = {172}, pages = {114666}, abstract = {The article investigates the application of NeuroEvolution of Augmenting Topologies (NEAT) to generate and parameterize artificial neural networks (ANN) on determining allocation and sequencing decisions in a two-stage hybrid flow shop scheduling environment with family setup times. NEAT is a machine-learning and neural architecture search algorithm, which generates both, the structure and the hyper-parameters of an ANN. Our experiments show that NEAT can compete with state-of-the-art approaches in terms of solution quality and outperforms them regarding computational efficiency. The main contributions of this article are: (i) A comparison of five different strategies, evaluated with 14 different experiments, on how ANNs can be applied for solving allocation and sequencing problems in a hybrid flow shop environment, (ii) a comparison of the best identified NEAT strategy with traditional heuristic and metaheuristic approaches concerning solution quality and computational efficiency.}, keywords = {}, pubstate = {published}, tppubtype = {article} } The article investigates the application of NeuroEvolution of Augmenting Topologies (NEAT) to generate and parameterize artificial neural networks (ANN) on determining allocation and sequencing decisions in a two-stage hybrid flow shop scheduling environment with family setup times. NEAT is a machine-learning and neural architecture search algorithm, which generates both, the structure and the hyper-parameters of an ANN. Our experiments show that NEAT can compete with state-of-the-art approaches in terms of solution quality and outperforms them regarding computational efficiency. The main contributions of this article are: (i) A comparison of five different strategies, evaluated with 14 different experiments, on how ANNs can be applied for solving allocation and sequencing problems in a hybrid flow shop environment, (ii) a comparison of the best identified NEAT strategy with traditional heuristic and metaheuristic approaches concerning solution quality and computational efficiency. |
| 989. | Jie, R; Gao, J Differentiable Neural Architecture Search for High-Dimensional Time Series Forecasting Journal Article IEEE Access, 9 , pp. 20922-20932, 2021. @article{9340253, title = {Differentiable Neural Architecture Search for High-Dimensional Time Series Forecasting}, author = {R Jie and J Gao}, url = {https://ieeexplore.ieee.org/document/9340253}, doi = {10.1109/ACCESS.2021.3055555}, year = {2021}, date = {2021-01-01}, journal = {IEEE Access}, volume = {9}, pages = {20922-20932}, keywords = {}, pubstate = {published}, tppubtype = {article} } |
| 988. | Liang, Shuang; Tang, Changcheng; Ning, Xuefei; Zeng, Shulin; Yu, Jincheng; Wang, Yu; Guo, Kaiyuan; Yang, Diange; Lu, Tianyi; Yang, Huazhong Efficient Computing Platform Design for Autonomous Driving Systems Inproceedings Proceedings of the 26th Asia and South Pacific Design Automation Conference, pp. 734–741, Association for Computing Machinery, Tokyo, Japan, 2021, ISBN: 9781450379991. @inproceedings{10.1145/3394885.3431620, title = {Efficient Computing Platform Design for Autonomous Driving Systems}, author = {Shuang Liang and Changcheng Tang and Xuefei Ning and Shulin Zeng and Jincheng Yu and Yu Wang and Kaiyuan Guo and Diange Yang and Tianyi Lu and Huazhong Yang}, url = {https://doi.org/10.1145/3394885.3431620}, doi = {10.1145/3394885.3431620}, isbn = {9781450379991}, year = {2021}, date = {2021-01-01}, booktitle = {Proceedings of the 26th Asia and South Pacific Design Automation Conference}, pages = {734–741}, publisher = {Association for Computing Machinery}, address = {Tokyo, Japan}, series = {ASPDAC '21}, abstract = {Autonomous driving is becoming a hot topic in both academic and industrial communities. Traditional algorithms can hardly achieve the complex tasks and meet the high safety criteria. Recent research on deep learning shows significant performance improvement over traditional algorithms and is believed to be a strong candidate in autonomous driving system. Despite the attractive performance, deep learning does not solve the problem totally. The application scenario requires that an autonomous driving system must work in real-time to keep safety. But the high computation complexity of neural network model, together with complicated pre-process and post-process, brings great challenges. System designers need to do dedicated optimizations to make a practical computing platform for autonomous driving. In this paper, we introduce our work on efficient computing platform design for autonomous driving systems. In the software level, we introduce neural network compression and hardware-aware architecture search to reduce the workload. In the hardware level, we propose customized hardware accelerators for pre- and post-process of deep learning algorithms. Finally, we introduce the hardware platform design, NOVA-30, and our on-vehicle evaluation project.}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } Autonomous driving is becoming a hot topic in both academic and industrial communities. Traditional algorithms can hardly achieve the complex tasks and meet the high safety criteria. Recent research on deep learning shows significant performance improvement over traditional algorithms and is believed to be a strong candidate in autonomous driving system. Despite the attractive performance, deep learning does not solve the problem totally. The application scenario requires that an autonomous driving system must work in real-time to keep safety. But the high computation complexity of neural network model, together with complicated pre-process and post-process, brings great challenges. System designers need to do dedicated optimizations to make a practical computing platform for autonomous driving. In this paper, we introduce our work on efficient computing platform design for autonomous driving systems. In the software level, we introduce neural network compression and hardware-aware architecture search to reduce the workload. In the hardware level, we propose customized hardware accelerators for pre- and post-process of deep learning algorithms. Finally, we introduce the hardware platform design, NOVA-30, and our on-vehicle evaluation project. |
| 987. | Odema, Mohanad; Rashid, Nafiul; Faruque, Mohammad Abdullah Al Energy-Aware Design Methodology for Myocardial Infarction Detection on Low-Power Wearable Devices Inproceedings Proceedings of the 26th Asia and South Pacific Design Automation Conference, pp. 621–626, Association for Computing Machinery, Tokyo, Japan, 2021, ISBN: 9781450379991. @inproceedings{10.1145/3394885.3431513, title = {Energy-Aware Design Methodology for Myocardial Infarction Detection on Low-Power Wearable Devices}, author = {Mohanad Odema and Nafiul Rashid and Mohammad Abdullah Al Faruque}, url = {https://doi.org/10.1145/3394885.3431513}, doi = {10.1145/3394885.3431513}, isbn = {9781450379991}, year = {2021}, date = {2021-01-01}, booktitle = {Proceedings of the 26th Asia and South Pacific Design Automation Conference}, pages = {621–626}, publisher = {Association for Computing Machinery}, address = {Tokyo, Japan}, series = {ASPDAC '21}, abstract = {Myocardial Infarction (MI) is a heart disease that damages the heart muscle and requires immediate treatment. Its silent and recurrent nature necessitates real-time continuous monitoring of patients. Nowadays, wearable devices are smart enough to perform on-device processing of heartbeat segments and report any irregularities in them. However, the small form factor of wearable devices imposes resource constraints and requires energy-efficient solutions to satisfy them. In this paper, we propose a design methodology to automate the design space exploration of neural network architectures for MI detection. This methodology incorporates Neural Architecture Search (NAS) using Multi-Objective Bayesian Optimization (MOBO) to render Pareto optimal architectural models. These models minimize both detection error and energy consumption on the target device. The design space is inspired by Binary Convolutional Neural Networks (BCNNs) suited for mobile health applications with limited resources. The models' performance is validated using the PTB diagnostic ECG database from PhysioNet. Moreover, energy-related measurements are directly obtained from the target device in a typical hardware-in-the-loop fashion. Finally, we benchmark our models against other related works. One model exceeds state-of-the-art accuracy on wearable devices (reaching 91.22%), whereas others trade off some accuracy to reduce their energy consumption (by a factor reaching 8.26x).}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } Myocardial Infarction (MI) is a heart disease that damages the heart muscle and requires immediate treatment. Its silent and recurrent nature necessitates real-time continuous monitoring of patients. Nowadays, wearable devices are smart enough to perform on-device processing of heartbeat segments and report any irregularities in them. However, the small form factor of wearable devices imposes resource constraints and requires energy-efficient solutions to satisfy them. In this paper, we propose a design methodology to automate the design space exploration of neural network architectures for MI detection. This methodology incorporates Neural Architecture Search (NAS) using Multi-Objective Bayesian Optimization (MOBO) to render Pareto optimal architectural models. These models minimize both detection error and energy consumption on the target device. The design space is inspired by Binary Convolutional Neural Networks (BCNNs) suited for mobile health applications with limited resources. The models' performance is validated using the PTB diagnostic ECG database from PhysioNet. Moreover, energy-related measurements are directly obtained from the target device in a typical hardware-in-the-loop fashion. Finally, we benchmark our models against other related works. One model exceeds state-of-the-art accuracy on wearable devices (reaching 91.22%), whereas others trade off some accuracy to reduce their energy consumption (by a factor reaching 8.26x). |
| 986. | Lin, Ming; Wang, Pichao; Sun, Zhenhong; Chen, Hesen; Sun, Xiuyu; Qian, Qi; Li, Hao; Jin, Rong Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognition Technical Report , 2021. @techreport{DBLP:journals/corr/abs-2102-01063, title = {Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognition}, author = {Ming Lin and Pichao Wang and Zhenhong Sun and Hesen Chen and Xiuyu Sun and Qi Qian and Hao Li and Rong Jin}, url = {https://arxiv.org/abs/2102.01063}, year = {2021}, date = {2021-01-01}, journal = {CoRR}, volume = {abs/2102.01063}, keywords = {}, pubstate = {published}, tppubtype = {techreport} } |
| 985. | Gomez-Rosero, Santiago; Capretz, Miriam A M; Mir, Syed Transfer Learning by Similarity Centred Architecture Evolution for Multiple Residential Load Forecasting Journal Article Smart Cities, 4 (1), pp. 217–240, 2021, ISSN: 2624-6511. @article{smartcities4010014, title = {Transfer Learning by Similarity Centred Architecture Evolution for Multiple Residential Load Forecasting}, author = {Santiago Gomez-Rosero and Miriam A M Capretz and Syed Mir}, url = {https://www.mdpi.com/2624-6511/4/1/14}, doi = {10.3390/smartcities4010014}, issn = {2624-6511}, year = {2021}, date = {2021-01-01}, journal = {Smart Cities}, volume = {4}, number = {1}, pages = {217--240}, abstract = {The development from traditional low voltage grids to smart systems has become extensive and adopted worldwide. Expanding the demand response program to cover the residential sector raises a wide range of challenges. Short term load forecasting for residential consumers in a neighbourhood could lead to a better understanding of low voltage consumption behaviour. Nevertheless, users with similar characteristics can present diversity in consumption patterns. Consequently, transfer learning methods have become a useful tool to tackle differences among residential time series. This paper proposes a method combining evolutionary algorithms for neural architecture search with transfer learning to perform short term load forecasting in a neighbourhood with multiple household load consumption. The approach centres its efforts on neural architecture search using evolutionary algorithms. The neural architecture evolution process retains the patterns of the centre-most house, and later the architecture weights are adjusted for each house in a multihouse set from a neighbourhood. In addition, a sensitivity analysis was conducted to ensure model performance. Experimental results on a large dataset containing hourly load consumption for ten houses in London, Ontario showed that the performance of the proposed approach performs better than the compared techniques. Moreover, the proposed method presents the average accuracy performance of 3.17 points higher than the state-of-the-art LSTM one shot method.}, keywords = {}, pubstate = {published}, tppubtype = {article} } The development from traditional low voltage grids to smart systems has become extensive and adopted worldwide. Expanding the demand response program to cover the residential sector raises a wide range of challenges. Short term load forecasting for residential consumers in a neighbourhood could lead to a better understanding of low voltage consumption behaviour. Nevertheless, users with similar characteristics can present diversity in consumption patterns. Consequently, transfer learning methods have become a useful tool to tackle differences among residential time series. This paper proposes a method combining evolutionary algorithms for neural architecture search with transfer learning to perform short term load forecasting in a neighbourhood with multiple household load consumption. The approach centres its efforts on neural architecture search using evolutionary algorithms. The neural architecture evolution process retains the patterns of the centre-most house, and later the architecture weights are adjusted for each house in a multihouse set from a neighbourhood. In addition, a sensitivity analysis was conducted to ensure model performance. Experimental results on a large dataset containing hourly load consumption for ten houses in London, Ontario showed that the performance of the proposed approach performs better than the compared techniques. Moreover, the proposed method presents the average accuracy performance of 3.17 points higher than the state-of-the-art LSTM one shot method. |
| 984. | Lyu, B; Yuan, H; Lu, L; Zhang, Y Resource-constrained Neural Architecture Search on Edge Devices Journal Article IEEE Transactions on Network Science and Engineering, pp. 1-1, 2021. @article{9336306, title = {Resource-constrained Neural Architecture Search on Edge Devices}, author = {B Lyu and H Yuan and L Lu and Y Zhang}, url = {https://ieeexplore.ieee.org/document/9336306}, doi = {10.1109/TNSE.2021.3054583, keywords-NAS}, year = {2021}, date = {2021-01-01}, journal = {IEEE Transactions on Network Science and Engineering}, pages = {1-1}, keywords = {}, pubstate = {published}, tppubtype = {article} } |
| 983. | Pinos, Michal; Mrazek, Vojtech; á, Luk Evolutionary Neural Architecture Search Supporting Approximate Multipliers Technical Report , 2021. @techreport{DBLP:journals/corr/abs-2101-11883, title = {Evolutionary Neural Architecture Search Supporting Approximate Multipliers}, author = {Michal Pinos and Vojtech Mrazek and Luk á}, url = {https://arxiv.org/abs/2101.11883}, year = {2021}, date = {2021-01-01}, journal = {CoRR}, volume = {abs/2101.11883}, keywords = {}, pubstate = {published}, tppubtype = {techreport} } |
| 982. | Lu, Longfei; Lyu, Bo Reducing energy consumption of Neural Architecture Search: An inference latency prediction framework Journal Article Sustainable Cities and Society, 67 , pp. 102747, 2021, ISSN: 2210-6707. @article{LU2021102747, title = {Reducing energy consumption of Neural Architecture Search: An inference latency prediction framework}, author = {Longfei Lu and Bo Lyu}, url = {https://www.sciencedirect.com/science/article/pii/S221067072100041X}, doi = {https://doi.org/10.1016/j.scs.2021.102747}, issn = {2210-6707}, year = {2021}, date = {2021-01-01}, journal = {Sustainable Cities and Society}, volume = {67}, pages = {102747}, abstract = {Benefit from the success of NAS (Neural Architecture Search) in deep learning, humans are hopefully been released from the tremendous labor of manual tuning of structure and hyper-parameters. However, the success of NAS comes at the cost of much more computational resource consumption, thousands of times more computational power than ordinary training of manual-designed models, especially for the resource-aware multi-objective NAS, which must be serialized as a sequential loop of sampling, training, deployment, and inference. Recent research has shown that deep learning leads to huge energy consumption and CO2 emission (training of the namely Transformer can emit CO2 as much as five cars in their lifetimes Strubell et al. (2019)). Aiming to alleviate this issue, we propose the end-to-end inference latency prediction framework to empower the NAS process with a direct resource-aware efficiency indicator. Namely, we first propose the end-to-end latency prediction framework, which can predict latency quickly and accurately based on the dataset collected by ourselves. Eventually, we experimentally show that with the encoding scheme we designed, our proposed best model, LSTM-GBDT Latency Predictor(LGLP) achieves an excellent result of 0.9349 MSE, 0.5249 MAE, 0.9842 R2, and 0.9925 corrcoef. In other words, our limited dataset and encoding scheme already provide the precise knowledge representation of this large search space. By equipping NAS with the proposed framework, taking NEMO for example, it will save 1588 kWh⋅PUE energy, 1515 pounds CO2 emissions, and $3176 cloud compute cost of AWS. For NAS is now widely exploited in research or industry applications, this will bring incalculable benefits to society and the environment.}, keywords = {}, pubstate = {published}, tppubtype = {article} } Benefit from the success of NAS (Neural Architecture Search) in deep learning, humans are hopefully been released from the tremendous labor of manual tuning of structure and hyper-parameters. However, the success of NAS comes at the cost of much more computational resource consumption, thousands of times more computational power than ordinary training of manual-designed models, especially for the resource-aware multi-objective NAS, which must be serialized as a sequential loop of sampling, training, deployment, and inference. Recent research has shown that deep learning leads to huge energy consumption and CO2 emission (training of the namely Transformer can emit CO2 as much as five cars in their lifetimes Strubell et al. (2019)). Aiming to alleviate this issue, we propose the end-to-end inference latency prediction framework to empower the NAS process with a direct resource-aware efficiency indicator. Namely, we first propose the end-to-end latency prediction framework, which can predict latency quickly and accurately based on the dataset collected by ourselves. Eventually, we experimentally show that with the encoding scheme we designed, our proposed best model, LSTM-GBDT Latency Predictor(LGLP) achieves an excellent result of 0.9349 MSE, 0.5249 MAE, 0.9842 R2, and 0.9925 corrcoef. In other words, our limited dataset and encoding scheme already provide the precise knowledge representation of this large search space. By equipping NAS with the proposed framework, taking NEMO for example, it will save 1588 kWh⋅PUE energy, 1515 pounds CO2 emissions, and $3176 cloud compute cost of AWS. For NAS is now widely exploited in research or industry applications, this will bring incalculable benefits to society and the environment. |
| 981. | Yang, Yibo; You, Shan; Li, Hongyang; Wang, Fei; Qian, Chen; Lin, Zhouchen Towards Improving the Consistency, Efficiency, and Flexibility of Differentiable Neural Architecture Search Technical Report , 2021. @techreport{DBLP:journals/corr/abs-2101-11342, title = {Towards Improving the Consistency, Efficiency, and Flexibility of Differentiable Neural Architecture Search}, author = {Yibo Yang and Shan You and Hongyang Li and Fei Wang and Chen Qian and Zhouchen Lin}, url = {https://arxiv.org/abs/2101.11342}, year = {2021}, date = {2021-01-01}, journal = {CoRR}, volume = {abs/2101.11342}, keywords = {}, pubstate = {published}, tppubtype = {techreport} } |
| 980. | Liang, Xinle; Liu, Yang; Luo, Jiahuan; He, Yuanqin; Chen, Tianjian; Yang, Qiang Self-supervised Cross-silo Federated Neural Architecture Search Technical Report , 2021. @techreport{DBLP:journals/corr/abs-2101-11896, title = {Self-supervised Cross-silo Federated Neural Architecture Search}, author = {Xinle Liang and Yang Liu and Jiahuan Luo and Yuanqin He and Tianjian Chen and Qiang Yang}, url = {https://arxiv.org/abs/2101.11896}, year = {2021}, date = {2021-01-01}, journal = {CoRR}, volume = {abs/2101.11896}, keywords = {}, pubstate = {published}, tppubtype = {techreport} } |
| 979. | Zhang, Xuanyang; Hou, Pengfei; Zhang, Xiangyu; Sun, Jian Neural Architecture Search with Random Labels Technical Report , 2021. @techreport{DBLP:journals/corr/abs-2101-11834, title = {Neural Architecture Search with Random Labels}, author = {Xuanyang Zhang and Pengfei Hou and Xiangyu Zhang and Jian Sun}, url = {https://arxiv.org/abs/2101.11834}, year = {2021}, date = {2021-01-01}, journal = {CoRR}, volume = {abs/2101.11834}, keywords = {}, pubstate = {published}, tppubtype = {techreport} } |
| 978. | Kertész, Gábor; Szénási, Sándor; Vámossy, Zoltán Comparative analysis of image projection-based descriptors in Siamese neural networks Journal Article Advances in Engineering Software, 154 , pp. 102963, 2021, ISSN: 0965-9978. @article{KERTESZ2021102963, title = {Comparative analysis of image projection-based descriptors in Siamese neural networks}, author = {Gábor Kertész and Sándor Szénási and Zoltán Vámossy}, url = {https://www.sciencedirect.com/science/article/pii/S0965997820310097}, doi = {https://doi.org/10.1016/j.advengsoft.2020.102963}, issn = {0965-9978}, year = {2021}, date = {2021-01-01}, journal = {Advances in Engineering Software}, volume = {154}, pages = {102963}, abstract = {Low-level object matching can be done using projection signatures. In case of a large number of projections, the matching algorithm has to deal with less significant slices. A trivial approach would be to do statistical analysis or apply machine learning to determine the significant features. To take adjacent values of the projection matrices into account, a convolutional neural network should be used. To compare two matrices, a Siamese-structure of convolutional heads can be applied. In this paper, an experiment is designed and implemented to analyze the object matching performance of Siamese Convolutional Neural Networks based on multi-directional image projection data. A backtracking search-based Neural Architecture Generation method is used to create convolutional architectures, and a Master/Worker structured distributed processing with highly efficient scheduling based on the Longest Processing Times-heuristics is used for parallel training and evaluation of the models. Results show that the projection-based methods are Pareto optimal in terms of one-shot classification accuracy and memory consumption.}, keywords = {}, pubstate = {published}, tppubtype = {article} } Low-level object matching can be done using projection signatures. In case of a large number of projections, the matching algorithm has to deal with less significant slices. A trivial approach would be to do statistical analysis or apply machine learning to determine the significant features. To take adjacent values of the projection matrices into account, a convolutional neural network should be used. To compare two matrices, a Siamese-structure of convolutional heads can be applied. In this paper, an experiment is designed and implemented to analyze the object matching performance of Siamese Convolutional Neural Networks based on multi-directional image projection data. A backtracking search-based Neural Architecture Generation method is used to create convolutional architectures, and a Master/Worker structured distributed processing with highly efficient scheduling based on the Longest Processing Times-heuristics is used for parallel training and evaluation of the models. Results show that the projection-based methods are Pareto optimal in terms of one-shot classification accuracy and memory consumption. |
| 977. | Zhao, Jiakun; Zhang, Ruifeng; Zhou, Zheng; Chen, Si; Jin, Ju; Liu, Qingfang A Neural Architecture Search Method Based on Gradient Descent for Remaining Useful Life Estimation Journal Article Neurocomputing, 2021, ISSN: 0925-2312. @article{ZHAO2021, title = {A Neural Architecture Search Method Based on Gradient Descent for Remaining Useful Life Estimation}, author = {Jiakun Zhao and Ruifeng Zhang and Zheng Zhou and Si Chen and Ju Jin and Qingfang Liu}, url = {https://www.sciencedirect.com/science/article/pii/S092523122100148X}, doi = {https://doi.org/10.1016/j.neucom.2021.01.072}, issn = {0925-2312}, year = {2021}, date = {2021-01-01}, journal = {Neurocomputing}, abstract = {Remaining useful life is the estimated continuous normal working time of a component or system from the current moment to the potential failure. The traditional methods have high trial-and-error costs and poor migration capabilities. Fortunately, the neural architecture search (NAS) that has emerged partially solves the problem of automatic construction of network models. However, the search strategy for NAS is reinforcement learning or evolutionary algorithms, which essentially search in discrete space and treating the objective function as a black box, which is very time-consuming. To solve this problem, we proposed a gradient-based neural architecture search method. This method regards a cell in the search space as a directed acyclic graph (DAG) containing N ordered nodes. Each node is a latent representation, and the directed edges represent the conversion operation of two nodes. By mixing the candidate operations (ReLU, tanh) with the softmax function, the search space becomes a continuous space and the objective function becomes a differentiable function, so gradient-based optimization methods can be used to find the optimal structure. A neural architecture search method based on gradient descent for RUL estimation, with extensive experiments showing apparently, outperforms traditional approaches as well as Long Short-Term Memory (LSTM), and it takes much less computing resources than the reinforcement neural architecture search method.}, keywords = {}, pubstate = {published}, tppubtype = {article} } Remaining useful life is the estimated continuous normal working time of a component or system from the current moment to the potential failure. The traditional methods have high trial-and-error costs and poor migration capabilities. Fortunately, the neural architecture search (NAS) that has emerged partially solves the problem of automatic construction of network models. However, the search strategy for NAS is reinforcement learning or evolutionary algorithms, which essentially search in discrete space and treating the objective function as a black box, which is very time-consuming. To solve this problem, we proposed a gradient-based neural architecture search method. This method regards a cell in the search space as a directed acyclic graph (DAG) containing N ordered nodes. Each node is a latent representation, and the directed edges represent the conversion operation of two nodes. By mixing the candidate operations (ReLU, tanh) with the softmax function, the search space becomes a continuous space and the objective function becomes a differentiable function, so gradient-based optimization methods can be used to find the optimal structure. A neural architecture search method based on gradient descent for RUL estimation, with extensive experiments showing apparently, outperforms traditional approaches as well as Long Short-Term Memory (LSTM), and it takes much less computing resources than the reinforcement neural architecture search method. |
| 976. | He, Xin; Wang, Shihao; Ying, Guohao; Zhang, Jiyong; Chu, Xiaowen Efficient Multi-objective Evolutionary 3D Neural Architecture Search for COVID-19 Detection with Chest CT Scans Technical Report , 2021. @techreport{DBLP:journals/corr/abs-2101-10667, title = {Efficient Multi-objective Evolutionary 3D Neural Architecture Search for COVID-19 Detection with Chest CT Scans}, author = {Xin He and Shihao Wang and Guohao Ying and Jiyong Zhang and Xiaowen Chu}, url = {https://arxiv.org/abs/2101.10667}, year = {2021}, date = {2021-01-01}, journal = {CoRR}, volume = {abs/2101.10667}, keywords = {}, pubstate = {published}, tppubtype = {techreport} } |
| 975. | Benmeziane, Hadjer; Maghraoui, Kaoutar El; Ouarnoughi, Hamza; ï, Sma; Wistuba, Martin; Wang, Naigang A Comprehensive Survey on Hardware-Aware Neural Architecture Search Technical Report , 2021. @techreport{DBLP:journals/corr/abs-2101-09336, title = {A Comprehensive Survey on Hardware-Aware Neural Architecture Search}, author = {Hadjer Benmeziane and Kaoutar El Maghraoui and Hamza Ouarnoughi and Sma ï and Martin Wistuba and Naigang Wang}, url = {https://arxiv.org/abs/2101.09336}, year = {2021}, date = {2021-01-01}, journal = {CoRR}, volume = {abs/2101.09336}, keywords = {}, pubstate = {published}, tppubtype = {techreport} } |
| 974. | Abdelfattah, Mohamed S; Mehrotra, Abhinav; Dudziak, Lukasz; Lane, Nicholas D Zero-Cost Proxies for Lightweight NAS Technical Report , 2021. @techreport{DBLP:journals/corr/abs-2101-08134, title = {Zero-Cost Proxies for Lightweight NAS}, author = {Mohamed S Abdelfattah and Abhinav Mehrotra and Lukasz Dudziak and Nicholas D Lane}, url = {https://arxiv.org/abs/2101.08134}, year = {2021}, date = {2021-01-01}, journal = {CoRR}, volume = {abs/2101.08134}, keywords = {}, pubstate = {published}, tppubtype = {techreport} } |
| 973. | Fu, Chaoyou; Hu, Yibo; Wu, Xiang; Shi, Hailin; Mei, Tao; He, Ran CM-NAS: Rethinking Cross-Modality Neural Architectures for Visible-Infrared Person Re-Identification Technical Report , 2021. @techreport{DBLP:journals/corr/abs-2101-08467, title = {CM-NAS: Rethinking Cross-Modality Neural Architectures for Visible-Infrared Person Re-Identification}, author = {Chaoyou Fu and Yibo Hu and Xiang Wu and Hailin Shi and Tao Mei and Ran He}, url = {https://arxiv.org/abs/2101.08467}, year = {2021}, date = {2021-01-01}, journal = {CoRR}, volume = {abs/2101.08467}, keywords = {}, pubstate = {published}, tppubtype = {techreport} } |
| 972. | Wu, Haiwei; Zhou, Jiantao GIID-Net: Generalizable Image Inpainting Detection via Neural Architecture Search and Attention Technical Report , 2021. @techreport{DBLP:journals/corr/abs-2101-07419, title = {GIID-Net: Generalizable Image Inpainting Detection via Neural Architecture Search and Attention}, author = {Haiwei Wu and Jiantao Zhou}, url = {https://arxiv.org/abs/2101.07419}, year = {2021}, date = {2021-01-01}, journal = {CoRR}, volume = {abs/2101.07419}, keywords = {}, pubstate = {published}, tppubtype = {techreport} } |
| 971. | Song, Xingyou; Choromanski, Krzysztof; -, Jack Parker; Tang, Yunhao; Peng, Daiyi; Jain, Deepali; Gao, Wenbo; Pacchiano, Aldo; ó, Tamás Sarl; Yang, Yuxiang ES-ENAS: Combining Evolution Strategies with Neural Architecture Search at No Extra Cost for Reinforcement Learning Technical Report , 2021. @techreport{DBLP:journals/corr/abs-2101-07415, title = {ES-ENAS: Combining Evolution Strategies with Neural Architecture Search at No Extra Cost for Reinforcement Learning}, author = {Xingyou Song and Krzysztof Choromanski and Jack Parker - and Yunhao Tang and Daiyi Peng and Deepali Jain and Wenbo Gao and Aldo Pacchiano and Tamás Sarl ó and Yuxiang Yang}, url = {https://arxiv.org/abs/2101.07415}, year = {2021}, date = {2021-01-01}, journal = {CoRR}, volume = {abs/2101.07415}, keywords = {}, pubstate = {published}, tppubtype = {techreport} } |
| 970. | Li, Qing; Wu, Xia; Liu, Tianming Differentiable Neural Architecture Search for Optimal Spatial/Temporal Brain Function Network Decomposition Journal Article Medical Image Analysis, pp. 101974, 2021, ISSN: 1361-8415. @article{LI2021101974, title = {Differentiable Neural Architecture Search for Optimal Spatial/Temporal Brain Function Network Decomposition}, author = {Qing Li and Xia Wu and Tianming Liu}, url = {https://www.sciencedirect.com/science/article/pii/S1361841521000207}, doi = {https://doi.org/10.1016/j.media.2021.101974}, issn = {1361-8415}, year = {2021}, date = {2021-01-01}, journal = {Medical Image Analysis}, pages = {101974}, abstract = {ABSTRACT It has been a key topic to decompose the brain's spatial/temporal function networks from 4D functional magnetic resonance imaging (fMRI) data. With the advantages of robust and meaningful brain pattern extraction, deep neural networks have been shown to be more powerful and flexible in fMRI data modeling than other traditional methods. However, the challenge of designing neural network architecture for high-dimensional and complex fMRI data has also been realized recently. In this paper, we propose a new spatial/temporal differentiable neural architecture search algorithm (ST-DARTS) for optimal brain network decomposition. The core idea of ST-DARTS is to optimize the inner cell structure of the vanilla recurrent neural network (RNN) in order to effectively decompose spatial/temporal brain function networks from fMRI data. Based on the evaluations on all seven fMRI tasks in human connectome project (HCP) dataset, the ST-DARTS model is shown to perform promisingly, both spatially (i.e., it can recognize the most stimuli-correlated spatial brain network activation that is very similar to the benchmark) and temporally (i.e., its temporal activity is highly positively correlated with the task-design). To further improve the efficiency of ST-DARTS model, we introduce a flexible early-stopping mechanism, named as ST-DARTS±, which further improves experimental results significantly. To our best knowledge, the proposed ST-DARTS and ST-DARTS+ models are among the early efforts in optimally decomposing spatial/temporal brain function networks from fMRI data with neural architecture search strategy and they demonstrate great promise.}, keywords = {}, pubstate = {published}, tppubtype = {article} } ABSTRACT It has been a key topic to decompose the brain's spatial/temporal function networks from 4D functional magnetic resonance imaging (fMRI) data. With the advantages of robust and meaningful brain pattern extraction, deep neural networks have been shown to be more powerful and flexible in fMRI data modeling than other traditional methods. However, the challenge of designing neural network architecture for high-dimensional and complex fMRI data has also been realized recently. In this paper, we propose a new spatial/temporal differentiable neural architecture search algorithm (ST-DARTS) for optimal brain network decomposition. The core idea of ST-DARTS is to optimize the inner cell structure of the vanilla recurrent neural network (RNN) in order to effectively decompose spatial/temporal brain function networks from fMRI data. Based on the evaluations on all seven fMRI tasks in human connectome project (HCP) dataset, the ST-DARTS model is shown to perform promisingly, both spatially (i.e., it can recognize the most stimuli-correlated spatial brain network activation that is very similar to the benchmark) and temporally (i.e., its temporal activity is highly positively correlated with the task-design). To further improve the efficiency of ST-DARTS model, we introduce a flexible early-stopping mechanism, named as ST-DARTS±, which further improves experimental results significantly. To our best knowledge, the proposed ST-DARTS and ST-DARTS+ models are among the early efforts in optimally decomposing spatial/temporal brain function networks from fMRI data with neural architecture search strategy and they demonstrate great promise. |
| 969. | Vaccaro, Lorenzo; Sansonetti, Giuseppe; Micarelli, Alessandro An Empirical Review of Automated Machine Learning Journal Article Computers, 10 (1), 2021, ISSN: 2073-431X. @article{computers10010011, title = {An Empirical Review of Automated Machine Learning}, author = {Lorenzo Vaccaro and Giuseppe Sansonetti and Alessandro Micarelli}, url = {https://www.mdpi.com/2073-431X/10/1/11}, doi = {10.3390/computers10010011}, issn = {2073-431X}, year = {2021}, date = {2021-01-01}, journal = {Computers}, volume = {10}, number = {1}, abstract = {In recent years, Automated Machine Learning (AutoML) has become increasingly important in Computer Science due to the valuable potential it offers. This is testified by the high number of works published in the academic field and the significant efforts made in the industrial sector. However, some problems still need to be resolved. In this paper, we review some Machine Learning (ML) models and methods proposed in the literature to analyze their strengths and weaknesses. Then, we propose their use—alone or in combination with other approaches—to provide possible valid AutoML solutions. We analyze those solutions from a theoretical point of view and evaluate them empirically on three Atari games from the Arcade Learning Environment. Our goal is to identify what, we believe, could be some promising ways to create truly effective AutoML frameworks, therefore able to replace the human expert as much as possible, thereby making easier the process of applying ML approaches to typical problems of specific domains. We hope that the findings of our study will provide useful insights for future research work in AutoML.}, keywords = {}, pubstate = {published}, tppubtype = {article} } In recent years, Automated Machine Learning (AutoML) has become increasingly important in Computer Science due to the valuable potential it offers. This is testified by the high number of works published in the academic field and the significant efforts made in the industrial sector. However, some problems still need to be resolved. In this paper, we review some Machine Learning (ML) models and methods proposed in the literature to analyze their strengths and weaknesses. Then, we propose their use—alone or in combination with other approaches—to provide possible valid AutoML solutions. We analyze those solutions from a theoretical point of view and evaluate them empirically on three Atari games from the Arcade Learning Environment. Our goal is to identify what, we believe, could be some promising ways to create truly effective AutoML frameworks, therefore able to replace the human expert as much as possible, thereby making easier the process of applying ML approaches to typical problems of specific domains. We hope that the findings of our study will provide useful insights for future research work in AutoML. |