Maintained by Difan Deng and Marius Lindauer.
The following list considers papers related to neural architecture search. It is by no means complete. If you miss a paper on the list, please let us know.
Please note that although NAS methods steadily improve, the quality of empirical evaluations in this field are still lagging behind compared to other areas in machine learning, AI and optimization. We would therefore like to share some best practices for empirical evaluations of NAS methods, which we believe will facilitate sustained and measurable progress in the field. If you are interested in a teaser, please read our blog post or directly jump to our checklist.
Transformers have gained increasing popularity in different domains. For a comprehensive list of papers focusing on Neural Architecture Search for Transformer-Based spaces, the awesome-transformer-search repo is all you need.
2022
2108.
Oloulade, Babatounde Moctard; Gao, Jianliang; Chen, Jiamin; Lyu, Tengfei; Al-Sabri, Raeed
Graph neural architecture search: A survey Journal Article
In: Tsinghua Science and Technology, vol. 27, no. 4, pp. 692-708, 2022.
@article{9645440,
title = {Graph neural architecture search: A survey},
author = {Babatounde Moctard Oloulade and Jianliang Gao and Jiamin Chen and Tengfei Lyu and Raeed Al-Sabri},
url = {https://ieeexplore.ieee.org/document/9645440},
doi = {10.26599/TST.2021.9010057},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Tsinghua Science and Technology},
volume = {27},
number = {4},
pages = {692-708},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2107.
Baymurzina, Dilyara; Golikov, Eugene; Burtsev, Mikhail
A review of neural architecture search Journal Article
In: Neurocomputing, vol. 474, pp. 82-93, 2022, ISSN: 0925-2312.
@article{BAYMURZINA202282,
title = {A review of neural architecture search},
author = {Dilyara Baymurzina and Eugene Golikov and Mikhail Burtsev},
url = {https://www.sciencedirect.com/science/article/pii/S0925231221018439},
doi = {https://doi.org/10.1016/j.neucom.2021.12.014},
issn = {0925-2312},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Neurocomputing},
volume = {474},
pages = {82-93},
abstract = {Despite the impressive progress in neural network architecture design, improving the performance of the existing state-of-the-art models has become increasingly challenging. For this reason, the paradigm for neural architecture design is shifting from being expert-driven to almost fully automated. An emerging body of research related to such machine-aided design is called a Neural Architecture Search (NAS). This paper reviews the recent works on NAS and highlights several crucial concepts and problems of this field.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Despite the impressive progress in neural network architecture design, improving the performance of the existing state-of-the-art models has become increasingly challenging. For this reason, the paradigm for neural architecture design is shifting from being expert-driven to almost fully automated. An emerging body of research related to such machine-aided design is called a Neural Architecture Search (NAS). This paper reviews the recent works on NAS and highlights several crucial concepts and problems of this field.
2106.
Fan, Zhenkun; Hu, Guosheng; Sun, Xin; Wang, Gaige; Dong, Junyu; Su, Chi
Self-attention neural architecture search for semantic image segmentation Journal Article
In: Knowledge-Based Systems, vol. 239, pp. 107968, 2022, ISSN: 0950-7051.
@article{FAN2022107968,
title = {Self-attention neural architecture search for semantic image segmentation},
author = {Zhenkun Fan and Guosheng Hu and Xin Sun and Gaige Wang and Junyu Dong and Chi Su},
url = {https://www.sciencedirect.com/science/article/pii/S0950705121010947},
doi = {https://doi.org/10.1016/j.knosys.2021.107968},
issn = {0950-7051},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Knowledge-Based Systems},
volume = {239},
pages = {107968},
abstract = {Self-attention can capture long-distance dependencies and is widely used in semantic segmentation. Existing methods mainly use two kinds of self-attentions, i.e., spatial attention and channel attention, which can capture the relations in HW dimension (image plane, height and width) and C dimension (channels), respectively. Very little research investigates self-attention along other dimensions, which can potentially improve the segmentation performance. In this work, we investigate the self-attentions along all the possible dimensions H,W,C,HW,HC,CW,HWC. Then we explore the aggregation of all the possible self-attentions. We apply the neural architecture search (NAS) technique to achieve optimal aggregation. Specifically, we carefully design (1) the search space and (2) the optimization method. For (1), we introduce a building block, a basic self-attention search unit (BSU), which can model self-attentions along all the dimensions. And the search space contains within-BSU and cross-BSU operations. In addition, we propose an attention-map splitting method, which can reduce the computations by 1/3. For (2), we apply an efficient differentiable optimization method to search the optimal aggregation. We conduct extensive experiments on Cityscapes and ADE20K datasets. The results show the effectiveness of the proposed method, and we achieve very competitive performance against state-of-the-art methods.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Self-attention can capture long-distance dependencies and is widely used in semantic segmentation. Existing methods mainly use two kinds of self-attentions, i.e., spatial attention and channel attention, which can capture the relations in HW dimension (image plane, height and width) and C dimension (channels), respectively. Very little research investigates self-attention along other dimensions, which can potentially improve the segmentation performance. In this work, we investigate the self-attentions along all the possible dimensions H,W,C,HW,HC,CW,HWC. Then we explore the aggregation of all the possible self-attentions. We apply the neural architecture search (NAS) technique to achieve optimal aggregation. Specifically, we carefully design (1) the search space and (2) the optimization method. For (1), we introduce a building block, a basic self-attention search unit (BSU), which can model self-attentions along all the dimensions. And the search space contains within-BSU and cross-BSU operations. In addition, we propose an attention-map splitting method, which can reduce the computations by 1/3. For (2), we apply an efficient differentiable optimization method to search the optimal aggregation. We conduct extensive experiments on Cityscapes and ADE20K datasets. The results show the effectiveness of the proposed method, and we achieve very competitive performance against state-of-the-art methods.
2105.
Liu, Shengfeng; Ge, Fangfei; Zhao, Lin; Wang, Tianfu; Ni, Dong; Liu, Tianming
NAS-optimized topology-preserving transfer learning for differentiating cortical folding patterns Journal Article
In: Medical Image Analysis, vol. 77, pp. 102316, 2022, ISSN: 1361-8415.
@article{LIU2022102316,
title = {NAS-optimized topology-preserving transfer learning for differentiating cortical folding patterns},
author = {Shengfeng Liu and Fangfei Ge and Lin Zhao and Tianfu Wang and Dong Ni and Tianming Liu},
url = {https://www.sciencedirect.com/science/article/pii/S1361841521003613},
doi = {https://doi.org/10.1016/j.media.2021.102316},
issn = {1361-8415},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Medical Image Analysis},
volume = {77},
pages = {102316},
abstract = {Increasing evidence suggests that cortical folding patterns of human cerebral cortex manifest overt structural and functional differences. However, for interpretability, few studies leverage advanced techniques (e.g., deep learning) to investigate the difference among cortical folds, resulting in more differences yet to be extensively explored. To this end, we proposed an effective topology-preserving transfer learning framework to differentiate cortical fMRI time series extracted from cortical folds. Our framework consists of three main parts: (1) Neural architecture search (NAS), which is used to devise a well-performing network structure based on an initialized hand-designed super-graph in an image dataset; (2) Topology-preserving transfer, which takes the model searched by NAS as the source network, keeping the topological connectivity in the network unchanged, while transforming all 2D operations including convolution and pooling into 1D, therefore resulting in a topology-preserving network, named TPNAS-Net; (3) Classification and correlation analysis, which involves using the TPNAS-Net to classify 1D cortical fMRI time series for each individual brain, and performing a group difference analysis between autism spectrum disorder (ASD) and healthy control (HC) and correlation analysis with clinical information (i.e., age). Extensive experiments on two ASD datasets obtain consistent results, demonstrating that the TPNAS-Net not only discriminates cortical folding patterns at high classification accuracy, but also captures subtle differences between ASD and HC (p-value = 0.042). In addition, we discover that there is a positive correlation between the classification accuracy and age in ASD (r = 0.39},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Increasing evidence suggests that cortical folding patterns of human cerebral cortex manifest overt structural and functional differences. However, for interpretability, few studies leverage advanced techniques (e.g., deep learning) to investigate the difference among cortical folds, resulting in more differences yet to be extensively explored. To this end, we proposed an effective topology-preserving transfer learning framework to differentiate cortical fMRI time series extracted from cortical folds. Our framework consists of three main parts: (1) Neural architecture search (NAS), which is used to devise a well-performing network structure based on an initialized hand-designed super-graph in an image dataset; (2) Topology-preserving transfer, which takes the model searched by NAS as the source network, keeping the topological connectivity in the network unchanged, while transforming all 2D operations including convolution and pooling into 1D, therefore resulting in a topology-preserving network, named TPNAS-Net; (3) Classification and correlation analysis, which involves using the TPNAS-Net to classify 1D cortical fMRI time series for each individual brain, and performing a group difference analysis between autism spectrum disorder (ASD) and healthy control (HC) and correlation analysis with clinical information (i.e., age). Extensive experiments on two ASD datasets obtain consistent results, demonstrating that the TPNAS-Net not only discriminates cortical folding patterns at high classification accuracy, but also captures subtle differences between ASD and HC (p-value = 0.042). In addition, we discover that there is a positive correlation between the classification accuracy and age in ASD (r = 0.39
2104.
Sivangi, Kaushik Bhargav; Dasari, Chandra Mohan; Amilpur, Santhosh; Bhukya, Raju
NoAS-DS: Neural optimal architecture search for detection of diverse DNA signals Journal Article
In: Neural Networks, vol. 147, pp. 63-71, 2022, ISSN: 0893-6080.
@article{SIVANGI202263,
title = {NoAS-DS: Neural optimal architecture search for detection of diverse DNA signals},
author = {Kaushik Bhargav Sivangi and Chandra Mohan Dasari and Santhosh Amilpur and Raju Bhukya},
url = {https://www.sciencedirect.com/science/article/pii/S0893608021004822},
doi = {https://doi.org/10.1016/j.neunet.2021.12.009},
issn = {0893-6080},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Neural Networks},
volume = {147},
pages = {63-71},
abstract = {Neural network architectures are high-performing variable models that can solve many learning tasks. Designing architectures manually require substantial time and also prior knowledge and expertise to develop a high-accuracy model. Most of the architecture search methods are developed over the task of image classification resulting in the building of complex architectures intended for large data inputs such as images. Motivated by the applications of DNA computing in Neural Architecture Search (NAS), we propose NoAS-DS which is specifically built for the architecture search of sequence-based classification tasks. Furthermore, NoAS-DS is applied to the task of predicting binding sites. Unlike other methods that implement only Convolution layers, NoAS-DS, specifically combines Convolution and LSTM layers that helps in the process of automatic architecture building. This hybrid approach helped in achieving high accuracy results on TFBS and RBP datasets which outperformed other models in TF-DNA binding prediction tasks. The best architectures generated by the proposed model can be applied to other DNA datasets of similar nature using transfer learning technique that demonstrates its generalization capability. This greatly reduces the effort required to build new architectures for other prediction tasks.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Neural network architectures are high-performing variable models that can solve many learning tasks. Designing architectures manually require substantial time and also prior knowledge and expertise to develop a high-accuracy model. Most of the architecture search methods are developed over the task of image classification resulting in the building of complex architectures intended for large data inputs such as images. Motivated by the applications of DNA computing in Neural Architecture Search (NAS), we propose NoAS-DS which is specifically built for the architecture search of sequence-based classification tasks. Furthermore, NoAS-DS is applied to the task of predicting binding sites. Unlike other methods that implement only Convolution layers, NoAS-DS, specifically combines Convolution and LSTM layers that helps in the process of automatic architecture building. This hybrid approach helped in achieving high accuracy results on TFBS and RBP datasets which outperformed other models in TF-DNA binding prediction tasks. The best architectures generated by the proposed model can be applied to other DNA datasets of similar nature using transfer learning technique that demonstrates its generalization capability. This greatly reduces the effort required to build new architectures for other prediction tasks.
2103.
Zhang, Yang; Zong, Ruohan; Kou, Ziyi; Shang, Lanyu; Wang, Dong
On streaming disaster damage assessment in social sensing: A crowd-driven dynamic neural architecture searching approach Journal Article
In: Knowledge-Based Systems, vol. 239, pp. 107984, 2022, ISSN: 0950-7051.
@article{ZHANG2022107984,
title = {On streaming disaster damage assessment in social sensing: A crowd-driven dynamic neural architecture searching approach},
author = {Yang Zhang and Ruohan Zong and Ziyi Kou and Lanyu Shang and Dong Wang},
url = {https://www.sciencedirect.com/science/article/pii/S0950705121011023},
doi = {https://doi.org/10.1016/j.knosys.2021.107984},
issn = {0950-7051},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Knowledge-Based Systems},
volume = {239},
pages = {107984},
abstract = {Motivated by the recent advances in Internet and communication techniques and the proliferation of online social media, social sensing has emerged as a new sensing paradigm to obtain timely observations of the physical world from “human sensors”. In this study, we focus on an emerging application in social sensing – streaming disaster damage assessment (DDA), which aims to automatically assess the damage severity of affected areas in a disaster event on the fly by leveraging the streaming imagery data about the disaster on social media. In particular, we study a dynamic optimal neural architecture searching (NAS) problem in streaming DDA applications. Our goal is to dynamically determine the optimal neural network architecture that accurately estimates the damage severity for each newly arrived image in the stream by leveraging human intelligence from the crowdsourcing systems. The present study is motivated by the observation that the neural network architectures in current DDA solutions are mainly designed by artificial intelligence (AI) experts, which often leads to non-negligible costs and errors given the dynamic nature of the streaming DDA applications and the lack of real-time annotations of the massive social media data inputs. Two critical technical challenges exist in solving our problem: (i) it is non-trivial to dynamically identify the optimal neural network architecture for each image on the fly without knowing its ground-truth label a priori; (ii) it is challenging to effectively leverage the imperfect crowd intelligence to correctly identify the optimal neural network architecture for each image. To address the above challenges, we developed CD-NAS, a dynamic crowd-AI collaborative NAS framework that carefully explores the human intelligence from crowdsourcing systems to solve the dynamic optimal NAS problem and optimize the performance of streaming DDA applications. The evaluation results from a real-world streaming DDA application show that CD-NAS consistently outperforms the state-of-the-art AI and NAS baselines by achieving the highest disaster damage assessment accuracy while maintaining the lowest computational cost.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Motivated by the recent advances in Internet and communication techniques and the proliferation of online social media, social sensing has emerged as a new sensing paradigm to obtain timely observations of the physical world from “human sensors”. In this study, we focus on an emerging application in social sensing – streaming disaster damage assessment (DDA), which aims to automatically assess the damage severity of affected areas in a disaster event on the fly by leveraging the streaming imagery data about the disaster on social media. In particular, we study a dynamic optimal neural architecture searching (NAS) problem in streaming DDA applications. Our goal is to dynamically determine the optimal neural network architecture that accurately estimates the damage severity for each newly arrived image in the stream by leveraging human intelligence from the crowdsourcing systems. The present study is motivated by the observation that the neural network architectures in current DDA solutions are mainly designed by artificial intelligence (AI) experts, which often leads to non-negligible costs and errors given the dynamic nature of the streaming DDA applications and the lack of real-time annotations of the massive social media data inputs. Two critical technical challenges exist in solving our problem: (i) it is non-trivial to dynamically identify the optimal neural network architecture for each image on the fly without knowing its ground-truth label a priori; (ii) it is challenging to effectively leverage the imperfect crowd intelligence to correctly identify the optimal neural network architecture for each image. To address the above challenges, we developed CD-NAS, a dynamic crowd-AI collaborative NAS framework that carefully explores the human intelligence from crowdsourcing systems to solve the dynamic optimal NAS problem and optimize the performance of streaming DDA applications. The evaluation results from a real-world streaming DDA application show that CD-NAS consistently outperforms the state-of-the-art AI and NAS baselines by achieving the highest disaster damage assessment accuracy while maintaining the lowest computational cost.
2102.
Chen, Dong; Shen, Hao; Shen, Yuchen
PT-NAS: Designing efficient keypoint-based object detectors for desktop CPU platforms Journal Article
In: Neurocomputing, vol. 476, pp. 38-52, 2022, ISSN: 0925-2312.
@article{CHEN202238,
title = {PT-NAS: Designing efficient keypoint-based object detectors for desktop CPU platforms},
author = {Dong Chen and Hao Shen and Yuchen Shen},
url = {https://www.sciencedirect.com/science/article/pii/S0925231221019238},
doi = {https://doi.org/10.1016/j.neucom.2021.12.067},
issn = {0925-2312},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Neurocomputing},
volume = {476},
pages = {38-52},
abstract = {Recently, keypoint-based object detectors have attracted widespread attention, due to their novel structure and excellent performance. However, in terms of their design, there are still two limitations: 1) Most keypoint-based methods are designed for GPU platforms, which makes them inefficient on desktop CPU platforms. 2) Existing works still rely heavily on manual design and prior knowledge. To this end, this work aims to offer a practical solution for designing CPU-efficient key-point detectors. First, we present a set of practical design guidelines by comparing different detection architectures. Following the proposed guidelines, we further develop a progressive three-phase network architecture search (PT-NAS) to achieve the automated design of detection architectures. Benefiting from our hierarchical search space and novel search pipeline, our PT-NAS not only achieves higher search efficiency, but also satisfies the practicality of CPU platforms. On the MS-COCO benchmark, we utilize our PT-NAS to generate several key-point detectors for fast inference on desktop CPUs. Finally, comprehensive comparison experiments prove that the proposed PT-NAS can produce new state-of-the-art keypoint-based detectors for CPU platforms.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Recently, keypoint-based object detectors have attracted widespread attention, due to their novel structure and excellent performance. However, in terms of their design, there are still two limitations: 1) Most keypoint-based methods are designed for GPU platforms, which makes them inefficient on desktop CPU platforms. 2) Existing works still rely heavily on manual design and prior knowledge. To this end, this work aims to offer a practical solution for designing CPU-efficient key-point detectors. First, we present a set of practical design guidelines by comparing different detection architectures. Following the proposed guidelines, we further develop a progressive three-phase network architecture search (PT-NAS) to achieve the automated design of detection architectures. Benefiting from our hierarchical search space and novel search pipeline, our PT-NAS not only achieves higher search efficiency, but also satisfies the practicality of CPU platforms. On the MS-COCO benchmark, we utilize our PT-NAS to generate several key-point detectors for fast inference on desktop CPUs. Finally, comprehensive comparison experiments prove that the proposed PT-NAS can produce new state-of-the-art keypoint-based detectors for CPU platforms.
2101.
Zhang, Zhen; Liu, Shanghao; Zhang, Yang; Chen, Wenbo
RS-DARTS: A Convolutional Neural Architecture Search for Remote Sensing Image Scene Classification Journal Article
In: Remote Sensing, vol. 14, no. 1, 2022, ISSN: 2072-4292.
@article{rs14010141,
title = {RS-DARTS: A Convolutional Neural Architecture Search for Remote Sensing Image Scene Classification},
author = {Zhen Zhang and Shanghao Liu and Yang Zhang and Wenbo Chen},
url = {https://www.mdpi.com/2072-4292/14/1/141},
doi = {10.3390/rs14010141},
issn = {2072-4292},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Remote Sensing},
volume = {14},
number = {1},
abstract = {Due to the superiority of convolutional neural networks, many deep learning methods have been used in image classification. The enormous difference between natural images and remote sensing images makes it difficult to directly utilize or modify existing CNN models for remote sensing scene classification tasks. In this article, a new paradigm is proposed that can automatically design a suitable CNN architecture for scene classification. A more efficient search framework, RS-DARTS, is adopted to find the optimal network architecture. This framework has two phases. In the search phase, some new strategies are presented, making the calculation process smoother, and better distinguishing the optimal and other operations. In addition, we added noise to suppress skip connections in order to close the gap between trained and validation processing and ensure classification accuracy. Moreover, a small part of the neural network is sampled to reduce the redundancy in exploring the network space and speed up the search processing. In the evaluation phase, the optimal cell architecture is stacked to construct the final network. Extensive experiments demonstrated the validity of the search strategy and the impressive classification performance of RS-DARTS on four public benchmark datasets. The proposed method showed more effectiveness than the manually designed CNN model and other methods of neural architecture search. Especially, in terms of search cost, RS-DARTS consumed less time than other NAS methods.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Due to the superiority of convolutional neural networks, many deep learning methods have been used in image classification. The enormous difference between natural images and remote sensing images makes it difficult to directly utilize or modify existing CNN models for remote sensing scene classification tasks. In this article, a new paradigm is proposed that can automatically design a suitable CNN architecture for scene classification. A more efficient search framework, RS-DARTS, is adopted to find the optimal network architecture. This framework has two phases. In the search phase, some new strategies are presented, making the calculation process smoother, and better distinguishing the optimal and other operations. In addition, we added noise to suppress skip connections in order to close the gap between trained and validation processing and ensure classification accuracy. Moreover, a small part of the neural network is sampled to reduce the redundancy in exploring the network space and speed up the search processing. In the evaluation phase, the optimal cell architecture is stacked to construct the final network. Extensive experiments demonstrated the validity of the search strategy and the impressive classification performance of RS-DARTS on four public benchmark datasets. The proposed method showed more effectiveness than the manually designed CNN model and other methods of neural architecture search. Especially, in terms of search cost, RS-DARTS consumed less time than other NAS methods.
2100.
Reiser, Daniel; Reichel, Peter; Pechmann, Stefan; Mallah, Maen; Oppelt, Maximilian; Hagelauer, Amelie; Breiling, Marco; Fey, Dietmar; Reichenbach, Marc
A Framework for Ultra Low-Power Hardware Accelerators Using NNs for Embedded Time Series Classification Journal Article
In: Journal of Low Power Electronics and Applications, vol. 12, no. 1, 2022, ISSN: 2079-9268.
@article{jlpea12010002,
title = {A Framework for Ultra Low-Power Hardware Accelerators Using NNs for Embedded Time Series Classification},
author = {Daniel Reiser and Peter Reichel and Stefan Pechmann and Maen Mallah and Maximilian Oppelt and Amelie Hagelauer and Marco Breiling and Dietmar Fey and Marc Reichenbach},
url = {https://www.mdpi.com/2079-9268/12/1/2},
doi = {10.3390/jlpea12010002},
issn = {2079-9268},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Journal of Low Power Electronics and Applications},
volume = {12},
number = {1},
abstract = {In embedded applications that use neural networks (NNs) for classification tasks, it is important to not only minimize the power consumption of the NN calculation, but of the whole system. Optimization approaches for individual parts exist, such as quantization of the NN or analog calculation of arithmetic operations. However, there is no holistic approach for a complete embedded system design that is generic enough in the design process to be used for different applications, but specific in the hardware implementation to waste no energy for a given application. Therefore, we present a novel framework that allows an end-to-end ASIC implementation of a low-power hardware for time series classification using NNs. This includes a neural architecture search (NAS), which optimizes the NN configuration for accuracy and energy efficiency at the same time. This optimization targets a custom designed hardware architecture that is derived from the key properties of time series classification tasks. Additionally, a hardware generation tool is used that creates a complete system from the definition of the NN. This system uses local multi-level RRAM memory as weight and bias storage to avoid external memory access. Exploiting the non-volatility of these devices, such a system can use a power-down mode to save significant energy during the data acquisition process. Detection of atrial fibrillation (AFib) in electrocardiogram (ECG) data is used as an example for evaluation of the framework. It is shown that a reduction of more than 95% of the energy consumption compared to state-of-the-art solutions is achieved.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
In embedded applications that use neural networks (NNs) for classification tasks, it is important to not only minimize the power consumption of the NN calculation, but of the whole system. Optimization approaches for individual parts exist, such as quantization of the NN or analog calculation of arithmetic operations. However, there is no holistic approach for a complete embedded system design that is generic enough in the design process to be used for different applications, but specific in the hardware implementation to waste no energy for a given application. Therefore, we present a novel framework that allows an end-to-end ASIC implementation of a low-power hardware for time series classification using NNs. This includes a neural architecture search (NAS), which optimizes the NN configuration for accuracy and energy efficiency at the same time. This optimization targets a custom designed hardware architecture that is derived from the key properties of time series classification tasks. Additionally, a hardware generation tool is used that creates a complete system from the definition of the NN. This system uses local multi-level RRAM memory as weight and bias storage to avoid external memory access. Exploiting the non-volatility of these devices, such a system can use a power-down mode to save significant energy during the data acquisition process. Detection of atrial fibrillation (AFib) in electrocardiogram (ECG) data is used as an example for evaluation of the framework. It is shown that a reduction of more than 95% of the energy consumption compared to state-of-the-art solutions is achieved.
2099.
Ding, Yadong; Wu, Yu; Huang, Chengyue; Tang, Siliang; Wu, Fei; Yang, Yi; Zhu, Wenwu; Zhuang, Yueting
NAP: Neural Architecture search with Pruning Journal Article
In: Neurocomputing, 2022, ISSN: 0925-2312.
@article{DING2022,
title = {NAP: Neural Architecture search with Pruning},
author = {Yadong Ding and Yu Wu and Chengyue Huang and Siliang Tang and Fei Wu and Yi Yang and Wenwu Zhu and Yueting Zhuang},
url = {https://www.sciencedirect.com/science/article/pii/S0925231221018361},
doi = {https://doi.org/10.1016/j.neucom.2021.12.002},
issn = {0925-2312},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Neurocomputing},
abstract = {There has been continuously increasing attention attracted by Neural Architecture Search (NAS). Due to its computational efficiency, gradient-based NAS methods like DARTS have become the most popular framework for NAS tasks. Nevertheless, as the search iterates, the derived model in previous NAS frameworks becomes dominated by skip-connects, causing the performance downfall. In this work, we present a novel approach to alleviate this issue, named Neural Architecture search with Pruning (NAP). Unlike prior differentiable architecture search works, our approach draws the idea from network pruning. We first train an over-parameterized network, including all candidate operations. Then we propose a criterion to prune the network. Based on a newly designed relaxation of architecture representation, NAP can derive the most potent model by removing trivial and redundant edges from the whole network topology. Experiments show the effectiveness of our proposed approach. Specifically, the model searched by NAP achieves state-of-the-art performances (2.48% test error) on CIFAR-10. We transfer the model to ImageNet and obtains a 25.1% test error with only 5.0M parameters, which is on par with modern NAS methods.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
There has been continuously increasing attention attracted by Neural Architecture Search (NAS). Due to its computational efficiency, gradient-based NAS methods like DARTS have become the most popular framework for NAS tasks. Nevertheless, as the search iterates, the derived model in previous NAS frameworks becomes dominated by skip-connects, causing the performance downfall. In this work, we present a novel approach to alleviate this issue, named Neural Architecture search with Pruning (NAP). Unlike prior differentiable architecture search works, our approach draws the idea from network pruning. We first train an over-parameterized network, including all candidate operations. Then we propose a criterion to prune the network. Based on a newly designed relaxation of architecture representation, NAP can derive the most potent model by removing trivial and redundant edges from the whole network topology. Experiments show the effectiveness of our proposed approach. Specifically, the model searched by NAP achieves state-of-the-art performances (2.48% test error) on CIFAR-10. We transfer the model to ImageNet and obtains a 25.1% test error with only 5.0M parameters, which is on par with modern NAS methods.
2098.
Zhou, Kaixiong; Liu, Zirui; Duan, Keyu; Hu, Xia
Graph Neural Networks: AutoML Book Chapter
In: pp. 371-389, 2022.
@inbook{Zhou2022,
title = {Graph Neural Networks: AutoML},
author = {Kaixiong Zhou and Zirui Liu and Keyu Duan and Xia Hu},
url = {https://link.springer.com/chapter/10.1007/978-981-16-6054-2_17},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
pages = {371-389},
keywords = {},
pubstate = {published},
tppubtype = {inbook}
}
2097.
Rorabaugh, Ariel Keller; Caíno-Lores, Silvina; Johnston, Travis; Taufer, Michela
High frequency accuracy and loss data of random neural networks trained on image datasets Journal Article
In: Data in Brief, vol. 40, pp. 107780, 2022, ISSN: 2352-3409.
@article{RORABAUGH2022107780,
title = {High frequency accuracy and loss data of random neural networks trained on image datasets},
author = {Ariel Keller Rorabaugh and Silvina Caíno-Lores and Travis Johnston and Michela Taufer},
url = {https://www.sciencedirect.com/science/article/pii/S2352340921010544},
doi = {https://doi.org/10.1016/j.dib.2021.107780},
issn = {2352-3409},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Data in Brief},
volume = {40},
pages = {107780},
abstract = {Neural Networks (NNs) are increasingly used across scientific domains to extract knowledge from experimental or computational data. An NN is composed of natural or artificial neurons that serve as simple processing units and are interconnected into a model architecture; it acquires knowledge from the environment through a learning process and stores this knowledge in its connections. The learning process is conducted by training. During NN training, the learning process can be tracked by periodically validating the NN and calculating its fitness. The resulting sequence of fitness values (i.e., validation accuracy or validation loss) is called the NN learning curve. The development of tools for NN design requires knowledge of diverse NNs and their complete learning curves. Generally, only final fully-trained fitness values for highly accurate NNs are made available to the community, hampering efforts to develop tools for NN design and leaving unaddressed aspects such as explaining the generation of an NN and reproducing its learning process. Our dataset fills this gap by fully recording the structure, metadata, and complete learning curves for a wide variety of random NNs throughout their training. Our dataset captures the lifespan of 6000 NNs throughout generation, training, and validation stages. It consists of a suite of 6000 tables, each table representing the lifespan of one NN. We generate each NN with randomized parameter values and train it for 40 epochs on one of three diverse image datasets (i.e., CIFAR-100, FashionMNIST, SVHN). We calculate and record each NN’s fitness with high frequency—every half epoch—to capture the evolution of the training and validation process. As a result, for each NN, we record the generated parameter values describing the structure of that NN, the image dataset on which the NN trained, and all loss and accuracy values for the NN every half epoch. We put our dataset to the service of researchers studying NN performance and its evolution throughout training and validation. Statistical methods can be applied to our dataset to analyze the shape of learning curves in diverse NNs, and the relationship between an NN’s structure and its fitness. Additionally, the structural data and metadata that we record enable the reconstruction and reproducibility of the associated NN.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Neural Networks (NNs) are increasingly used across scientific domains to extract knowledge from experimental or computational data. An NN is composed of natural or artificial neurons that serve as simple processing units and are interconnected into a model architecture; it acquires knowledge from the environment through a learning process and stores this knowledge in its connections. The learning process is conducted by training. During NN training, the learning process can be tracked by periodically validating the NN and calculating its fitness. The resulting sequence of fitness values (i.e., validation accuracy or validation loss) is called the NN learning curve. The development of tools for NN design requires knowledge of diverse NNs and their complete learning curves. Generally, only final fully-trained fitness values for highly accurate NNs are made available to the community, hampering efforts to develop tools for NN design and leaving unaddressed aspects such as explaining the generation of an NN and reproducing its learning process. Our dataset fills this gap by fully recording the structure, metadata, and complete learning curves for a wide variety of random NNs throughout their training. Our dataset captures the lifespan of 6000 NNs throughout generation, training, and validation stages. It consists of a suite of 6000 tables, each table representing the lifespan of one NN. We generate each NN with randomized parameter values and train it for 40 epochs on one of three diverse image datasets (i.e., CIFAR-100, FashionMNIST, SVHN). We calculate and record each NN’s fitness with high frequency—every half epoch—to capture the evolution of the training and validation process. As a result, for each NN, we record the generated parameter values describing the structure of that NN, the image dataset on which the NN trained, and all loss and accuracy values for the NN every half epoch. We put our dataset to the service of researchers studying NN performance and its evolution throughout training and validation. Statistical methods can be applied to our dataset to analyze the shape of learning curves in diverse NNs, and the relationship between an NN’s structure and its fitness. Additionally, the structural data and metadata that we record enable the reconstruction and reproducibility of the associated NN.
2096.
Wang, Xin; Zhang, Ziwei; Zhu, Wenwu
Automated Graph Machine Learning: Approaches, Libraries and Directions Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2201-01288,
title = {Automated Graph Machine Learning: Approaches, Libraries and Directions},
author = {Xin Wang and Ziwei Zhang and Wenwu Zhu},
url = {https://arxiv.org/abs/2201.01288},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2201.01288},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2095.
Rorabaugh, Ariel Keller; Caino-Lores, Silvina; Johnston, Travis; Taufer, Michela
Building High-throughput Neural Architecture Search Workflows via a Decoupled Fitness Prediction Engine Journal Article
In: IEEE Transactions on Parallel and Distributed Systems, pp. 1-1, 2022.
@article{9674227,
title = {Building High-throughput Neural Architecture Search Workflows via a Decoupled Fitness Prediction Engine},
author = {Ariel Keller Rorabaugh and Silvina Caino-Lores and Travis Johnston and Michela Taufer},
url = {https://ieeexplore.ieee.org/abstract/document/9674227},
doi = {10.1109/TPDS.2022.3140681},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {IEEE Transactions on Parallel and Distributed Systems},
pages = {1-1},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2094.
Zhan, Cheng; Zhang, Licheng; Zhao, Xin; Lee, Chang-Chun; Huang, Shujiao
Neural Architecture Search for Inversion Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2201-01772,
title = {Neural Architecture Search for Inversion},
author = {Cheng Zhan and Licheng Zhang and Xin Zhao and Chang-Chun Lee and Shujiao Huang},
url = {https://arxiv.org/abs/2201.01772},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2201.01772},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2093.
Zhou, Jian; Zheng, Lianyu; Wang, Yiwei; Wang, Cheng; Gao, Robert X.
Automated model generation for machinery fault diagnosis based on reinforcement learning and neural architecture search Journal Article
In: IEEE Transactions on Instrumentation and Measurement, pp. 1-1, 2022.
@article{9673794,
title = {Automated model generation for machinery fault diagnosis based on reinforcement learning and neural architecture search},
author = {Jian Zhou and Lianyu Zheng and Yiwei Wang and Cheng Wang and Robert X. Gao},
url = {https://ieeexplore.ieee.org/abstract/document/9673794},
doi = {10.1109/TIM.2022.3141166},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {IEEE Transactions on Instrumentation and Measurement},
pages = {1-1},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2092.
Zhang, Haoyu; Jin, Yaochu; Jin, Yaochu; Hao, Kuangrong
Evolutionary Search for Complete Neural Network Architectures with Partial Weight Sharing Journal Article
In: IEEE Transactions on Evolutionary Computation, pp. 1-1, 2022.
@article{9672175,
title = {Evolutionary Search for Complete Neural Network Architectures with Partial Weight Sharing},
author = {Haoyu Zhang and Yaochu Jin and Yaochu Jin and Kuangrong Hao},
url = {https://ieeexplore.ieee.org/abstract/document/9672175},
doi = {10.1109/TEVC.2022.3140855},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {IEEE Transactions on Evolutionary Computation},
pages = {1-1},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2091.
Yipeng, Lv; Wenbing, Lv; Kaixuan, Han; Wentao, Tao; Ling, Zheng; Shizhuang, Weng; Lingsheng, Huang
In: Food Control, pp. 108819, 2022, ISSN: 0956-7135.
@article{YIPENG2022108819,
title = {Determination of wheat kernels damaged by Fusarium head blight using monochromatic images of effective wavelengths from hyperspectral imaging coupled with an architecture self-search deep network},
author = {Lv Yipeng and Lv Wenbing and Han Kaixuan and Tao Wentao and Zheng Ling and Weng Shizhuang and Huang Lingsheng},
url = {https://www.sciencedirect.com/science/article/pii/S0956713522000123},
doi = {https://doi.org/10.1016/j.foodcont.2022.108819},
issn = {0956-7135},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Food Control},
pages = {108819},
abstract = {Wheat kernels damaged by Fusarium head blight (FHB) lose moisture, protein and starch and carry dangerous toxins. Classification of degree of damage can be helpful to customise the use of wheat kernels, reduce financial losses and ensure grain safety. In this study, hyperspectral imaging (HSI) and deep learning network were explored to determine sound, mildly, moderately and severely damaged wheat kernels. Effective wavelengths (EWs) were selected from the reflectance spectroscopy of HSI images by ReliefF, uninformative variable elimination, random frog and shuffled frog leaping algorithm, and the monochromatic images of different combinations of EWs were adopted to develop the classification models combined with an architecture self-search deep network (ASSDN). ASSDN and the images composed of 941, 876 and 732 nm achieved the best determination with average accuracy of 100% and 98.31% in the training and prediction sets, respectively, outperforming other images or methods. And the average area under the curve of 0.9985 indicated its excellent robustness. Using images of sporadic wavelengths, the computation and operation complexity were evidently reduced, and a simple and custom-built instrument can be easily designed for practical recognition of FHB-damaged wheat kernels. Meanwhile, ASSDN can generate and optimise the high-performance classification network by itself, which is user–friendly and largely expands application potential of deep network.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Wheat kernels damaged by Fusarium head blight (FHB) lose moisture, protein and starch and carry dangerous toxins. Classification of degree of damage can be helpful to customise the use of wheat kernels, reduce financial losses and ensure grain safety. In this study, hyperspectral imaging (HSI) and deep learning network were explored to determine sound, mildly, moderately and severely damaged wheat kernels. Effective wavelengths (EWs) were selected from the reflectance spectroscopy of HSI images by ReliefF, uninformative variable elimination, random frog and shuffled frog leaping algorithm, and the monochromatic images of different combinations of EWs were adopted to develop the classification models combined with an architecture self-search deep network (ASSDN). ASSDN and the images composed of 941, 876 and 732 nm achieved the best determination with average accuracy of 100% and 98.31% in the training and prediction sets, respectively, outperforming other images or methods. And the average area under the curve of 0.9985 indicated its excellent robustness. Using images of sporadic wavelengths, the computation and operation complexity were evidently reduced, and a simple and custom-built instrument can be easily designed for practical recognition of FHB-damaged wheat kernels. Meanwhile, ASSDN can generate and optimise the high-performance classification network by itself, which is user–friendly and largely expands application potential of deep network.
2090.
Wu, Meng-Ting; Lin, Hung-I; Tsai, Chun-Wei
A Training-Free Genetic Neural Architecture Search Inproceedings
In: Proceedings of the 2021 ACM International Conference on Intelligent Computing and Its Emerging Applications, pp. 65–70, Association for Computing Machinery, Jinan, China, 2022, ISBN: 9781450391603.
@inproceedings{10.1145/3491396.3506510,
title = {A Training-Free Genetic Neural Architecture Search},
author = {Meng-Ting Wu and Hung-I Lin and Chun-Wei Tsai},
url = {https://doi.org/10.1145/3491396.3506510},
doi = {10.1145/3491396.3506510},
isbn = {9781450391603},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
booktitle = {Proceedings of the 2021 ACM International Conference on Intelligent Computing and Its Emerging Applications},
pages = {65–70},
publisher = {Association for Computing Machinery},
address = {Jinan, China},
series = {ACM ICEA '21},
abstract = {The so-called neural architecture search (NAS) provides an alternative way to construct a "good neural architecture," which would normally outperform hand-made architectures, for solving complex problems without domain knowledge. However, a critical issue for most of the NAS techniques is in that it is computationally very expensive because several complete/partial training processes are involved in evaluating the goodness of a neural architecture during the process of NAS. To mitigate this problem for evaluating a single neural architecture found by the search algorithm of NAS, we present an efficient NAS in this study, called genetic algorithm and noise immunity for neural architecture search without training (GA-NINASWOT). The genetic algorithm (GA) in the proposed algorithm is used to search for high potential neural architectures while a modified scoring method based on the neural architecture search without training (NASWOT) is used to replace the training process of each neural architecture found by the GA for measuring its quality. To evaluate the performance of GA-NINASWOT, we compared it with several state-of-the-art NAS techniques, which include weight-sharing methods, non-weight-sharing methods, and NASWOT. Simulation results show that GA-NINASWOT outperforms all the other state-of-the-art weight-sharing methods and NASWOT compared in this study in terms of the accuracy and computational time. Moreover, GA-NINASWOT gives a result that is comparable to those found by the non-weight-sharing methods while reducing 99% of the search time.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
The so-called neural architecture search (NAS) provides an alternative way to construct a "good neural architecture," which would normally outperform hand-made architectures, for solving complex problems without domain knowledge. However, a critical issue for most of the NAS techniques is in that it is computationally very expensive because several complete/partial training processes are involved in evaluating the goodness of a neural architecture during the process of NAS. To mitigate this problem for evaluating a single neural architecture found by the search algorithm of NAS, we present an efficient NAS in this study, called genetic algorithm and noise immunity for neural architecture search without training (GA-NINASWOT). The genetic algorithm (GA) in the proposed algorithm is used to search for high potential neural architectures while a modified scoring method based on the neural architecture search without training (NASWOT) is used to replace the training process of each neural architecture found by the GA for measuring its quality. To evaluate the performance of GA-NINASWOT, we compared it with several state-of-the-art NAS techniques, which include weight-sharing methods, non-weight-sharing methods, and NASWOT. Simulation results show that GA-NINASWOT outperforms all the other state-of-the-art weight-sharing methods and NASWOT compared in this study in terms of the accuracy and computational time. Moreover, GA-NINASWOT gives a result that is comparable to those found by the non-weight-sharing methods while reducing 99% of the search time.
2089.
Chen, Yi-Ping; Tsai, Chun-Wei
An Effective Neural Architecture Optimization Algorithm for CNN Based on Search Economics Inproceedings
In: Proceedings of the 2021 ACM International Conference on Intelligent Computing and Its Emerging Applications, pp. 40–45, Association for Computing Machinery, Jinan, China, 2022, ISBN: 9781450391603.
@inproceedings{10.1145/3491396.3506505,
title = {An Effective Neural Architecture Optimization Algorithm for CNN Based on Search Economics},
author = {Yi-Ping Chen and Chun-Wei Tsai},
url = {https://doi.org/10.1145/3491396.3506505},
doi = {10.1145/3491396.3506505},
isbn = {9781450391603},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
booktitle = {Proceedings of the 2021 ACM International Conference on Intelligent Computing and Its Emerging Applications},
pages = {40–45},
publisher = {Association for Computing Machinery},
address = {Jinan, China},
series = {ACM ICEA '21},
abstract = {Developing a good convolutional neural network (CNN) architecture manually by trial and error is typically extremely time-consuming and requires a lot of effort. That is why several recent studies have attempted to develop ways to automatically construct a suitable CNN architecture. In this study, an effective neural architecture search (NAS) algorithm based on a novel metaheuristic algorithm, search economics (SE), is presented for CNN to improve the accuracy of image classification. The basic idea of the proposed algorithm is to use the "expected value" instead of the öbjective value" to evaluate a set of searched solutions, i.e., neural architectures in this case. As such, the searches of the proposed algorithm will trend to high potential regions in the solution space. Simulation results show that the proposed algorithm outperforms genetic algorithm-based NAS algorithm in terms of the accuracy, especially for complex image classification problems.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Developing a good convolutional neural network (CNN) architecture manually by trial and error is typically extremely time-consuming and requires a lot of effort. That is why several recent studies have attempted to develop ways to automatically construct a suitable CNN architecture. In this study, an effective neural architecture search (NAS) algorithm based on a novel metaheuristic algorithm, search economics (SE), is presented for CNN to improve the accuracy of image classification. The basic idea of the proposed algorithm is to use the "expected value" instead of the öbjective value" to evaluate a set of searched solutions, i.e., neural architectures in this case. As such, the searches of the proposed algorithm will trend to high potential regions in the solution space. Simulation results show that the proposed algorithm outperforms genetic algorithm-based NAS algorithm in terms of the accuracy, especially for complex image classification problems.
2088.
Pochelu, Pierrick; Petiton, Serge G.; Conche, Bruno
A Deep Neural Networks Ensemble Workflow from Hyperparameter Search to Inference Leveraging GPU Clusters Inproceedings
In: International Conference on High Performance Computing in Asia-Pacific Region, pp. 61–71, Association for Computing Machinery, Virtual Event, Japan, 2022, ISBN: 9781450384988.
@inproceedings{10.1145/3492805.3492819,
title = {A Deep Neural Networks Ensemble Workflow from Hyperparameter Search to Inference Leveraging GPU Clusters},
author = {Pierrick Pochelu and Serge G. Petiton and Bruno Conche},
url = {https://doi.org/10.1145/3492805.3492819},
doi = {10.1145/3492805.3492819},
isbn = {9781450384988},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
booktitle = {International Conference on High Performance Computing in Asia-Pacific Region},
pages = {61–71},
publisher = {Association for Computing Machinery},
address = {Virtual Event, Japan},
series = {HPCAsia2022},
abstract = {Automated Machine Learning with ensembling (or AutoML with ensembling) seeks to automatically build ensembles of Deep Neural Networks (DNNs) to achieve qualitative predictions. Ensemble of DNNs are well known to avoid over-fitting but they are memory and time consuming approaches. Therefore, an ideal AutoML would produce in one single run time different ensembles regarding accuracy and inference speed. While previous works on AutoML focus to search for the best model to maximize its generalization ability, we rather propose a new AutoML to build a larger library of accurate and diverse individual models to then construct ensembles. First, our extensive benchmarks show asynchronous Hyperband is an efficient and robust way to build a large number of diverse models to combine them. Then, a new ensemble selection method based on a multi-objective greedy algorithm is proposed to generate accurate ensembles by controlling their computing cost. Finally, we propose a novel algorithm to optimize the inference of the DNNs ensemble in a GPU cluster based on allocation optimization. The produced AutoML with ensemble method shows robust results on two datasets using efficiently GPU clusters during both the training phase and the inference phase.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Automated Machine Learning with ensembling (or AutoML with ensembling) seeks to automatically build ensembles of Deep Neural Networks (DNNs) to achieve qualitative predictions. Ensemble of DNNs are well known to avoid over-fitting but they are memory and time consuming approaches. Therefore, an ideal AutoML would produce in one single run time different ensembles regarding accuracy and inference speed. While previous works on AutoML focus to search for the best model to maximize its generalization ability, we rather propose a new AutoML to build a larger library of accurate and diverse individual models to then construct ensembles. First, our extensive benchmarks show asynchronous Hyperband is an efficient and robust way to build a large number of diverse models to combine them. Then, a new ensemble selection method based on a multi-objective greedy algorithm is proposed to generate accurate ensembles by controlling their computing cost. Finally, we propose a novel algorithm to optimize the inference of the DNNs ensemble in a GPU cluster based on allocation optimization. The produced AutoML with ensemble method shows robust results on two datasets using efficiently GPU clusters during both the training phase and the inference phase.
2087.
Durbha, Krishna Srikar; Amuru, Saidhiraj
AutoML Models for Wireless Signals Classification and their effectiveness against Adversarial Attacks Inproceedings
In: 2022 14th International Conference on COMmunication Systems NETworkS (COMSNETS), pp. 265-269, 2022.
@inproceedings{9668448,
title = {AutoML Models for Wireless Signals Classification and their effectiveness against Adversarial Attacks},
author = {Krishna Srikar Durbha and Saidhiraj Amuru},
url = {https://ieeexplore.ieee.org/abstract/document/9668448},
doi = {10.1109/COMSNETS53615.2022.9668448},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
booktitle = {2022 14th International Conference on COMmunication Systems NETworkS (COMSNETS)},
pages = {265-269},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
2086.
Jiang, Chunheng; Pedapati, Tejaswini; Chen, Pin-Yu; Sun, Yizhou; Gao, Jianxi
Neural Capacitance: A New Perspective of Neural Network Selection via Edge Dynamics Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2201-04194,
title = {Neural Capacitance: A New Perspective of Neural Network Selection via Edge Dynamics},
author = {Chunheng Jiang and Tejaswini Pedapati and Pin-Yu Chen and Yizhou Sun and Jianxi Gao},
url = {https://arxiv.org/abs/2201.04194},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2201.04194},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2085.
Cai, Han; Lin, Ji; Han, Song
Chapter 4 - Efficient methods for deep learning Incollection
In: Davies, E. R.; Turk, Matthew A. (Ed.): Advanced Methods and Deep Learning in Computer Vision, pp. 159-190, Academic Press, 2022, ISBN: 978-0-12-822109-9.
@incollection{CAI2022159,
title = {Chapter 4 - Efficient methods for deep learning},
author = {Han Cai and Ji Lin and Song Han},
editor = {E. R. Davies and Matthew A. Turk},
url = {https://www.sciencedirect.com/science/article/pii/B9780128221099000138},
doi = {https://doi.org/10.1016/B978-0-12-822109-9.00013-8},
isbn = {978-0-12-822109-9},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
booktitle = {Advanced Methods and Deep Learning in Computer Vision},
pages = {159-190},
publisher = {Academic Press},
series = {Computer Vision and Pattern Recognition},
abstract = {Deep neural networks (DNNs) have achieved unprecedented success in computer vision. However, their superior performance comes at the considerable cost of computational complexity, which greatly hinders their applications in many resource-constrained devices, such as mobile phones and Internet of Things (IoT) devices. Therefore, methods and techniques that can lift the efficiency bottleneck while preserving the high accuracy of DNNs are in great demand to enable numerous edge AI applications. This chapter provides an overview of efficient deep learning methods. We start from introducing popular model compression methods, including pruning, factorization, and quantization. We then describe compact model design techniques including efficient convolution layers and representative efficient CNN architectures. Finally, to reduce the large design cost of these manual solutions, we discuss the AutoML framework for each of them, such as neural architecture search (NAS) and automated pruning and quantization.},
keywords = {},
pubstate = {published},
tppubtype = {incollection}
}
Deep neural networks (DNNs) have achieved unprecedented success in computer vision. However, their superior performance comes at the considerable cost of computational complexity, which greatly hinders their applications in many resource-constrained devices, such as mobile phones and Internet of Things (IoT) devices. Therefore, methods and techniques that can lift the efficiency bottleneck while preserving the high accuracy of DNNs are in great demand to enable numerous edge AI applications. This chapter provides an overview of efficient deep learning methods. We start from introducing popular model compression methods, including pruning, factorization, and quantization. We then describe compact model design techniques including efficient convolution layers and representative efficient CNN architectures. Finally, to reduce the large design cost of these manual solutions, we discuss the AutoML framework for each of them, such as neural architecture search (NAS) and automated pruning and quantization.
2084.
Tan, Huixin; Lai, Jiewei; Liu, Yunbi; Song, Yuzhang; Wang, Jinliang; Chen, Mingyang; Yan, Yong; Zhong, Liming; Feng, Qianjin; Yang, Wei
Neural architecture search for real-time quality assessment of wearable multi-lead ECG on mobile devices Journal Article
In: Biomedical Signal Processing and Control, vol. 74, pp. 103495, 2022, ISSN: 1746-8094.
@article{TAN2022103495,
title = {Neural architecture search for real-time quality assessment of wearable multi-lead ECG on mobile devices},
author = {Huixin Tan and Jiewei Lai and Yunbi Liu and Yuzhang Song and Jinliang Wang and Mingyang Chen and Yong Yan and Liming Zhong and Qianjin Feng and Wei Yang},
url = {https://www.sciencedirect.com/science/article/pii/S1746809422000179},
doi = {https://doi.org/10.1016/j.bspc.2022.103495},
issn = {1746-8094},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Biomedical Signal Processing and Control},
volume = {74},
pages = {103495},
abstract = {Users need to pay for the medical services of the wearable ECG signals. The complicated interferences of wearable ECG may cause invalid medical diagnosis. To save medical expenses and reduce the workload of cardiologists, unacceptable wearable ECG should be filtered out on mobile devices in real time. Therefore, a robust and lightweight quality assessment model is essential. Existing convolutional neural network models are more robust than hand-crafted features based models, but most are too cumbersome to run on mobile devices. In this study, we utilize neural architecture search algorithm to search a network with a few parameters and low latency automatically for real-time quality assessment of wearable ECG data. To reach this goal, we construct a searching space with lightweight network blocks and add the hardware latency loss. We also propose a novel and effective strategy, any-lead, to assess the quality of all leads using a single model. We evaluated our method on a large-scale (10,709 signals) wearable 12-lead ECG dataset and a public dataset named Physionet Cinc Challenge 2011. The parameters and FLOPs of the architecture we searched were about 66.76 K and 36.44 M. Our model achieved excellent performance on the aforementioned datasets, with AUC of 98.32% and 97.64%, F1 scores of 94.36% and 93.52%, MCC of 84.74% and 83.22%, respectively, and with the inference time on an Android emulator of about 78 ms. Extensive experimental results demonstrate the effectiveness of our method in assessing the quality of all leads of wearable ECG data on mobile devices in real time.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Users need to pay for the medical services of the wearable ECG signals. The complicated interferences of wearable ECG may cause invalid medical diagnosis. To save medical expenses and reduce the workload of cardiologists, unacceptable wearable ECG should be filtered out on mobile devices in real time. Therefore, a robust and lightweight quality assessment model is essential. Existing convolutional neural network models are more robust than hand-crafted features based models, but most are too cumbersome to run on mobile devices. In this study, we utilize neural architecture search algorithm to search a network with a few parameters and low latency automatically for real-time quality assessment of wearable ECG data. To reach this goal, we construct a searching space with lightweight network blocks and add the hardware latency loss. We also propose a novel and effective strategy, any-lead, to assess the quality of all leads using a single model. We evaluated our method on a large-scale (10,709 signals) wearable 12-lead ECG dataset and a public dataset named Physionet Cinc Challenge 2011. The parameters and FLOPs of the architecture we searched were about 66.76 K and 36.44 M. Our model achieved excellent performance on the aforementioned datasets, with AUC of 98.32% and 97.64%, F1 scores of 94.36% and 93.52%, MCC of 84.74% and 83.22%, respectively, and with the inference time on an Android emulator of about 78 ms. Extensive experimental results demonstrate the effectiveness of our method in assessing the quality of all leads of wearable ECG data on mobile devices in real time.
2083.
Traoré, Kalifou René; Camero, Andrés; Zhu, Xiao Xiang
Landscape of Neural Architecture Search across sensors: how much do they differ ? Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2201-06321,
title = {Landscape of Neural Architecture Search across sensors: how much do they differ ?},
author = {Kalifou René Traoré and Andrés Camero and Xiao Xiang Zhu},
url = {https://arxiv.org/abs/2201.06321},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2201.06321},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2082.
Fedorov, Igor; Navarro, Ramon Matas; Tann, Hokchhay; Zhou, Chuteng; Mattina, Matthew; Whatmough, Paul N.
UDC: Unified DNAS for Compressible TinyML Models Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2201-05842,
title = {UDC: Unified DNAS for Compressible TinyML Models},
author = {Igor Fedorov and Ramon Matas Navarro and Hokchhay Tann and Chuteng Zhou and Matthew Mattina and Paul N. Whatmough},
url = {https://arxiv.org/abs/2201.05842},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2201.05842},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2081.
Liu, Xin; Zhao, Jianwei; Li, Jie; Cao, Bin; Lv, Zhihan
Federated Neural Architecture Search for Medical Data Security Journal Article
In: IEEE Transactions on Industrial Informatics, pp. 1-1, 2022.
@article{9684972,
title = {Federated Neural Architecture Search for Medical Data Security},
author = {Xin Liu and Jianwei Zhao and Jie Li and Bin Cao and Zhihan Lv},
url = {https://ieeexplore.ieee.org/abstract/document/9684972},
doi = {10.1109/TII.2022.3144016},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {IEEE Transactions on Industrial Informatics},
pages = {1-1},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2080.
Guo, Qingbei; Wu, Xiao-Jun; Kittler, Josef; Feng, Zhiquan
Differentiable neural architecture learning for efficient neural networks Journal Article
In: Pattern Recognition, vol. 126, pp. 108448, 2022, ISSN: 0031-3203.
@article{GUO2022108448,
title = {Differentiable neural architecture learning for efficient neural networks},
author = {Qingbei Guo and Xiao-Jun Wu and Josef Kittler and Zhiquan Feng},
url = {https://www.sciencedirect.com/science/article/pii/S0031320321006245},
doi = {https://doi.org/10.1016/j.patcog.2021.108448},
issn = {0031-3203},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Pattern Recognition},
volume = {126},
pages = {108448},
abstract = {Efficient neural networks has received ever-increasing attention with the evolution of convolutional neural networks (CNNs), especially involving their deployment on embedded and mobile platforms. One of the biggest problems to obtaining such efficient neural networks is efficiency, even recent differentiable neural architecture search (DNAS) requires to sample a small number of candidate neural architectures for the selection of the optimal neural architecture. To address this computational efficiency issue, we introduce a novel architecture parameterization based on scaled sigmoid function, and propose a general Differentiable Neural Architecture Learning (DNAL) method to obtain efficient neural networks without the need to evaluate candidate neural networks. Specifically, for stochastic supernets as well as conventional CNNs, we build a new channel-wise module layer with the architecture components controlled by a scaled sigmoid function. We train these neural network models from scratch. The network optimization is decoupled into the weight optimization and the architecture optimization, which avoids the interaction between the two types of parameters and alleviates the vanishing gradient problem. We address the non-convex optimization problem of efficient neural networks by the continuous scaled sigmoid method instead of the common softmax method. Extensive experiments demonstrate our DNAL method delivers superior performance in terms of efficiency, and adapts to conventional CNNs (e.g., VGG16 and ResNet50), lightweight CNNs (e.g., MobileNetV2) and stochastic supernets (e.g., ProxylessNAS). The optimal neural networks learned by DNAL surpass those produced by the state-of-the-art methods on the benchmark CIFAR-10 and ImageNet-1K dataset in accuracy, model size and computational complexity. Our source code is available at https://github.com/QingbeiGuo/DNAL.git.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Efficient neural networks has received ever-increasing attention with the evolution of convolutional neural networks (CNNs), especially involving their deployment on embedded and mobile platforms. One of the biggest problems to obtaining such efficient neural networks is efficiency, even recent differentiable neural architecture search (DNAS) requires to sample a small number of candidate neural architectures for the selection of the optimal neural architecture. To address this computational efficiency issue, we introduce a novel architecture parameterization based on scaled sigmoid function, and propose a general Differentiable Neural Architecture Learning (DNAL) method to obtain efficient neural networks without the need to evaluate candidate neural networks. Specifically, for stochastic supernets as well as conventional CNNs, we build a new channel-wise module layer with the architecture components controlled by a scaled sigmoid function. We train these neural network models from scratch. The network optimization is decoupled into the weight optimization and the architecture optimization, which avoids the interaction between the two types of parameters and alleviates the vanishing gradient problem. We address the non-convex optimization problem of efficient neural networks by the continuous scaled sigmoid method instead of the common softmax method. Extensive experiments demonstrate our DNAL method delivers superior performance in terms of efficiency, and adapts to conventional CNNs (e.g., VGG16 and ResNet50), lightweight CNNs (e.g., MobileNetV2) and stochastic supernets (e.g., ProxylessNAS). The optimal neural networks learned by DNAL surpass those produced by the state-of-the-art methods on the benchmark CIFAR-10 and ImageNet-1K dataset in accuracy, model size and computational complexity. Our source code is available at https://github.com/QingbeiGuo/DNAL.git.
2079.
Lin, Qiuzhen; Fang, Zhixiong; Chen, Yi; Tan, Kay Chen; Li, Yun
Evolutionary Architectural Search for Generative Adversarial Networks Journal Article
In: IEEE Transactions on Emerging Topics in Computational Intelligence, pp. 1-12, 2022.
@article{9686068,
title = {Evolutionary Architectural Search for Generative Adversarial Networks},
author = {Qiuzhen Lin and Zhixiong Fang and Yi Chen and Kay Chen Tan and Yun Li},
url = {https://ieeexplore.ieee.org/abstract/document/9686068},
doi = {10.1109/TETCI.2021.3137377},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {IEEE Transactions on Emerging Topics in Computational Intelligence},
pages = {1-12},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2078.
Wang, Yabin; Ma, Zhiheng; Wei, Xing; Zheng, Shuai; Wang, Yaowei; Hong, Xiaopeng
ECCNAS: Efficient Crowd Counting Neural Architecture Search Journal Article
In: ACM Trans. Multimedia Comput. Commun. Appl., vol. 18, no. 1s, 2022, ISSN: 1551-6857.
@article{10.1145/3465455,
title = {ECCNAS: Efficient Crowd Counting Neural Architecture Search},
author = {Yabin Wang and Zhiheng Ma and Xing Wei and Shuai Zheng and Yaowei Wang and Xiaopeng Hong},
url = {https://doi.org/10.1145/3465455},
doi = {10.1145/3465455},
issn = {1551-6857},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {ACM Trans. Multimedia Comput. Commun. Appl.},
volume = {18},
number = {1s},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
abstract = {Recent solutions to crowd counting problems have already achieved promising performance across various benchmarks. However, applying these approaches to real-world applications is still challenging, because they are computation intensive and lack the flexibility to meet various resource budgets. In this article, we propose an efficient crowd counting neural architecture search (ECCNAS) framework to search efficient crowd counting network structures, which can fill this research gap. A novel search from pre-trained strategy enables our cross-task NAS to explore the significantly large and flexible search space with less search time and get more proper network structures. Moreover, our well-designed search space can intrinsically provide candidate neural network structures with high performance and efficiency. In order to search network structures according to hardwares with different computational performance, we develop a novel latency cost estimation algorithm in our ECCNAS. Experiments show our searched models get an excellent trade-off between computational complexity and accuracy and have the potential to deploy in practical scenarios with various resource budgets. We reduce the computational cost, in terms of multiply-and-accumulate (MACs), by up to 96% with comparable accuracy. And we further designed experiments to validate the efficiency and the stability improvement of our proposed search from pre-trained strategy.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Recent solutions to crowd counting problems have already achieved promising performance across various benchmarks. However, applying these approaches to real-world applications is still challenging, because they are computation intensive and lack the flexibility to meet various resource budgets. In this article, we propose an efficient crowd counting neural architecture search (ECCNAS) framework to search efficient crowd counting network structures, which can fill this research gap. A novel search from pre-trained strategy enables our cross-task NAS to explore the significantly large and flexible search space with less search time and get more proper network structures. Moreover, our well-designed search space can intrinsically provide candidate neural network structures with high performance and efficiency. In order to search network structures according to hardwares with different computational performance, we develop a novel latency cost estimation algorithm in our ECCNAS. Experiments show our searched models get an excellent trade-off between computational complexity and accuracy and have the potential to deploy in practical scenarios with various resource budgets. We reduce the computational cost, in terms of multiply-and-accumulate (MACs), by up to 96% with comparable accuracy. And we further designed experiments to validate the efficiency and the stability improvement of our proposed search from pre-trained strategy.
2077.
Kim, Youngeun; Li, Yuhang; Park, Hyoungseob; Venkatesha, Yeshwanth; Panda, Priyadarshini
Neural Architecture Search for Spiking Neural Networks Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2201-10355,
title = {Neural Architecture Search for Spiking Neural Networks},
author = {Youngeun Kim and Yuhang Li and Hyoungseob Park and Yeshwanth Venkatesha and Priyadarshini Panda},
url = {https://arxiv.org/abs/2201.10355},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2201.10355},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2076.
Rho, Daniel; Park, Jinhyeok; Ko, Jong Hwan
NAS-VAD: Neural Architecture Search for Voice Activity Detection Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2201-09032,
title = {NAS-VAD: Neural Architecture Search for Voice Activity Detection},
author = {Daniel Rho and Jinhyeok Park and Jong Hwan Ko},
url = {https://arxiv.org/abs/2201.09032},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2201.09032},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2075.
Shu, Yao; Dai, Zhongxiang; Wu, Zhaoxuan; Low, Bryan Kian Hsiang
Unifying and Boosting Gradient-Based Training-Free Neural Architecture Search Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2201-09785,
title = {Unifying and Boosting Gradient-Based Training-Free Neural Architecture Search},
author = {Yao Shu and Zhongxiang Dai and Zhaoxuan Wu and Bryan Kian Hsiang Low},
url = {https://arxiv.org/abs/2201.09785},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2201.09785},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2074.
Chen, Mingzhe; Xiao, Xi; Zhang, Bin; Liu, Xinyu; Lu, Runiu
Neural Architecture Searching for Facial Attributes-based Depression Recognition Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2201-09799,
title = {Neural Architecture Searching for Facial Attributes-based Depression Recognition},
author = {Mingzhe Chen and Xi Xiao and Bin Zhang and Xinyu Liu and Runiu Lu},
url = {https://arxiv.org/abs/2201.09799},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2201.09799},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2073.
Mao, Pengli; Lin, Yan; Xue, Song; Zhang, Baochang
Remaining Useful Life Estimation of Aircraft Engines Using Differentiable Architecture Search Journal Article
In: Mathematics, vol. 10, no. 3, 2022, ISSN: 2227-7390.
@article{math10030352,
title = {Remaining Useful Life Estimation of Aircraft Engines Using Differentiable Architecture Search},
author = {Pengli Mao and Yan Lin and Song Xue and Baochang Zhang},
url = {https://www.mdpi.com/2227-7390/10/3/352},
doi = {10.3390/math10030352},
issn = {2227-7390},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Mathematics},
volume = {10},
number = {3},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2072.
Zhang, Xiaofan; Zhou, Zongwei; Chen, Deming; Wang, Yu Emma
AutoDistill: an End-to-End Framework to Explore and Distill Hardware-Efficient Language Models Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2201-08539,
title = {AutoDistill: an End-to-End Framework to Explore and Distill Hardware-Efficient Language Models},
author = {Xiaofan Zhang and Zongwei Zhou and Deming Chen and Yu Emma Wang},
url = {https://arxiv.org/abs/2201.08539},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2201.08539},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2071.
Litany, Or; Maron, Haggai; Acuna, David; Kautz, Jan; Chechik, Gal; Fidler, Sanja
Federated Learning with Heterogeneous Architectures using Graph HyperNetworks Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2201-08459,
title = {Federated Learning with Heterogeneous Architectures using Graph HyperNetworks},
author = {Or Litany and Haggai Maron and David Acuna and Jan Kautz and Gal Chechik and Sanja Fidler},
url = {https://arxiv.org/abs/2201.08459},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2201.08459},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2070.
Hong, Zhenhou; Wang, Jianzong; Qu, Xiaoyang; Zhao, Chendong; Liu, Jie; Xiao, Jing
Neural Architecture Search as Self-assessor in Semi-supervised Learning Inproceedings
In: Liao, Xiangke; Zhao, Wei; Chen, Enhong; Xiao, Nong; Wang, Li; Gao, Yang; Shi, Yinghuan; Wang, Changdong; Huang, Dan (Ed.): Big Data, pp. 95–107, Springer Singapore, Singapore, 2022, ISBN: 978-981-16-9709-8.
@inproceedings{10.1007/978-981-16-9709-8_7,
title = {Neural Architecture Search as Self-assessor in Semi-supervised Learning},
author = {Zhenhou Hong and Jianzong Wang and Xiaoyang Qu and Chendong Zhao and Jie Liu and Jing Xiao},
editor = {Xiangke Liao and Wei Zhao and Enhong Chen and Nong Xiao and Li Wang and Yang Gao and Yinghuan Shi and Changdong Wang and Dan Huang},
url = {https://link.springer.com/chapter/10.1007/978-981-16-9709-8_7},
isbn = {978-981-16-9709-8},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
booktitle = {Big Data},
pages = {95--107},
publisher = {Springer Singapore},
address = {Singapore},
abstract = {Neural Architecture Search (NAS) forms powerful automatic learning, which has helped achieve remarkable performance in several applications in recent years. Previous research focused on NAS in standard supervised learning to explore its performance, requiring labeled data. In this paper, our goal is to examine the implementation of NAS with large amounts of unlabeled data. We propose the NAS as a self-assessor, called NAS-SA, by adding the consistency method and prior knowledge. We design an adaptive search strategy, a balanced search space, and a multi-object optimization to generate a robust and efficient small model in NAS-SA. The image and text classification tasks proved that our NAS-SA method had achieved the best performance.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Neural Architecture Search (NAS) forms powerful automatic learning, which has helped achieve remarkable performance in several applications in recent years. Previous research focused on NAS in standard supervised learning to explore its performance, requiring labeled data. In this paper, our goal is to examine the implementation of NAS with large amounts of unlabeled data. We propose the NAS as a self-assessor, called NAS-SA, by adding the consistency method and prior knowledge. We design an adaptive search strategy, a balanced search space, and a multi-object optimization to generate a robust and efficient small model in NAS-SA. The image and text classification tasks proved that our NAS-SA method had achieved the best performance.
2069.
Gao, Feng; Song, Bin; Wang, Dan; Qin, Hao
MR-DARTS: Restricted connectivity differentiable architecture search in multi-path search space Journal Article
In: Neurocomputing, vol. 482, pp. 27-39, 2022, ISSN: 0925-2312.
@article{GAO202227,
title = {MR-DARTS: Restricted connectivity differentiable architecture search in multi-path search space},
author = {Feng Gao and Bin Song and Dan Wang and Hao Qin},
url = {https://www.sciencedirect.com/science/article/pii/S0925231222001047},
doi = {https://doi.org/10.1016/j.neucom.2022.01.080},
issn = {0925-2312},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Neurocomputing},
volume = {482},
pages = {27-39},
abstract = {Differentiable search methods can be used to find effective network architectures fast. However, these approaches are accompanied by low accuracy when evaluating a searched architecture, especially evaluating a searched architecture after transferring it to a different dataset. Two reasons can explain this phenomenon. The one is that the networks composed of cells have the depth gap in their structures between the search and evaluate stage. Another is that cells have insufficient ability to extract diverse features. This paper presents the Multi-path Restricted DARTS method to address these critical problems, using a multi-path search space and a restricted connectivity algorithm to perform a more exact search with limited resources. Restricted connectivity algorithm deepens the cells’ structure and makes cells more suitable for deep networks to bridge the depth gap. Multi-path search space enables cells to extract and fuse different-scales features to improve the representation capacity of a network. Our approach achieves state-of-the-art performance on CIFAR10 and CIFAR100 with the smallest parameters (only 2.5 M), demonstrating strong transfer learning ability in complex datasets.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Differentiable search methods can be used to find effective network architectures fast. However, these approaches are accompanied by low accuracy when evaluating a searched architecture, especially evaluating a searched architecture after transferring it to a different dataset. Two reasons can explain this phenomenon. The one is that the networks composed of cells have the depth gap in their structures between the search and evaluate stage. Another is that cells have insufficient ability to extract diverse features. This paper presents the Multi-path Restricted DARTS method to address these critical problems, using a multi-path search space and a restricted connectivity algorithm to perform a more exact search with limited resources. Restricted connectivity algorithm deepens the cells’ structure and makes cells more suitable for deep networks to bridge the depth gap. Multi-path search space enables cells to extract and fuse different-scales features to improve the representation capacity of a network. Our approach achieves state-of-the-art performance on CIFAR10 and CIFAR100 with the smallest parameters (only 2.5 M), demonstrating strong transfer learning ability in complex datasets.
2068.
Ahn, Saehyun; Chang, Jung-Woo; Yoon, Hyeon-Seok; Kang, Suk-Ju
TouchNAS: Efficient Touch Detection Model Design Methodology for Resource-Constrained Devices Journal Article
In: IEEE Sensors Journal, pp. 1-1, 2022.
@article{9695433,
title = {TouchNAS: Efficient Touch Detection Model Design Methodology for Resource-Constrained Devices},
author = {Saehyun Ahn and Jung-Woo Chang and Hyeon-Seok Yoon and Suk-Ju Kang},
url = {https://ieeexplore.ieee.org/abstract/document/9695433},
doi = {10.1109/JSEN.2022.3147469},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {IEEE Sensors Journal},
pages = {1-1},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2067.
Hassantabar, Shayan; Dai, Xiaoliang; Jha, Niraj K.
CURIOUS: Efficient Neural Architecture Search Based on a Performance Predictor and Evolutionary Search Journal Article
In: IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, pp. 1-1, 2022.
@article{9698855,
title = {CURIOUS: Efficient Neural Architecture Search Based on a Performance Predictor and Evolutionary Search},
author = {Shayan Hassantabar and Xiaoliang Dai and Niraj K. Jha},
url = {https://ieeexplore.ieee.org/abstract/document/9698855},
doi = {10.1109/TCAD.2022.3148202},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems},
pages = {1-1},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2066.
Li, Wei; Wen, Shiping; Shi, Kaibo; Yang, Yin; Huang, Tingwen
Neural Architecture Search with a Lightweight Transformer for Text-to-Image Synthesis Journal Article
In: IEEE Transactions on Network Science and Engineering, pp. 1-1, 2022.
@article{9699403,
title = {Neural Architecture Search with a Lightweight Transformer for Text-to-Image Synthesis},
author = {Wei Li and Shiping Wen and Kaibo Shi and Yin Yang and Tingwen Huang},
url = {https://ieeexplore.ieee.org/abstract/document/9699403},
doi = {10.1109/TNSE.2022.3147787},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {IEEE Transactions on Network Science and Engineering},
pages = {1-1},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2065.
Pauletto, Loïc; Amini, Massih-Reza; Winckler, Nicolas
Self Semi Supervised Neural Architecture Search for Semantic Segmentation Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2201-12646,
title = {Self Semi Supervised Neural Architecture Search for Semantic Segmentation},
author = {Loïc Pauletto and Massih-Reza Amini and Nicolas Winckler},
url = {https://arxiv.org/abs/2201.12646},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2201.12646},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2064.
Xu, Dongkuan; Mukherjee, Subhabrata; Liu, Xiaodong; Dey, Debadeepta; Wang, Wenhui; Zhang, Xiang; Awadallah, Ahmed Hassan; Gao, Jianfeng
AutoDistil: Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2201-12507,
title = {AutoDistil: Few-shot Task-agnostic Neural Architecture Search for Distilling Large Language Models},
author = {Dongkuan Xu and Subhabrata Mukherjee and Xiaodong Liu and Debadeepta Dey and Wenhui Wang and Xiang Zhang and Ahmed Hassan Awadallah and Jianfeng Gao},
url = {https://arxiv.org/abs/2201.12507},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2201.12507},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2063.
Wei, Hui; Lee, Feifei; Hu, Chunyan; Chen, Qiu
MOO-DNAS: Efficient Neural Network Design via Differentiable Architecture Search Based on Multi-Objective Optimization Journal Article
In: IEEE Access, vol. 10, pp. 14195-14207, 2022.
@article{9698215,
title = {MOO-DNAS: Efficient Neural Network Design via Differentiable Architecture Search Based on Multi-Objective Optimization},
author = {Hui Wei and Feifei Lee and Chunyan Hu and Qiu Chen},
url = {https://ieeexplore.ieee.org/abstract/document/9698215},
doi = {10.1109/ACCESS.2022.3148323},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {IEEE Access},
volume = {10},
pages = {14195-14207},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2062.
Guo, Bicheng; He, Shibo; Chen, Tao; Chen, Jiming; Ye, Peng
Neural Architecture Ranker Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2201-12725,
title = {Neural Architecture Ranker},
author = {Bicheng Guo and Shibo He and Tao Chen and Jiming Chen and Peng Ye},
url = {https://arxiv.org/abs/2201.12725},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2201.12725},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2061.
Na, Byunggook; Mok, Jisoo; Park, Seongsik; Lee, Dongjin; Choe, Hyeokjun; Yoon, Sungroh
AutoSNN: Towards Energy-Efficient Spiking Neural Networks Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2201-12738,
title = {AutoSNN: Towards Energy-Efficient Spiking Neural Networks},
author = {Byunggook Na and Jisoo Mok and Seongsik Park and Dongjin Lee and Hyeokjun Choe and Sungroh Yoon},
url = {https://arxiv.org/abs/2201.12738},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2201.12738},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2060.
Rguibi, Zakaria; Hajami, Abdelmajid; Dya, Zitouni
Äutomatic Searching of Deep Neural Networks for Medical Imaging Diagnostic Inproceedings
In: Saidi, Rajaa; Bhiri, Brahim El; Maleh, Yassine; Mosallam, Ayman; Essaaidi, Mohammed (Ed.): Ädvanced Technologies for Humanity", pp. 129–140, Springer International Publishing, Cham, 2022, ISBN: 978-3-030-94188-8.
@inproceedings{10.1007/978-3-030-94188-8_13,
title = {Äutomatic Searching of Deep Neural Networks for Medical Imaging Diagnostic},
author = {Zakaria Rguibi and Abdelmajid Hajami and Zitouni Dya},
editor = {Rajaa Saidi and Brahim El Bhiri and Yassine Maleh and Ayman Mosallam and Mohammed Essaaidi},
isbn = {978-3-030-94188-8},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
booktitle = {Ädvanced Technologies for Humanity"},
pages = {129--140},
publisher = {Springer International Publishing},
address = {Cham},
abstract = {Medical imaging diagnosis is the most assisted method to help physicians diagnose patient diseases using different imaging test modalities. But Imbalanced data is one of the biggest challenges in the field of medical imaging. To advance this field, this work proposes a framework that can be used to find the optimal DNN architectures for a database with the challenge of imbalanced datasets. In our paper, we present a framework for automatic deep neural network search for medical imaging diagnosis.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Medical imaging diagnosis is the most assisted method to help physicians diagnose patient diseases using different imaging test modalities. But Imbalanced data is one of the biggest challenges in the field of medical imaging. To advance this field, this work proposes a framework that can be used to find the optimal DNN architectures for a database with the challenge of imbalanced datasets. In our paper, we present a framework for automatic deep neural network search for medical imaging diagnosis.
2059.
Cardoso, Rui P.; Hart, Emma; Kurka, David Burth; Pitt, Jeremy V.
Augmenting Novelty Search with a Surrogate Model to Engineer Meta-Diversity in Ensembles of Classifiers Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2201-12896,
title = {Augmenting Novelty Search with a Surrogate Model to Engineer Meta-Diversity in Ensembles of Classifiers},
author = {Rui P. Cardoso and Emma Hart and David Burth Kurka and Jeremy V. Pitt},
url = {https://arxiv.org/abs/2201.12896},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2201.12896},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}