Maintained by Difan Deng and Marius Lindauer.
The following list considers papers related to neural architecture search. It is by no means complete. If you miss a paper on the list, please let us know.
Please note that although NAS methods steadily improve, the quality of empirical evaluations in this field are still lagging behind compared to other areas in machine learning, AI and optimization. We would therefore like to share some best practices for empirical evaluations of NAS methods, which we believe will facilitate sustained and measurable progress in the field. If you are interested in a teaser, please read our blog post or directly jump to our checklist.
Transformers have gained increasing popularity in different domains. For a comprehensive list of papers focusing on Neural Architecture Search for Transformer-Based spaces, the awesome-transformer-search repo is all you need.
5555
1515.
Nguyen, N.; Chang, M. J.
CSNAS: Contrastive Self-supervised Learning Neural Architecture Search via Sequential Model-Based Optimization Journal Article
In: IEEE Transactions on Artificial Intelligence, 1 (01), pp. 1-1, 5555.
@article{9594655,
title = {CSNAS: Contrastive Self-supervised Learning Neural Architecture Search via Sequential Model-Based Optimization},
author = {N. Nguyen and M. J. Chang},
url = {https://www.computer.org/csdl/journal/ai/5555/01/09594655/1y5ZonWBo2c},
doi = {10.1109/TAI.2021.3121663},
year = {5555},
date = {5555-10-01},
urldate = {5555-10-01},
journal = {IEEE Transactions on Artificial Intelligence},
volume = {1},
number = {01},
pages = {1-1},
publisher = {IEEE Computer Society},
address = {Los Alamitos, CA, USA},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2022
1514.
Li, Xudong; Zheng, Jianhua; Li, Mingtao; Ma, Wenzhen; Hu, Yang
One-shot neural architecture search for fault diagnosis using vibration signals Journal Article
In: Expert Systems with Applications, 190 , pp. 116027, 2022, ISSN: 0957-4174.
@article{LI2022116027,
title = {One-shot neural architecture search for fault diagnosis using vibration signals},
author = {Xudong Li and Jianhua Zheng and Mingtao Li and Wenzhen Ma and Yang Hu},
url = {https://www.sciencedirect.com/science/article/pii/S0957417421013737},
doi = {https://doi.org/10.1016/j.eswa.2021.116027},
issn = {0957-4174},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Expert Systems with Applications},
volume = {190},
pages = {116027},
abstract = {Machine learning method has been widely applied in industrial fault diagnosis, especially the deep learning method. In the field of industrial fault diagnosis, deep learning is mostly used to extract features of vibration signals to achieve end-to-end fault diagnosis systems. Due to the complexity and variety of actual industrial datasets, some deep learning models are designed to be complicated. However, designing neural network architectures requires rich professional knowledge, experience, and a large number of experiments, increasing the difficulty of developing deep learning models. Fortunately, Neural Architecture Search (NAS), a branch of Automated Machine Learning (AutoML), is developing rapidly. Given a search space, NAS can search for networks that perform better than manually designed. In this paper, a one-shot NAS method for fault diagnosis is proposed. The one-shot model is a supernet that contains all candidate networks in a given search space. The supernet is trained to evaluate the actual performance of candidate networks by measuring the difference between its output probability and the true labels. According to the prediction of supernet, the networks with excellent performance can be searched, using some common search methods such as random search or evolutionary algorithm. Finally, the searched network is trained by reusing the weights of the supernet. To evaluate the proposed method, two search spaces are designed, ResNet and Inception search spaces, to search on PHM 2009 Data Challenge gearbox dataset. The state-of-the-art results are obtained, and accuracies of searched ResNet-A and Inception-A are 84.11% and 83.81%, which are 3.29% and 10.88% higher than Reinforcement Learning based NAS.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Machine learning method has been widely applied in industrial fault diagnosis, especially the deep learning method. In the field of industrial fault diagnosis, deep learning is mostly used to extract features of vibration signals to achieve end-to-end fault diagnosis systems. Due to the complexity and variety of actual industrial datasets, some deep learning models are designed to be complicated. However, designing neural network architectures requires rich professional knowledge, experience, and a large number of experiments, increasing the difficulty of developing deep learning models. Fortunately, Neural Architecture Search (NAS), a branch of Automated Machine Learning (AutoML), is developing rapidly. Given a search space, NAS can search for networks that perform better than manually designed. In this paper, a one-shot NAS method for fault diagnosis is proposed. The one-shot model is a supernet that contains all candidate networks in a given search space. The supernet is trained to evaluate the actual performance of candidate networks by measuring the difference between its output probability and the true labels. According to the prediction of supernet, the networks with excellent performance can be searched, using some common search methods such as random search or evolutionary algorithm. Finally, the searched network is trained by reusing the weights of the supernet. To evaluate the proposed method, two search spaces are designed, ResNet and Inception search spaces, to search on PHM 2009 Data Challenge gearbox dataset. The state-of-the-art results are obtained, and accuracies of searched ResNet-A and Inception-A are 84.11% and 83.81%, which are 3.29% and 10.88% higher than Reinforcement Learning based NAS.
1513.
Grahn, Lea; Rachner, Jonas; Göppert, Amon; Saeed, Sazvan; Schmitt, Robert H.
Framework for Potential Analysis by Approximating Line-Less Assembly Systems with AutoML Inproceedings
In: Ändersen, Ann-Louise; Andersen, Rasmus; Brunoe, Thomas Ditlev; Larsen, Maria Stoettrup Schioenning; Nielsen, Kjeld; Napoleone, Alessia; Kjeldgaard, Stefan" (Ed.): Towards Sustainable Customization: Bridging Smart Products and Manufacturing Systems, pp. 423–430, Springer International Publishing, Cham, 2022, ISBN: 978-3-030-90700-6.
@inproceedings{10.1007/978-3-030-90700-6_48,
title = {Framework for Potential Analysis by Approximating Line-Less Assembly Systems with AutoML},
author = {Lea Grahn and Jonas Rachner and Amon Göppert and Sazvan Saeed and Robert H. Schmitt},
editor = {Ann-Louise Ändersen and Rasmus Andersen and Thomas Ditlev Brunoe and Maria Stoettrup Schioenning Larsen and Kjeld Nielsen and Alessia Napoleone and Stefan" Kjeldgaard},
url = {https://link.springer.com/chapter/10.1007/978-3-030-90700-6_48},
isbn = {978-3-030-90700-6},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
booktitle = {Towards Sustainable Customization: Bridging Smart Products and Manufacturing Systems},
pages = {423--430},
publisher = {Springer International Publishing},
address = {Cham},
abstract = {Ässembly systems are required to be more flexible due to increasing product variety. This is accomplished by breaking up the prevailing rigid linking of assembly stations in classic line configurations to line-less assembly systems (LAS). For a quantified potential analysis, it is necessary to assess the dependency of performance indicators on input parameters in a large number of production scenarios. Existing methods are either experience-based or computationally expensive due to full-factorial experiment plans. Therefore, the contribution of this paper is threefold. First, a seamlessly automated scenario analysis tool is developed, simulating a large set of assembly scenarios using a discrete-event simulation. Second, an artificial neural network (ANN) based approximation pipeline of the scenario analysis is implemented. An integrated AutoML pipeline for hyperparameter optimization (HPO) and Neural Architecture Search (NAS) allows for a faster potential approximation with sufficiently accurate prediction. Last, an integrated decision support system including the mentioned components is defined. It includes a priori planning of the overall system and allows the assessment of adaptions in reaction to the current system status."},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Ässembly systems are required to be more flexible due to increasing product variety. This is accomplished by breaking up the prevailing rigid linking of assembly stations in classic line configurations to line-less assembly systems (LAS). For a quantified potential analysis, it is necessary to assess the dependency of performance indicators on input parameters in a large number of production scenarios. Existing methods are either experience-based or computationally expensive due to full-factorial experiment plans. Therefore, the contribution of this paper is threefold. First, a seamlessly automated scenario analysis tool is developed, simulating a large set of assembly scenarios using a discrete-event simulation. Second, an artificial neural network (ANN) based approximation pipeline of the scenario analysis is implemented. An integrated AutoML pipeline for hyperparameter optimization (HPO) and Neural Architecture Search (NAS) allows for a faster potential approximation with sufficiently accurate prediction. Last, an integrated decision support system including the mentioned components is defined. It includes a priori planning of the overall system and allows the assessment of adaptions in reaction to the current system status."
1512.
Cheng, Guoan; Matsune, Ai; Du, Hao; Liu, XinZhi; Zhan, Shu
Exploring more diverse network architectures for single image super-resolution Journal Article
In: Knowledge-Based Systems, 235 , pp. 107648, 2022, ISSN: 0950-7051.
@article{CHENG2022107648,
title = {Exploring more diverse network architectures for single image super-resolution},
author = {Guoan Cheng and Ai Matsune and Hao Du and XinZhi Liu and Shu Zhan},
url = {https://www.sciencedirect.com/science/article/pii/S0950705121009102},
doi = {https://doi.org/10.1016/j.knosys.2021.107648},
issn = {0950-7051},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Knowledge-Based Systems},
volume = {235},
pages = {107648},
abstract = {We propose a plug-and-play neural architecture search (NAS) method to explore diverse architectures for single image super-resolution (SISR). Unlike current NAS-based methods with the single path setting and pipeline setting, our proposed method achieves the trade-off between diverse network architectures and search cost. Our proposed method formulates the task in a differentiable manner, which inherits the architecture parameter optimization method from Discrete Stochastic Neural Architecture Search (DSNAS). Besides the straightforward searching of operations, we also search each node in a cell for the activation function, from-node, and skip-connection node, which diverse the searched architecture topologies. The individually searching of skip-connection node avoids skip-connection excessive phenomenon. Moreover, to alleviate the influence of inconsistent architecture between training and testing periods, we introduce random variables into the architecture parameter as regularization. Benchmark experiments show our state-of-the-art performance under specific parameters and FLOPs constraints. Compared with other NAS-based SISR methods, our proposed methods achieve better performance with less searching time and resources. The superior results further demonstrate the effectiveness of our proposed NAS methods.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
We propose a plug-and-play neural architecture search (NAS) method to explore diverse architectures for single image super-resolution (SISR). Unlike current NAS-based methods with the single path setting and pipeline setting, our proposed method achieves the trade-off between diverse network architectures and search cost. Our proposed method formulates the task in a differentiable manner, which inherits the architecture parameter optimization method from Discrete Stochastic Neural Architecture Search (DSNAS). Besides the straightforward searching of operations, we also search each node in a cell for the activation function, from-node, and skip-connection node, which diverse the searched architecture topologies. The individually searching of skip-connection node avoids skip-connection excessive phenomenon. Moreover, to alleviate the influence of inconsistent architecture between training and testing periods, we introduce random variables into the architecture parameter as regularization. Benchmark experiments show our state-of-the-art performance under specific parameters and FLOPs constraints. Compared with other NAS-based SISR methods, our proposed methods achieve better performance with less searching time and resources. The superior results further demonstrate the effectiveness of our proposed NAS methods.
1511.
Chen, Haoze; Zhang, Zhijie; Jin, Mingwu; Wang, Fengxiang
Prediction of dMRI signals with neural architecture search Journal Article
In: Journal of Neuroscience Methods, 365 , pp. 109389, 2022, ISSN: 0165-0270.
@article{CHEN2022109389,
title = {Prediction of dMRI signals with neural architecture search},
author = {Haoze Chen and Zhijie Zhang and Mingwu Jin and Fengxiang Wang},
url = {https://www.sciencedirect.com/science/article/pii/S0165027021003241},
doi = {https://doi.org/10.1016/j.jneumeth.2021.109389},
issn = {0165-0270},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Journal of Neuroscience Methods},
volume = {365},
pages = {109389},
abstract = {Background
There is growing interest in the neuroscience community in estimating and mapping microscopic properties of brain tissue non-invasively using magnetic resonance measurements. Machine learning methods are actively investigated to predict the signals measured in diffusion magnetic resonance imaging (dMRI).
New method
We applied the neural architecture search (NAS) to train a recurrent neural network to generate a multilayer perceptron to predict the dMRI data of unknown signals based on the different acquisition parameters and training data. The search space of NAS is the number of neurons in each layer of the multilayer perceptron network. To our best knowledge, this is the first time to apply NAS to solve the dMRI signal prediction problem.
Results
The experimental results demonstrate that the proposed NAS method can achieve fast training and predict dMRI signals accurately. For dMRI signals with four acquisition strategies of double diffusion encoding (DDE), double oscillating diffusion encoding (DODE), multi-shell and DSI-like pulsed gradient spin-echo (PGSE), the mean squared errors of the multilayer perceptron network designed by NAS are 0.0043, 0.0034, 0.0147 and 0.0199, respectively.
Comparison with existing method(s)
We also compared NAS with other machine learning prediction methods, such as support vector regression (SVR), decision tree (DT) and random forest (RF), k-nearest neighbors (KNN), adaboost regressor (AR), gradient boosting regressor (GBR) and extra-trees regressor (ET). NAS achieved the better prediction performance in most cases.
Conclusion
In this study, NAS was developed for the prediction of dMRI signals and could become an effective prediction tool.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Background
There is growing interest in the neuroscience community in estimating and mapping microscopic properties of brain tissue non-invasively using magnetic resonance measurements. Machine learning methods are actively investigated to predict the signals measured in diffusion magnetic resonance imaging (dMRI).
New method
We applied the neural architecture search (NAS) to train a recurrent neural network to generate a multilayer perceptron to predict the dMRI data of unknown signals based on the different acquisition parameters and training data. The search space of NAS is the number of neurons in each layer of the multilayer perceptron network. To our best knowledge, this is the first time to apply NAS to solve the dMRI signal prediction problem.
Results
The experimental results demonstrate that the proposed NAS method can achieve fast training and predict dMRI signals accurately. For dMRI signals with four acquisition strategies of double diffusion encoding (DDE), double oscillating diffusion encoding (DODE), multi-shell and DSI-like pulsed gradient spin-echo (PGSE), the mean squared errors of the multilayer perceptron network designed by NAS are 0.0043, 0.0034, 0.0147 and 0.0199, respectively.
Comparison with existing method(s)
We also compared NAS with other machine learning prediction methods, such as support vector regression (SVR), decision tree (DT) and random forest (RF), k-nearest neighbors (KNN), adaboost regressor (AR), gradient boosting regressor (GBR) and extra-trees regressor (ET). NAS achieved the better prediction performance in most cases.
Conclusion
In this study, NAS was developed for the prediction of dMRI signals and could become an effective prediction tool.
There is growing interest in the neuroscience community in estimating and mapping microscopic properties of brain tissue non-invasively using magnetic resonance measurements. Machine learning methods are actively investigated to predict the signals measured in diffusion magnetic resonance imaging (dMRI).
New method
We applied the neural architecture search (NAS) to train a recurrent neural network to generate a multilayer perceptron to predict the dMRI data of unknown signals based on the different acquisition parameters and training data. The search space of NAS is the number of neurons in each layer of the multilayer perceptron network. To our best knowledge, this is the first time to apply NAS to solve the dMRI signal prediction problem.
Results
The experimental results demonstrate that the proposed NAS method can achieve fast training and predict dMRI signals accurately. For dMRI signals with four acquisition strategies of double diffusion encoding (DDE), double oscillating diffusion encoding (DODE), multi-shell and DSI-like pulsed gradient spin-echo (PGSE), the mean squared errors of the multilayer perceptron network designed by NAS are 0.0043, 0.0034, 0.0147 and 0.0199, respectively.
Comparison with existing method(s)
We also compared NAS with other machine learning prediction methods, such as support vector regression (SVR), decision tree (DT) and random forest (RF), k-nearest neighbors (KNN), adaboost regressor (AR), gradient boosting regressor (GBR) and extra-trees regressor (ET). NAS achieved the better prediction performance in most cases.
Conclusion
In this study, NAS was developed for the prediction of dMRI signals and could become an effective prediction tool.
1510.
Nistor, Sergiu Cosmin; Czibula, Gabriela
IntelliSwAS: Optimizing deep neural network architectures using a particle swarm-based approach Journal Article
In: Expert Systems with Applications, 187 , pp. 115945, 2022, ISSN: 0957-4174.
@article{NISTOR2022115945,
title = {IntelliSwAS: Optimizing deep neural network architectures using a particle swarm-based approach},
author = {Sergiu Cosmin Nistor and Gabriela Czibula},
url = {https://www.sciencedirect.com/science/article/pii/S0957417421012987},
doi = {https://doi.org/10.1016/j.eswa.2021.115945},
issn = {0957-4174},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Expert Systems with Applications},
volume = {187},
pages = {115945},
abstract = {Deep learning is in a continuous evolution and many domains benefit from this substantial progress in the development of intelligent solutions. While this progress has been swift, there are more and more applications and the specific requirements of each application domain entails much work done by researchers to adapt existing models or create new ones. This traditional approach has produced many successful designs, but recently, automated methods of finding neural network architectures emerged. We introduce IntelliSwAS approach for optimizing deep neural network architectures for a classification or regression task. Image classification was selected as the task, but IntelliSwAS is generic enough such that it could be successfully applied on other classification or regression problems. A particle swarm-based optimization algorithm is proposed for the automatic search for convolutional neural network architectures. The search technique is enhanced by a machine learning model (DAGRNN) which we designed for predicting the quality of the network architectures and thus increasing the performance of the algorithm. The proposed model is able to process data structured as directed acyclic graphs in general and we applied it specifically on network architectures. The network architecture that we discovered using IntelliSwAS surpassed 89.8% of the competing image classification models that we considered for comparison.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Deep learning is in a continuous evolution and many domains benefit from this substantial progress in the development of intelligent solutions. While this progress has been swift, there are more and more applications and the specific requirements of each application domain entails much work done by researchers to adapt existing models or create new ones. This traditional approach has produced many successful designs, but recently, automated methods of finding neural network architectures emerged. We introduce IntelliSwAS approach for optimizing deep neural network architectures for a classification or regression task. Image classification was selected as the task, but IntelliSwAS is generic enough such that it could be successfully applied on other classification or regression problems. A particle swarm-based optimization algorithm is proposed for the automatic search for convolutional neural network architectures. The search technique is enhanced by a machine learning model (DAGRNN) which we designed for predicting the quality of the network architectures and thus increasing the performance of the algorithm. The proposed model is able to process data structured as directed acyclic graphs in general and we applied it specifically on network architectures. The network architecture that we discovered using IntelliSwAS surpassed 89.8% of the competing image classification models that we considered for comparison.
1509.
Fei, Ke; Li, Qi; Zhu, Congcong
Non-technical losses detection using missing values’ pattern and neural architecture search Journal Article
In: International Journal of Electrical Power & Energy Systems, 134 , pp. 107410, 2022, ISSN: 0142-0615.
@article{FEI2022107410,
title = {Non-technical losses detection using missing values’ pattern and neural architecture search},
author = {Ke Fei and Qi Li and Congcong Zhu},
url = {https://www.sciencedirect.com/science/article/pii/S0142061521006499},
doi = {https://doi.org/10.1016/j.ijepes.2021.107410},
issn = {0142-0615},
year = {2022},
date = {2022-01-01},
journal = {International Journal of Electrical Power & Energy Systems},
volume = {134},
pages = {107410},
abstract = {The fast growth of non-technical loss (NTL) has gradually become one of the main concerns for distribution network operators (DNOs). Electricity theft which constitutes the main part of NTL not only brings losses to the DNOs, but also reduces the quality of the supply. A traditional detection method relies on utility workers’ experience and consumes a large amount of manpower. Thanks to the emerging of advanced metering infrastructure (AMI), utility companies can now collect detailed data reflecting consumers’ electricity usage, which enabled algorithms-based non-technical loss detection. The current data-based methods focus on the characteristics of electricity consumption thereby less efficient when dealing with rapidly changed electricity theft techniques. This article introduced a new data set, the location information of missing values, to improve the accuracy of non-technical loss detection. The relationship between missing values and electricity theft techniques is analyzed and a neural network model is built through neural architecture search (NAS). The improved model achieved an excellent Area Under Curve (AUC) value around 0.926 which verified the close link between missing values and electricity theft techniques. The nature of neural architecture search allows automatic model update which makes it a user-friendly tool even for engineers without any neural network expertise. A case study was carried out in which the missing value pattern was analyzed through Density-Based Spatial Clustering of Applications with Noise (DBSCAN) clustering algorithm.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
The fast growth of non-technical loss (NTL) has gradually become one of the main concerns for distribution network operators (DNOs). Electricity theft which constitutes the main part of NTL not only brings losses to the DNOs, but also reduces the quality of the supply. A traditional detection method relies on utility workers’ experience and consumes a large amount of manpower. Thanks to the emerging of advanced metering infrastructure (AMI), utility companies can now collect detailed data reflecting consumers’ electricity usage, which enabled algorithms-based non-technical loss detection. The current data-based methods focus on the characteristics of electricity consumption thereby less efficient when dealing with rapidly changed electricity theft techniques. This article introduced a new data set, the location information of missing values, to improve the accuracy of non-technical loss detection. The relationship between missing values and electricity theft techniques is analyzed and a neural network model is built through neural architecture search (NAS). The improved model achieved an excellent Area Under Curve (AUC) value around 0.926 which verified the close link between missing values and electricity theft techniques. The nature of neural architecture search allows automatic model update which makes it a user-friendly tool even for engineers without any neural network expertise. A case study was carried out in which the missing value pattern was analyzed through Density-Based Spatial Clustering of Applications with Noise (DBSCAN) clustering algorithm.
2021
1508.
Nangi, Sharmila Reddy; Tyagi, Atharv; Mundra, Jay; Mukherjee, Sagnik; Snehal, Raj; Chhaya, Niyati; Garimella, Aparna
AUTOSUMM: Automatic Model Creation for Text Summarization Inproceedings
In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 10162–10172, Ässociation for Computational Linguistics, Önline and Punta Cana, Dominican Republic, 2021.
@inproceedings{nangi-etal-2021-autosumm,
title = {AUTOSUMM: Automatic Model Creation for Text Summarization},
author = {Sharmila Reddy Nangi and Atharv Tyagi and Jay Mundra and Sagnik Mukherjee and Raj Snehal and Niyati Chhaya and Aparna Garimella},
url = {https://aclanthology.org/2021.emnlp-main.798},
year = {2021},
date = {2021-11-01},
urldate = {2021-11-01},
booktitle = {Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing},
pages = {10162--10172},
publisher = {Ässociation for Computational Linguistics},
address = {Önline and Punta Cana, Dominican Republic},
abstract = {Recent efforts to develop deep learning models for text generation tasks such as extractive and abstractive summarization have resulted in state-of-the-art performances on various datasets. However, obtaining the best model configuration for a given dataset requires an extensive knowledge of deep learning specifics like model architecture, tuning parameters etc., and is often extremely challenging for a non-expert. In this paper, we propose methods to automatically create deep learning models for the tasks of extractive and abstractive text summarization. Based on the recent advances in Automated Machine Learning and the success of large language models such as BERT and GPT-2 in encoding knowledge, we use a combination of Neural Architecture Search (NAS) and Knowledge Distillation (KD) techniques to perform model search and compression using the vast knowledge provided by these language models to develop smaller, customized models for any given dataset. We present extensive empirical results to illustrate the effectiveness of our model creation methods in terms of inference time and model size, while achieving near state-of-the-art performances in terms of accuracy across a range of datasets.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Recent efforts to develop deep learning models for text generation tasks such as extractive and abstractive summarization have resulted in state-of-the-art performances on various datasets. However, obtaining the best model configuration for a given dataset requires an extensive knowledge of deep learning specifics like model architecture, tuning parameters etc., and is often extremely challenging for a non-expert. In this paper, we propose methods to automatically create deep learning models for the tasks of extractive and abstractive text summarization. Based on the recent advances in Automated Machine Learning and the success of large language models such as BERT and GPT-2 in encoding knowledge, we use a combination of Neural Architecture Search (NAS) and Knowledge Distillation (KD) techniques to perform model search and compression using the vast knowledge provided by these language models to develop smaller, customized models for any given dataset. We present extensive empirical results to illustrate the effectiveness of our model creation methods in terms of inference time and model size, while achieving near state-of-the-art performances in terms of accuracy across a range of datasets.
1507.
Chen, Zerui; Huang, Yan; Yu, Hongyuan; Wang, Liang
Learning a Robust Part-Aware Monocular 3D Human Pose Estimator via Neural Architecture Search Journal Article
In: International Journal of Computer Vision (2021), 2021.
@article{nokey,
title = {Learning a Robust Part-Aware Monocular 3D Human Pose Estimator via Neural Architecture Search},
author = {Zerui Chen and Yan Huang and Hongyuan Yu and Liang Wang
},
url = {https://doi.org/10.1007/s11263-021-01525-0},
year = {2021},
date = {2021-10-26},
urldate = {2021-10-26},
journal = { International Journal of Computer Vision (2021)},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
1506.
Burrello, Alessio; Pagliari, Daniele Jahier; Risso, Matteo; Benatti, Simone; Macii, Enrico; Benini, Luca; Poncino, Massimo
Q-PPG: Energy-Efficient PPG-based Heart Rate Monitoring on Wearable Devices Journal Article
In: IEEE Trans Biomed Circuits Syst ., 2021.
@article{Burrello2201,
title = { Q-PPG: Energy-Efficient PPG-based Heart Rate Monitoring on Wearable Devices },
author = {Alessio Burrello and Daniele Jahier Pagliari and Matteo Risso and Simone Benatti and Enrico Macii and Luca Benini and Massimo Poncino
},
url = {https://pubmed.ncbi.nlm.nih.gov/34673496/},
doi = {10.1109/TBCAS.2021.3122017 },
year = {2021},
date = {2021-10-21},
journal = { IEEE Trans Biomed Circuits Syst .},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
1505.
Wang, Ning; Gao, Yang; Chen, Hao; Wang, Peng; Tian, Zhi; Shen, Chunhua; Zhang, Yanning
NAS-FCOS: Efficient Search for Object Detection Architectures Journal Article
In: International Journal of Computer Vision , 129 , pp. 3299–3312, 2021.
@article{Wang2021,
title = {NAS-FCOS: Efficient Search for Object Detection Architectures},
author = {Ning Wang and Yang Gao and Hao Chen and Peng Wang and Zhi Tian and Chunhua Shen and Yanning Zhang },
url = {https://link.springer.com/article/10.1007/s11263-021-01523-2},
year = {2021},
date = {2021-10-15},
urldate = {2021-10-15},
journal = {International Journal of Computer Vision },
volume = {129},
pages = {3299–3312},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
1504.
Oyelade, Olaide N.; Ezugwu, Absalom E.
A bioinspired neural architecture search based convolutional neural network for breast cancer detection using histopathology images Bachelor Thesis
2021.
@bachelorthesis{Oyelade2021,
title = {A bioinspired neural architecture search based convolutional neural network for breast cancer detection using histopathology images},
author = {Oyelade, Olaide N. and Ezugwu, Absalom E.},
url = {https://doi.org/10.1038/s41598-021-98978-7},
year = {2021},
date = {2021-10-07},
journal = {Scientific Reports},
keywords = {},
pubstate = {published},
tppubtype = {bachelorthesis}
}
1503.
Zhou, Qinqin; Zheng, Xiawu; Cao, Liujuan; Zhong, Bineng; Xi, Teng; Zhang, Gang; Ding, Errui; Xu, Mingliang; Ji, Rongrong
EC-DARTS: Inducing Equalized and Consistent Optimization Into DARTS Inproceedings
In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 11986-11995, 2021.
@inproceedings{Zhou_2021_ICCV,
title = {EC-DARTS: Inducing Equalized and Consistent Optimization Into DARTS},
author = {Qinqin Zhou and Xiawu Zheng and Liujuan Cao and Bineng Zhong and Teng Xi and Gang Zhang and Errui Ding and Mingliang Xu and Rongrong Ji},
url = {https://openaccess.thecvf.com/content/ICCV2021/html/Zhou_EC-DARTS_Inducing_Equalized_and_Consistent_Optimization_Into_DARTS_ICCV_2021_paper.html},
year = {2021},
date = {2021-10-01},
urldate = {2021-10-01},
booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
pages = {11986-11995},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
1502.
Lin, Ming; Wang, Pichao; Sun, Zhenhong; Chen, Hesen; Sun, Xiuyu; Qian, Qi; Li, Hao; Jin, Rong
Zen-NAS: A Zero-Shot NAS for High-Performance Image Recognition Inproceedings
In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 347-356, 2021.
@inproceedings{Lin_2021_ICCV,
title = {Zen-NAS: A Zero-Shot NAS for High-Performance Image Recognition},
author = {Ming Lin and Pichao Wang and Zhenhong Sun and Hesen Chen and Xiuyu Sun and Qi Qian and Hao Li and Rong Jin},
url = {https://openaccess.thecvf.com/content/ICCV2021/html/Lin_Zen-NAS_A_Zero-Shot_NAS_for_High-Performance_Image_Recognition_ICCV_2021_paper.html},
year = {2021},
date = {2021-10-01},
urldate = {2021-10-01},
booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
pages = {347-356},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
1501.
Shen, Mingzhu; Liang, Feng; Gong, Ruihao; Li, Yuhang; Li, Chuming; Lin, Chen; Yu, Fengwei; Yan, Junjie; Ouyang, Wanli
Once Quantization-Aware Training: High Performance Extremely Low-Bit Architecture Search Inproceedings
In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 5340-5349, 2021.
@inproceedings{Shen_2021_ICCV,
title = {Once Quantization-Aware Training: High Performance Extremely Low-Bit Architecture Search},
author = {Mingzhu Shen and Feng Liang and Ruihao Gong and Yuhang Li and Chuming Li and Chen Lin and Fengwei Yu and Junjie Yan and Wanli Ouyang},
url = {https://openaccess.thecvf.com/content/ICCV2021/html/Shen_Once_Quantization-Aware_Training_High_Performance_Extremely_Low-Bit_Architecture_Search_ICCV_2021_paper.html},
year = {2021},
date = {2021-10-01},
urldate = {2021-10-01},
booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
pages = {5340-5349},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
1500.
Xue, Song; Wang, Runqi; Zhang, Baochang; Wang, Tian; Guo, Guodong; Doermann, David
IDARTS: Interactive Differentiable Architecture Search Inproceedings
In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1163-1172, 2021.
@inproceedings{Xue_2021_ICCV,
title = {IDARTS: Interactive Differentiable Architecture Search},
author = {Song Xue and Runqi Wang and Baochang Zhang and Tian Wang and Guodong Guo and David Doermann},
url = {https://openaccess.thecvf.com/content/ICCV2021/html/Xue_IDARTS_Interactive_Differentiable_Architecture_Search_ICCV_2021_paper.html},
year = {2021},
date = {2021-10-01},
urldate = {2021-10-01},
booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
pages = {1163-1172},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
1499.
Chu, Xiangxiang; Zhang, Bo; Li, Qingyuan; Xu, Ruijun; Li, Xudong
SCARLET-NAS: Bridging the Gap Between Stability and Scalability in Weight-Sharing Neural Architecture Search Inproceedings
In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, pp. 317-325, 2021.
@inproceedings{Chu_2021_ICCV,
title = {SCARLET-NAS: Bridging the Gap Between Stability and Scalability in Weight-Sharing Neural Architecture Search},
author = {Xiangxiang Chu and Bo Zhang and Qingyuan Li and Ruijun Xu and Xudong Li},
url = {https://openaccess.thecvf.com/content/ICCV2021W/NeurArch/html/Chu_SCARLET-NAS_Bridging_the_Gap_Between_Stability_and_Scalability_in_Weight-Sharing_ICCVW_2021_paper.html},
year = {2021},
date = {2021-10-01},
urldate = {2021-10-01},
booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops},
pages = {317-325},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
1498.
Hu, Xiaobin; Ren, Wenqi; Yu, Kaicheng; Zhang, Kaihao; Cao, Xiaochun; Liu, Wei; Menze, Bjoern
Pyramid Architecture Search for Real-Time Image Deblurring Inproceedings
In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4298-4307, 2021.
@inproceedings{Hu_2021_ICCV,
title = {Pyramid Architecture Search for Real-Time Image Deblurring},
author = {Xiaobin Hu and Wenqi Ren and Kaicheng Yu and Kaihao Zhang and Xiaochun Cao and Wei Liu and Bjoern Menze},
url = {https://openaccess.thecvf.com/content/ICCV2021/html/Hu_Pyramid_Architecture_Search_for_Real-Time_Image_Deblurring_ICCV_2021_paper.html},
year = {2021},
date = {2021-10-01},
urldate = {2021-10-01},
booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
pages = {4298-4307},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
1497.
Chatzianastasis, Michail; Dasoulas, George; Siolas, Georgios; Vazirgiannis, Michalis
Graph-Based Neural Architecture Search With Operation Embeddings Inproceedings
In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, pp. 393-402, 2021.
@inproceedings{Chatzianastasis_2021_ICCV,
title = {Graph-Based Neural Architecture Search With Operation Embeddings},
author = {Michail Chatzianastasis and George Dasoulas and Georgios Siolas and Michalis Vazirgiannis},
url = {https://openaccess.thecvf.com/content/ICCV2021W/NeurArch/html/Chatzianastasis_Graph-Based_Neural_Architecture_Search_With_Operation_Embeddings_ICCVW_2021_paper.html},
year = {2021},
date = {2021-10-01},
urldate = {2021-10-01},
booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops},
pages = {393-402},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
1496.
Zeng, Dan; Huang, Yuhang; Bao, Qian; Zhang, Junjie; Su, Chi; Liu, Wu
Neural Architecture Search for Joint Human Parsing and Pose Estimation Inproceedings
In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 11385-11394, 2021.
@inproceedings{Zeng_2021_ICCV,
title = {Neural Architecture Search for Joint Human Parsing and Pose Estimation},
author = {Dan Zeng and Yuhang Huang and Qian Bao and Junjie Zhang and Chi Su and Wu Liu},
url = {https://openaccess.thecvf.com/content/ICCV2021/html/Zeng_Neural_Architecture_Search_for_Joint_Human_Parsing_and_Pose_Estimation_ICCV_2021_paper.html},
year = {2021},
date = {2021-10-01},
urldate = {2021-10-01},
booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
pages = {11385-11394},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
1495.
Chen, Ziye; Zhan, Yibing; Yu, Baosheng; Gong, Mingming; Du, Bo
Not All Operations Contribute Equally: Hierarchical Operation-Adaptive Predictor for Neural Architecture Search Inproceedings
In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10508-10517, 2021.
@inproceedings{Chen_2021_ICCV,
title = {Not All Operations Contribute Equally: Hierarchical Operation-Adaptive Predictor for Neural Architecture Search},
author = {Ziye Chen and Yibing Zhan and Baosheng Yu and Mingming Gong and Bo Du},
url = {https://openaccess.thecvf.com/content/ICCV2021/html/Chen_Not_All_Operations_Contribute_Equally_Hierarchical_Operation-Adaptive_Predictor_for_Neural_ICCV_2021_paper.html},
year = {2021},
date = {2021-10-01},
urldate = {2021-10-01},
booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
pages = {10508-10517},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
1494.
Wang, Yaoming; Liu, Yuchen; Dai, Wenrui; Li, Chenglin; Zou, Junni; Xiong, Hongkai
In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 12312-12321, 2021.
@inproceedings{Wang_2021_ICCV,
title = {Learning Latent Architectural Distribution in Differentiable Neural Architecture Search via Variational Information Maximization},
author = {Yaoming Wang and Yuchen Liu and Wenrui Dai and Chenglin Li and Junni Zou and Hongkai Xiong},
url = {https://openaccess.thecvf.com/content/ICCV2021/html/Wang_Learning_Latent_Architectural_Distribution_in_Differentiable_Neural_Architecture_Search_via_ICCV_2021_paper.html},
year = {2021},
date = {2021-10-01},
urldate = {2021-10-01},
booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
pages = {12312-12321},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
1493.
Rana, Amrita; Kim, Kyung Ki
Robust architecture search using network adaptation Journal Article
In: Journal of Sensor Science and Technology, 30 (5), pp. 290-294, 2021.
@article{Rana2021,
title = {Robust architecture search using network adaptation},
author = {Amrita Rana and Kyung Ki Kim},
url = {https://www.koreascience.or.kr/article/JAKO202129159601206.pdf},
year = {2021},
date = {2021-09-30},
urldate = {2021-09-30},
journal = {Journal of Sensor Science and Technology},
volume = {30},
number = {5},
pages = {290-294},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
1492.
Yan, Kefan; Wang, Shunlong; Guan, Yuwei; Jiang, Ailian
TNAS: Neural Network Architecture Search Sampling Module Based on Variational Autoencoder Journal Article
In: CONVERTER, pp. 288-296, 2021.
@article{Yan2021,
title = { TNAS: Neural Network Architecture Search Sampling Module Based on Variational Autoencoder},
author = {Kefan Yan and Shunlong Wang and Yuwei Guan and Ailian Jiang},
url = {http://converter-magazine.info/index.php/converter/article/view/499},
year = {2021},
date = {2021-09-07},
urldate = {2021-09-07},
journal = {CONVERTER},
pages = {288-296},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
1491.
Hao, Jie; Cai, Zhiling; Li, Ruijia; Zhu, William
Saliency: a new selection criterion of important architectures in neural architecture search Journal Article
In: Neural Computing and Applications, 2021.
@article{Hao2021b,
title = {Saliency: a new selection criterion of important architectures in neural architecture search},
author = {Jie Hao and Zhiling Cai and Ruijia Li and William Zhu },
url = {https://doi.org/10.1007/s00521-021-06418-4},
year = {2021},
date = {2021-09-05},
urldate = {2021-09-05},
journal = { Neural Computing and Applications},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
1490.
Cazasnoves, Anthony; Ganaye, Pierre-Antoine; Sanchis, Kévin; Ceillier, Tugdual
Neural Architecture Search in operational context: a remote sensing case-study Technical Report
Preligens 2021.
@techreport{cazasnoves:hal-03343944,
title = {Neural Architecture Search in operational context: a remote sensing case-study},
author = {Anthony Cazasnoves and Pierre-Antoine Ganaye and Kévin Sanchis and Tugdual Ceillier},
url = {https://hal.archives-ouvertes.fr/hal-03343944},
year = {2021},
date = {2021-09-01},
urldate = {2021-09-01},
institution = {Preligens},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1489.
Jia, Hong; Hu, Jiawei; Hu, Wen
SwingNet: Ubiquitous Fine-Grained Swing Tracking Framework via Stochastic Neural Architecture Search and Adversarial Learning Journal Article
In: Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., 5 (3), 2021.
@article{10.1145/3478082,
title = {SwingNet: Ubiquitous Fine-Grained Swing Tracking Framework via Stochastic Neural Architecture Search and Adversarial Learning},
author = {Hong Jia and Jiawei Hu and Wen Hu},
url = {https://doi.org/10.1145/3478082},
doi = {10.1145/3478082},
year = {2021},
date = {2021-09-01},
urldate = {2021-09-01},
journal = {Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.},
volume = {5},
number = {3},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
abstract = {Sports analytics in the wild (i.e., ubiquitously) is a thriving industry. Swing tracking
is a key feature in sports analytics. Therefore, a centimeter-level tracking resolution
solution is required. Recent research has explored deep neural networks for sensor
fusion to produce consistent swing-tracking performance. This is achieved by combining
the advantages of two sensor modalities (IMUs and depth sensors) for golf swing tracking.
Here, the IMUs are not affected by occlusion and can support high sampling rates.
Meanwhile, depth sensors produce significantly more accurate motion measurements than
those produced by IMUs. Nevertheless, this method can be further improved in terms
of accuracy and lacking information for different domains (e.g., subjects, sports,
and devices). Unfortunately, designing a deep neural network with good performance
is time consuming and labor intensive, which is challenging when a network model is
deployed to be used in new settings. To this end, we propose a network based on Neural
Architecture Search (NAS), called SwingNet, which is a regression-based automatic
generated deep neural network via stochastic neural network search. The proposed network
aims to learn the swing tracking feature for better prediction automatically. Furthermore,
SwingNet features a domain discriminator by using unsupervised learning and adversarial
learning to ensure that it can be adaptive to unobserved domains. We implemented SwingNet
prototypes with a smart wristband (IMU) and smartphone (depth sensor), which are ubiquitously
available. They enable accurate sports analytics (e.g., coaching, tracking, analysis
and assessment) in the wild.Our comprehensive experiment shows that SwingNet achieves
less than 10 cm errors of swing tracking with a subject-independent model covering
multiple sports (e.g., golf and tennis) and depth sensor hardware, which outperforms
state-of-the-art approaches.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Sports analytics in the wild (i.e., ubiquitously) is a thriving industry. Swing tracking
is a key feature in sports analytics. Therefore, a centimeter-level tracking resolution
solution is required. Recent research has explored deep neural networks for sensor
fusion to produce consistent swing-tracking performance. This is achieved by combining
the advantages of two sensor modalities (IMUs and depth sensors) for golf swing tracking.
Here, the IMUs are not affected by occlusion and can support high sampling rates.
Meanwhile, depth sensors produce significantly more accurate motion measurements than
those produced by IMUs. Nevertheless, this method can be further improved in terms
of accuracy and lacking information for different domains (e.g., subjects, sports,
and devices). Unfortunately, designing a deep neural network with good performance
is time consuming and labor intensive, which is challenging when a network model is
deployed to be used in new settings. To this end, we propose a network based on Neural
Architecture Search (NAS), called SwingNet, which is a regression-based automatic
generated deep neural network via stochastic neural network search. The proposed network
aims to learn the swing tracking feature for better prediction automatically. Furthermore,
SwingNet features a domain discriminator by using unsupervised learning and adversarial
learning to ensure that it can be adaptive to unobserved domains. We implemented SwingNet
prototypes with a smart wristband (IMU) and smartphone (depth sensor), which are ubiquitously
available. They enable accurate sports analytics (e.g., coaching, tracking, analysis
and assessment) in the wild.Our comprehensive experiment shows that SwingNet achieves
less than 10 cm errors of swing tracking with a subject-independent model covering
multiple sports (e.g., golf and tennis) and depth sensor hardware, which outperforms
state-of-the-art approaches.
is a key feature in sports analytics. Therefore, a centimeter-level tracking resolution
solution is required. Recent research has explored deep neural networks for sensor
fusion to produce consistent swing-tracking performance. This is achieved by combining
the advantages of two sensor modalities (IMUs and depth sensors) for golf swing tracking.
Here, the IMUs are not affected by occlusion and can support high sampling rates.
Meanwhile, depth sensors produce significantly more accurate motion measurements than
those produced by IMUs. Nevertheless, this method can be further improved in terms
of accuracy and lacking information for different domains (e.g., subjects, sports,
and devices). Unfortunately, designing a deep neural network with good performance
is time consuming and labor intensive, which is challenging when a network model is
deployed to be used in new settings. To this end, we propose a network based on Neural
Architecture Search (NAS), called SwingNet, which is a regression-based automatic
generated deep neural network via stochastic neural network search. The proposed network
aims to learn the swing tracking feature for better prediction automatically. Furthermore,
SwingNet features a domain discriminator by using unsupervised learning and adversarial
learning to ensure that it can be adaptive to unobserved domains. We implemented SwingNet
prototypes with a smart wristband (IMU) and smartphone (depth sensor), which are ubiquitously
available. They enable accurate sports analytics (e.g., coaching, tracking, analysis
and assessment) in the wild.Our comprehensive experiment shows that SwingNet achieves
less than 10 cm errors of swing tracking with a subject-independent model covering
multiple sports (e.g., golf and tennis) and depth sensor hardware, which outperforms
state-of-the-art approaches.
1488.
Yang, Xin; Fan, Jiangfeng; Wu, Chenhuan; Zhou, Dake; Li, Tao
NasmamSR: a fast image super-resolution network based on neural architecture search and multiple attention mechanism Journal Article
In: Multimedia Systems (2021), 2021.
@article{Yang2021f,
title = {NasmamSR: a fast image super-resolution network based on neural architecture search and multiple attention mechanism},
author = {Xin Yang and Jiangfeng Fan and Chenhuan Wu and Dake Zhou and Tao Li },
url = {https://link.springer.com/article/10.1007/s00530-021-00841-2},
year = {2021},
date = {2021-08-05},
urldate = {2021-08-05},
journal = {Multimedia Systems (2021)},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
1487.
Zhao, Zimin; Kang, Ying; Hou, Aiqin; Gan, Daguang
SP-DARTS: Synchronous Progressive Differentiable Neural Architecture Search for Image Classification Journal Article
In: IEICE TRANSACTIONS on Information and Systems , E104-D (8 ), pp. 1232-1238, 2021, ISBN: 1745-1361.
@article{Zhao2021b,
title = { SP-DARTS: Synchronous Progressive Differentiable Neural Architecture Search for Image Classification},
author = {Zimin Zhao and Ying Kang and Aiqin Hou and Daguang Gan},
url = {https://search.ieice.org/bin/summary.php?id=e104-d_8_1232},
doi = {10.1587/transinf.2020BDP0009},
isbn = {1745-1361},
year = {2021},
date = {2021-08-01},
journal = {IEICE TRANSACTIONS on Information and Systems },
volume = {E104-D },
number = {8 },
pages = {1232-1238},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
1486.
Wendlinger, Lorenz; Berndl, Emanuel; Granitzer, Michael
Methods for Automatic Machine-Learning Workflow Analysis Inproceedings
In: ECML PKDD 2021, 2021.
@inproceedings{WEndlingerECML2021,
title = {Methods for Automatic Machine-Learning Workflow Analysis},
author = {Lorenz Wendlinger and Emanuel Berndl and Michael Granitzer},
url = {https://2021.ecmlpkdd.org/wp-content/uploads/2021/07/sub_473.pdf},
year = {2021},
date = {2021-08-01},
booktitle = {ECML PKDD 2021},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
1485.
Zhu, Yanming; Meijering, Erik
Automatic Improvement of Deep Learning Based Cell Segmentation in Time-Lapse Microscopy by Neural Architecture Search Journal Article
In: Bioinformatics, 2021.
@article{Zhu2021b,
title = {Automatic Improvement of Deep Learning Based Cell Segmentation in Time-Lapse Microscopy by Neural Architecture Search},
author = {Zhu, Yanming and Erik Meijering },
url = {https://pubmed.ncbi.nlm.nih.gov/34329376/},
doi = {10.1093/bioinformatics/btab556},
year = {2021},
date = {2021-07-30},
journal = {Bioinformatics},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
1484.
Zhao, Jiejie; Lv, Weifeng; Du, Bowen; Ye, Junchen; Sun, Leilei; Xiong, Guixi
Deep multi-task learning with flexible and compact architecture search Journal Article
In: International Journal of Data Science and Analytics , 2021.
@article{ZhaoIJDSA2021,
title = {Deep multi-task learning with flexible and compact architecture search},
author = {Jiejie Zhao and Weifeng Lv and Bowen Du and Junchen Ye and Leilei Sun and Guixi Xiong },
url = {https://link.springer.com/article/10.1007/s41060-021-00274-0},
year = {2021},
date = {2021-07-24},
journal = {International Journal of Data Science and Analytics },
keywords = {},
pubstate = {published},
tppubtype = {article}
}
1483.
Nguyen, Vu; Le, Tam; Yamada, Makoto; Osborne, Michael A
Optimal Transport Kernels for Sequential and Parallel Neural Architecture Search Inproceedings
In: Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, 2021.
@inproceedings{Nguyen2020_cbc,
title = {Optimal Transport Kernels for Sequential and Parallel Neural Architecture Search},
author = {Vu Nguyen and Tam Le and Makoto Yamada and Michael A Osborne},
url = {http://proceedings.mlr.press/v139/nguyen21d/nguyen21d.pdf},
year = {2021},
date = {2021-07-18},
urldate = {2021-07-18},
booktitle = {Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event},
volume = {139},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
1482.
Ünal, Hamit Taner; Başçiftçi, Fatih
Evolutionary design of neural network architectures: a review of three decades of research Journal Article
In: Artificial Intelligence Review, 2021.
@article{Ünal2021,
title = {Evolutionary design of neural network architectures: a review of three decades of research},
author = {Hamit Taner Ünal and Fatih Başçiftçi },
url = {https://link.springer.com/article/10.1007/s10462-021-10049-5},
year = {2021},
date = {2021-07-17},
journal = {Artificial Intelligence Review},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
1481.
Coquelin, Daniel; Sedona, Rocco; Riedel, Morris; Götz, Markus
Evolutionary Optimization of Neural Architectures in Remote Sensing Classification Problems Technical Report
2021.
@techreport{Coquelin2021,
title = {Evolutionary Optimization of Neural Architectures in Remote Sensing Classification Problems },
author = {Daniel Coquelin and Rocco Sedona and Morris Riedel and Markus Götz},
url = {https://www.researchgate.net/profile/Morris-Riedel/publication/353352507_EVOLUTIONARY_OPTIMIZATION_OF_NEURAL_ARCHITECTURES_IN_REMOTE_SENSING_CLASSIFICATION_PROBLEMS/links/60f7462b0c2bfa282aeef5cf/EVOLUTIONARY-OPTIMIZATION-OF-NEURAL-ARCHITECTURES-IN-REMOTE-SENSING-CLASSIFICATION-PROBLEMS.pdf},
year = {2021},
date = {2021-07-01},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1480.
Yu, Kaicheng
From Human-Designed Convolutional Neural Networks Towards Robust Neural Architecture Search PhD Thesis
2021.
@phdthesis{YuPhD2021,
title = {From Human-Designed Convolutional Neural Networks Towards Robust Neural Architecture Search },
author = {Yu, Kaicheng},
url = {https://infoscience.epfl.ch/record/287432},
doi = { 10.5075/epfl-thesis-8035},
year = {2021},
date = {2021-07-01},
abstract = {Artificial intelligence has been an ultimate design goal since the inception of computers decades ago. Among the many attempts towards general artificial intelligence, modern machine learning successfully tackles many complex problems thanks to the progress in deep learning, and in particular in convolutional neural networks (CNN). To design a CNN for a specific task, one common approach consists of adapting the heuristics from the pre-deep-learning era to the CNN domain. In the first part of this thesis, we introduce two methods that follow this approach: i) We build a covariance descriptor, i.e., a local descriptor that is suitable for texture recognition, to replace the first-order fully connected layers in an ordinary CNN, showing that such a descriptor yields state-of-the-art performance on many fine-grained image classification tasks with orders of magnitude fewer feature dimensions; ii) we develop a light-weight recurrent U-Net for image semantic segmentation, inspired by the biological eye saccadic movements, that yields real-time predictions on devices with limited computational resources. As most methods pre-dating automatic machine learning (AutoML), the two above-mentioned CNNs were human-designed. In the past few years, however, neural architecture search~(NAS), which aims to facilitate the design of deep networks for new tasks, has drawn an increasing attention. In this context, the weight-sharing approach, which consists of utilizing a super-net to encompass all possible architectures within a search space, has become a de facto standard in NAS because it enables the search to be done on commodity hardware. In the second part of this thesis, we then provide an in-depth study of recent weight-sharing NAS algorithms. First, we discover a phenomenon in the weight-sharing NAS training pipeline, which we dub multi-model forgetting, that negatively impacts the super-net quality, and propose a statistically motivated approach to address it. Subsequently, we find that (i) on average, many popular NAS algorithms perform similarly to a random architecture sampling policy; (ii) the widely-adopted weight sharing strategy degrades the ranking of the NAS candidates to the point of not reflecting their true performance, thus reducing the effectiveness of the search process. We then further decouple weight sharing from the NAS sampling policy, and isolate 14 factors that play a key role in the success of super-net training. Finally, to improve the super-net quality, we propose a regularization term that aims to maximize the correlation between the performance rankings of the super-net and of the stand-alone architectures using a small set of landmark architectures.},
keywords = {},
pubstate = {published},
tppubtype = {phdthesis}
}
Artificial intelligence has been an ultimate design goal since the inception of computers decades ago. Among the many attempts towards general artificial intelligence, modern machine learning successfully tackles many complex problems thanks to the progress in deep learning, and in particular in convolutional neural networks (CNN). To design a CNN for a specific task, one common approach consists of adapting the heuristics from the pre-deep-learning era to the CNN domain. In the first part of this thesis, we introduce two methods that follow this approach: i) We build a covariance descriptor, i.e., a local descriptor that is suitable for texture recognition, to replace the first-order fully connected layers in an ordinary CNN, showing that such a descriptor yields state-of-the-art performance on many fine-grained image classification tasks with orders of magnitude fewer feature dimensions; ii) we develop a light-weight recurrent U-Net for image semantic segmentation, inspired by the biological eye saccadic movements, that yields real-time predictions on devices with limited computational resources. As most methods pre-dating automatic machine learning (AutoML), the two above-mentioned CNNs were human-designed. In the past few years, however, neural architecture search~(NAS), which aims to facilitate the design of deep networks for new tasks, has drawn an increasing attention. In this context, the weight-sharing approach, which consists of utilizing a super-net to encompass all possible architectures within a search space, has become a de facto standard in NAS because it enables the search to be done on commodity hardware. In the second part of this thesis, we then provide an in-depth study of recent weight-sharing NAS algorithms. First, we discover a phenomenon in the weight-sharing NAS training pipeline, which we dub multi-model forgetting, that negatively impacts the super-net quality, and propose a statistically motivated approach to address it. Subsequently, we find that (i) on average, many popular NAS algorithms perform similarly to a random architecture sampling policy; (ii) the widely-adopted weight sharing strategy degrades the ranking of the NAS candidates to the point of not reflecting their true performance, thus reducing the effectiveness of the search process. We then further decouple weight sharing from the NAS sampling policy, and isolate 14 factors that play a key role in the success of super-net training. Finally, to improve the super-net quality, we propose a regularization term that aims to maximize the correlation between the performance rankings of the super-net and of the stand-alone architectures using a small set of landmark architectures.
1479.
Siems, Julien; Klein, Aaron; Archambeau, Cedric; Mahsereci, Maren
Dynamic Pruning of a Neural Networkvia Gradient Signal-to-Noise Ratio Inproceedings
In: 8th ICML Workshop on Automated Machine Learning (2021), 2021.
@inproceedings{Siems2021icml,
title = {Dynamic Pruning of a Neural Networkvia Gradient Signal-to-Noise Ratio},
author = {Julien Siems and Aaron Klein and Cedric Archambeau and Maren Mahsereci},
url = {https://openreview.net/pdf?id=34awaeWZgya},
year = {2021},
date = {2021-07-01},
booktitle = {8th ICML Workshop on Automated Machine Learning (2021)},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
1478.
Tian, Ye; Peng, Shichen; Yang, Shangshang; Zhang, Xingyi; Tan, Kay Chen; Jin, Yaochu
Action Command Encoding for SurrogateAssisted Neural Architecture Search Technical Report
2021.
@techreport{Tian2921,
title = {Action Command Encoding for SurrogateAssisted Neural Architecture Search},
author = {Ye Tian and Shichen Peng and Shangshang Yang and Xingyi Zhang and Kay Chen Tan and Yaochu Jin},
url = {https://www.researchgate.net/profile/Ye-Tian-84/publication/353166978_Action_Command_Encoding_for_Surrogate_Assisted_Neural_Architecture_Search/links/60eb072b1c28af34586047a1/Action-Command-Encoding-for-Surrogate-Assisted-Neural-Architecture-Search.pdf},
year = {2021},
date = {2021-07-01},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1477.
Sääw, Daniel; Nijweide, Frederik
Exploring the feasibility of Weight Agnostic Neural Networks for low-end hardware Technical Report
2021.
@techreport{Sääw2021,
title = {Exploring the feasibility of Weight Agnostic Neural Networks for low-end hardware},
author = {Daniel Sääw and Frederik Nijweide},
url = {https://www.researchgate.net/profile/Fpj_Nijweide/publication/352738002_Exploring_the_feasibility_of_Weight_Agnostic_Neural_Networks_for_low-end_hardware/links/60d630d5a6fdccb745e418ea/Exploring-the-feasibility-of-Weight-Agnostic-Neural-Networks-for-low-end-hardware.pdf},
year = {2021},
date = {2021-07-01},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1476.
White, Colin; Nolen, Sam; Savani, Yash
Exploring the Loss Landscape in Neural Architecture Search Inproceedings
In: 37th Conference on Uncertainty in Artificial Intelligence (UAI 2021), 2021.
@inproceedings{white2021,
title = {Exploring the Loss Landscape in Neural Architecture Search},
author = {Colin White and Sam Nolen and Yash Savani},
url = {https://auai.org/uai2021/pdf/uai2021.242.pdf},
year = {2021},
date = {2021-07-01},
booktitle = {37th Conference on Uncertainty in Artificial Intelligence (UAI 2021)},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
1475.
Lin, Yi; Liu, Yanfei; Liu, Jingguang; Liu, Guocai; Ma, Kai; Zheng, Yefeng
LE-NAS: Learning-based Ensemble with Neural Architecture Search for 3D Radiotherapy Dose Prediction Journal Article
In: 2021.
@article{Lin2106,
title = {LE-NAS: Learning-based Ensemble with Neural Architecture Search for 3D Radiotherapy Dose Prediction},
author = {Yi Lin and Yanfei Liu and Jingguang Liu and Guocai Liu and Kai Ma and Yefeng Zheng},
url = {https://arxiv.org/pdf/2106.06733.pdf},
year = {2021},
date = {2021-07-01},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
1474.
Dimanov, Daniel; Balaguer-Ballester, Emili; Singleton, Colin; Rostami, Shahin
Moncae: Multi-Objective Neuroevolution of Convolutional Autoencoders Inproceedings
In: 2nd Workshop on Neural Architecture Search at ICLR 2021, 2021.
@inproceedings{Dimanov2021,
title = {Moncae: Multi-Objective Neuroevolution of Convolutional Autoencoders},
author = {Daniel Dimanov and Emili Balaguer-Ballester and Colin Singleton and Shahin Rostami},
url = {https://www.researchgate.net/profile/Daniel-Dimanov-2/publication/352248797_MONCAE_MultiI-Objective_Neuroevolution_of_Convolutional_Autoencoders/links/60c0a43592851ca6f8d2d98b/MONCAE-MultiI-Objective-Neuroevolution-of-Convolutional-Autoencoders.pdf},
year = {2021},
date = {2021-07-01},
booktitle = {2nd Workshop on Neural Architecture Search at ICLR 2021},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
1473.
Jin, Yaochu; Wang, Handing; Sun, Chaoli
Surrogate-Assisted Evolutionary Neural Architecture Search Inproceedings
In: Data-Driven Evolutionary Optimization, pp. 373-387, 2021, ISBN: 978-3-030-74640-7.
@inproceedings{Jin2021,
title = {Surrogate-Assisted Evolutionary Neural Architecture Search},
author = {Yaochu Jin and Handing Wang and Chaoli Sun},
url = {https://link.springer.com/chapter/10.1007/978-3-030-74640-7_12},
isbn = {978-3-030-74640-7},
year = {2021},
date = {2021-06-29},
booktitle = {Data-Driven Evolutionary Optimization},
volume = {975},
pages = {373-387},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
1472.
Zhu, Wei; Ni, Yuan; Wang, Xiaoling; Xie, Guotong; Zhang, Fang
Discovering Better Model Architectures for Medical Query Understanding Inproceedings
In: Proceedings of NAACL HLT 2021: IndustryTrack Papers, pp. 230-237, 2021.
@inproceedings{ZHU2021,
title = {Discovering Better Model Architectures for Medical Query Understanding},
author = {
Wei Zhu and Yuan Ni and Xiaoling Wang and Guotong Xie and Fang Zhang},
url = {https://www.aclweb.org/anthology/2021.naacl-industry.29.pdf},
year = {2021},
date = {2021-06-06},
booktitle = {Proceedings of NAACL HLT 2021: IndustryTrack Papers},
pages = {230-237},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
1471.
Dai, Xiaoliang; Wan, Alvin; Zhang, Peizhao; Wu, Bichen; He, Zijian; Wei, Zhen; Chen, Kan; Tian, Yuandong; Yu, Matthew; Vajda, Peter; Gonzalez, Joseph E.
FBNetV3: Joint Architecture-Recipe Search using Predictor Pretraining Incollection
In: CVPR2021, 2021.
@incollection{DaiCVPR2021,
title = {FBNetV3: Joint Architecture-Recipe Search using Predictor Pretraining},
author = {Xiaoliang Dai and Alvin Wan and Peizhao Zhang and Bichen Wu and Zijian He and Zhen Wei and Kan Chen and Yuandong Tian and Matthew Yu and Peter Vajda and Joseph E. Gonzalez},
url = {https://openaccess.thecvf.com/content/CVPR2021/papers/Dai_FBNetV3_Joint_Architecture-Recipe_Search_Using_Predictor_Pretraining_CVPR_2021_paper.pdf},
year = {2021},
date = {2021-06-01},
booktitle = {CVPR2021},
keywords = {},
pubstate = {published},
tppubtype = {incollection}
}
1470.
Ma, Xiaohan; Si, Chang; Wang, Ying; Liu, Cheng; Zhang1, Lei
NASA: Accelerating Neural Network Design with a NAS Processor Inproceedings
In: 2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA), 2021.
@inproceedings{MaISCA2021,
title = {NASA: Accelerating Neural Network Design with a NAS Processor},
author = {Xiaohan Ma and Chang Si and Ying Wang and Cheng Liu and Lei Zhang1},
url = {https://conferences.computer.org/iscapub/pdfs/ISCA2021-4ghucdBnCWYB7ES2Pe4YdT/333300a790/333300a790.pdf},
year = {2021},
date = {2021-06-01},
booktitle = {2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA)},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
1469.
Liu, Shaoli; Zheng, Chengjian; Lu, Kaidi; Gao, Si; Wang, Ning; Wang, Bofei; Zhang, Diankai; Zhang, Xiaofeng; Xu, Tianyu
EVSRNet: Efficient Video Super-Resolution With Neural Architecture Search Inproceedings
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 2480-2485, 2021.
@inproceedings{Liu_2021_CVPR,
title = {EVSRNet: Efficient Video Super-Resolution With Neural Architecture Search},
author = {Shaoli Liu and Chengjian Zheng and Kaidi Lu and Si Gao and Ning Wang and Bofei Wang and Diankai Zhang and Xiaofeng Zhang and Tianyu Xu},
url = {https://openaccess.thecvf.com/content/CVPR2021W/MAI/html/Liu_EVSRNet_Efficient_Video_Super-Resolution_With_Neural_Architecture_Search_CVPRW_2021_paper.html},
year = {2021},
date = {2021-06-01},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
pages = {2480-2485},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
1468.
Geraeinejad, V.; Seenan, S.; Modarressi, M.; Daneshtalab, M.
RoCo-NAS : Robust and Compact Neural Architecture Search Inproceedings
In: The international joint conference on neural networks IJCNN, 2021.
@inproceedings{Geraeinejad2021,
title = {RoCo-NAS : Robust and Compact Neural Architecture Search},
author = {V. Geraeinejad and S. Seenan and M. Modarressi and M. Daneshtalab},
url = {https://www.diva-portal.org/smash/record.jsf?pid=diva2%3A1558970&dswid=-1593},
year = {2021},
date = {2021-06-01},
booktitle = {The international joint conference on neural networks IJCNN},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
1467.
Tsokov, Stefan; Lazarova, Milena; Aleksieva-Petrova, Adelina
An Evolutionary Approach to the Design of Convolutional Neural Networks for Human Activity Recognition Journal Article
In: Indian Journal of Computer Science and Engineering, 2021.
@article{Tsokov2021,
title = {An Evolutionary Approach to the Design of Convolutional Neural Networks for Human Activity Recognition},
author = {Tsokov, Stefan and Lazarova, Milena and Aleksieva-Petrova, Adelina},
url = {http://www.ijcse.com/docs/INDJCSE21-12-02-145.pdf},
year = {2021},
date = {2021-06-01},
journal = {Indian Journal of Computer Science and Engineering},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
1466.
Huang, Shenyang; Francois-Lavet, Vincent; Rabusseau, Guillaume
Understanding Capacity Saturation in Incremental Learning Inproceedings
In: The 34th Canadian Conference on Artificial Intelligence, 2021.
@inproceedings{Huang2021,
title = {Understanding Capacity Saturation in Incremental Learning},
author = {Shenyang Huang and Vincent Francois-Lavet and Guillaume Rabusseau},
url = {https://assets.pubpub.org/xabw479b/51621564316955.pdf},
year = {2021},
date = {2021-05-25},
booktitle = {The 34th Canadian Conference on Artificial Intelligence},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}