Maintained by Difan Deng and Marius Lindauer.
The following list considers papers related to neural architecture search. It is by no means complete. If you miss a paper on the list, please let us know.
Please note that although NAS methods steadily improve, the quality of empirical evaluations in this field are still lagging behind compared to other areas in machine learning, AI and optimization. We would therefore like to share some best practices for empirical evaluations of NAS methods, which we believe will facilitate sustained and measurable progress in the field. If you are interested in a teaser, please read our blog post or directly jump to our checklist.
Transformers have gained increasing popularity in different domains. For a comprehensive list of papers focusing on Neural Architecture Search for Transformer-Based spaces, the awesome-transformer-search repo is all you need.
2023
2936.
Souquet, Léo; Shvai, Nadiya; Llanza, Arcadi; Nakib, Amir
Convolutional neural network architecture search based on fractal decomposition optimization algorithm Journal Article
In: Expert Systems with Applications, vol. 213, pp. 118947, 2023, ISSN: 0957-4174.
@article{SOUQUET2023118947,
title = {Convolutional neural network architecture search based on fractal decomposition optimization algorithm},
author = {Léo Souquet and Nadiya Shvai and Arcadi Llanza and Amir Nakib},
url = {https://www.sciencedirect.com/science/article/pii/S0957417422019650},
doi = {https://doi.org/10.1016/j.eswa.2022.118947},
issn = {0957-4174},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {Expert Systems with Applications},
volume = {213},
pages = {118947},
abstract = {This paper presents a new approach to design the architecture and optimize the hyperparameters of a deep convolutional neural network (CNN) via of the Fractal Decomposition Algorithm (FDA). This optimization algorithm was recently proposed to solve continuous optimization problems. It is based on a geometric fractal decomposition that divides the search space while searching for the best solution possible. As FDA is effective in single-objective optimization, in this work we aim to prove that it can also be successfully applied to fine-tuning deep neural network architectures. Moreover, a new formulation based on bi-level optimization is proposed to separate the architecture search composed of discrete parameters from hyperparameters’ optimization. This is motivated by the fact that automating the construction of deep neural architecture has been an important focus over recent years as manual construction is considerably time-consuming, error-prone, and requires in-depth knowledge. To solve the bi-level problem thus formulated, a random search is performed aiming to create a set of candidate architectures. Then, the best ones are finetuned using FDA. CIFAR-10 and CIFAR-100 benchmarks were used to evaluate the performance of the proposed approach. The results obtained are among the state of the art in the corresponding class of networks (low number of parameters and chained-structured CNN architectures). The results are emphasized by the fact that the whole process was performed using low computing power with only 3 NVIDIA V100 GPUs. The source code is available at https://github.com/alc1218/Convolutional-Neural-Network-Architecture-Search-Based-on-Fractal-Decomposition-Optimization.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
This paper presents a new approach to design the architecture and optimize the hyperparameters of a deep convolutional neural network (CNN) via of the Fractal Decomposition Algorithm (FDA). This optimization algorithm was recently proposed to solve continuous optimization problems. It is based on a geometric fractal decomposition that divides the search space while searching for the best solution possible. As FDA is effective in single-objective optimization, in this work we aim to prove that it can also be successfully applied to fine-tuning deep neural network architectures. Moreover, a new formulation based on bi-level optimization is proposed to separate the architecture search composed of discrete parameters from hyperparameters’ optimization. This is motivated by the fact that automating the construction of deep neural architecture has been an important focus over recent years as manual construction is considerably time-consuming, error-prone, and requires in-depth knowledge. To solve the bi-level problem thus formulated, a random search is performed aiming to create a set of candidate architectures. Then, the best ones are finetuned using FDA. CIFAR-10 and CIFAR-100 benchmarks were used to evaluate the performance of the proposed approach. The results obtained are among the state of the art in the corresponding class of networks (low number of parameters and chained-structured CNN architectures). The results are emphasized by the fact that the whole process was performed using low computing power with only 3 NVIDIA V100 GPUs. The source code is available at https://github.com/alc1218/Convolutional-Neural-Network-Architecture-Search-Based-on-Fractal-Decomposition-Optimization.
2935.
Wen, Yingpeng; Yu, Weijiang; Li, Dongsheng; Du, Jiangsu; Huang, Dan; Xiao, Nong
CosNAS: Enhancing estimation on cosmological parameters via neural architecture search Journal Article
In: New Astronomy, vol. 99, pp. 101955, 2023, ISSN: 1384-1076.
@article{WEN2023101955,
title = {CosNAS: Enhancing estimation on cosmological parameters via neural architecture search},
author = {Yingpeng Wen and Weijiang Yu and Dongsheng Li and Jiangsu Du and Dan Huang and Nong Xiao},
url = {https://www.sciencedirect.com/science/article/pii/S1384107622001373},
doi = {https://doi.org/10.1016/j.newast.2022.101955},
issn = {1384-1076},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {New Astronomy},
volume = {99},
pages = {101955},
abstract = {A great challenge of cosmology is estimating the cosmological parameters of the universe. With the development of deep learning, scientists adopt 3D deep neural networks to estimate cosmological parameters from the large-scale dark matter distribution of the universe, but these methods are time-consuming to design and train neural networks. While neural architecture search is an emerging approach to estimate cosmological parameters with its capability of automatically designing neural networks, the 3D operations on a 3D dataset prohibit the usage of traditional neural architecture search methods, due to its overwhelming time and memory consumption. To tackle these issues, we propose an efficient method, CosNAS, that can automatically design neural networks with 2D operations to estimate the cosmological parameters. In addition, processing 3D data with 2D operations will inevitably cause the loss of spatial information, thus we propose an efficient SABlock to retain more 3D spatial information. We also customize a space-focused search space to focus on important information in the dark matter distribution. The experimental results indicate that our estimation of the cosmological parameters Ω, σ and n, can be applied to large-scale 3D dark matter distribution and speedup the network search by 800x. The average relative errors of cosmological parameter estimations are (0.00163, 0.00065, 0.00080), significantly decreasing the average error of estimation by 85.5% compared to previous work.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
A great challenge of cosmology is estimating the cosmological parameters of the universe. With the development of deep learning, scientists adopt 3D deep neural networks to estimate cosmological parameters from the large-scale dark matter distribution of the universe, but these methods are time-consuming to design and train neural networks. While neural architecture search is an emerging approach to estimate cosmological parameters with its capability of automatically designing neural networks, the 3D operations on a 3D dataset prohibit the usage of traditional neural architecture search methods, due to its overwhelming time and memory consumption. To tackle these issues, we propose an efficient method, CosNAS, that can automatically design neural networks with 2D operations to estimate the cosmological parameters. In addition, processing 3D data with 2D operations will inevitably cause the loss of spatial information, thus we propose an efficient SABlock to retain more 3D spatial information. We also customize a space-focused search space to focus on important information in the dark matter distribution. The experimental results indicate that our estimation of the cosmological parameters Ω, σ and n, can be applied to large-scale 3D dark matter distribution and speedup the network search by 800x. The average relative errors of cosmological parameter estimations are (0.00163, 0.00065, 0.00080), significantly decreasing the average error of estimation by 85.5% compared to previous work.
2934.
Li, Chuanyou; Zhang, Kun; Li, Yifan; Shang, Jiangwei; Zhang, Xinyue; Qian, Lei
ANNA: Accelerating Neural Network Accelerator through software-hardware co-design for vertical applications in edge systems Journal Article
In: Future Generation Computer Systems, vol. 140, pp. 91-103, 2023, ISSN: 0167-739X.
@article{LI202391,
title = {ANNA: Accelerating Neural Network Accelerator through software-hardware co-design for vertical applications in edge systems},
author = {Chuanyou Li and Kun Zhang and Yifan Li and Jiangwei Shang and Xinyue Zhang and Lei Qian},
url = {https://www.sciencedirect.com/science/article/pii/S0167739X22003168},
doi = {https://doi.org/10.1016/j.future.2022.10.001},
issn = {0167-739X},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {Future Generation Computer Systems},
volume = {140},
pages = {91-103},
abstract = {In promising edge systems, AI algorithms and their hardware implementations are often joint optimized as integrated solutions to solve end-to-end design problems. Joint optimization depends on a delicate co-design of software and hardware. According to our knowledge, current co-design methodologies are still coarse-grained. In this paper, we proposed ANNA: Accelerating Neural Network Accelerator through a novel software-hardware co-design methodology. ANNA is a framework composed of three components: ANNA-NAS (Neural Architecture Search), ANNA-ARCH (hardware ARCHitecture) and ANNA-PERF (PERFormance optimizer & evaluator). ANNA-NAS adopts a cell-wise structure and is designed to be hardware aware. It aims at generating a neural network having high inference accuracy and low inference latency. To avoid tremendous time costs, ANNA-NAS synthetically uses differentiable architecture search and early stopping techniques. ANNA-ARCH starts to be designed as long as the architecture search space is defined. Based on the cell-wise structure, ANNA-ARCH specifies its main body which includes Convolution units, Activation Router and Buffer Pool. To well support different neural networks that could be generated by ANNA-NAS, the detailed part of ANNA-ARCH is configurable. ANNA-PERF harmonizes the co-design of ANNA-NAS and ANNA-ARCH. It takes a neural network and a hardware architecture as inputs. After optimizing the mapping strategy between the neural network and hardware accelerator, it feeds back a cycle-accurate latency to ANNA-NAS. Aiming at image classification, we carried out the experiments on ImageNet. Experimental results demonstrate that without loss of much inference accuracy, ANNA wins a significant low inference latency through a harmonious software and hardware co-design.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
In promising edge systems, AI algorithms and their hardware implementations are often joint optimized as integrated solutions to solve end-to-end design problems. Joint optimization depends on a delicate co-design of software and hardware. According to our knowledge, current co-design methodologies are still coarse-grained. In this paper, we proposed ANNA: Accelerating Neural Network Accelerator through a novel software-hardware co-design methodology. ANNA is a framework composed of three components: ANNA-NAS (Neural Architecture Search), ANNA-ARCH (hardware ARCHitecture) and ANNA-PERF (PERFormance optimizer & evaluator). ANNA-NAS adopts a cell-wise structure and is designed to be hardware aware. It aims at generating a neural network having high inference accuracy and low inference latency. To avoid tremendous time costs, ANNA-NAS synthetically uses differentiable architecture search and early stopping techniques. ANNA-ARCH starts to be designed as long as the architecture search space is defined. Based on the cell-wise structure, ANNA-ARCH specifies its main body which includes Convolution units, Activation Router and Buffer Pool. To well support different neural networks that could be generated by ANNA-NAS, the detailed part of ANNA-ARCH is configurable. ANNA-PERF harmonizes the co-design of ANNA-NAS and ANNA-ARCH. It takes a neural network and a hardware architecture as inputs. After optimizing the mapping strategy between the neural network and hardware accelerator, it feeds back a cycle-accurate latency to ANNA-NAS. Aiming at image classification, we carried out the experiments on ImageNet. Experimental results demonstrate that without loss of much inference accuracy, ANNA wins a significant low inference latency through a harmonious software and hardware co-design.
2933.
Zharikov, Ilia; Krivorotov, Ivan; Maximov, Egor; Korviakov, Vladimir; Letunovskiy, Alexey
Ä Review of One-Shot Neural Architecture Search Methods Proceedings Article
In: Kryzhanovsky, Boris; Dunin-Barkowski, Witali; Redko, Vladimir; Tiumentsev, Yury (Ed.): Ädvances in Neural Computation, Machine Learning, and Cognitive Research VI", pp. 130–147, Springer International Publishing, Cham, 2023, ISBN: 978-3-031-19032-2.
@inproceedings{10.1007/978-3-031-19032-2_14,
title = {Ä Review of One-Shot Neural Architecture Search Methods},
author = {Ilia Zharikov and Ivan Krivorotov and Egor Maximov and Vladimir Korviakov and Alexey Letunovskiy},
editor = {Boris Kryzhanovsky and Witali Dunin-Barkowski and Vladimir Redko and Yury Tiumentsev},
isbn = {978-3-031-19032-2},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
booktitle = {Ädvances in Neural Computation, Machine Learning, and Cognitive Research VI"},
pages = {130--147},
publisher = {Springer International Publishing},
address = {Cham},
abstract = {Neural network architecture design is a challenging and computational expensive problem. For this reason training a one-shot model becomes very popular way to obtain several architectures or find the best according to different requirements without retraining. In this paper we summarize the existing one-shot NAS methods, highlight base concepts and compare considered methods in terms of accuracy, number of needed for training GPU hours and ranking quality.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Neural network architecture design is a challenging and computational expensive problem. For this reason training a one-shot model becomes very popular way to obtain several architectures or find the best according to different requirements without retraining. In this paper we summarize the existing one-shot NAS methods, highlight base concepts and compare considered methods in terms of accuracy, number of needed for training GPU hours and ranking quality.
2932.
Kolganov, Pavel A.; Tiumentsev, Yury V.
Än Attempt to Formalize the Formulation of the Network Architecture Search Problem for Convolutional Neural Networks Proceedings Article
In: Kryzhanovsky, Boris; Dunin-Barkowski, Witali; Redko, Vladimir; Tiumentsev, Yury (Ed.): Ädvances in Neural Computation, Machine Learning, and Cognitive Research VI", pp. 550–556, Springer International Publishing, Cham, 2023, ISBN: 978-3-031-19032-2.
@inproceedings{10.1007/978-3-031-19032-2_55,
title = {Än Attempt to Formalize the Formulation of the Network Architecture Search Problem for Convolutional Neural Networks},
author = {Pavel A. Kolganov and Yury V. Tiumentsev},
editor = {Boris Kryzhanovsky and Witali Dunin-Barkowski and Vladimir Redko and Yury Tiumentsev},
isbn = {978-3-031-19032-2},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
booktitle = {Ädvances in Neural Computation, Machine Learning, and Cognitive Research VI"},
pages = {550--556},
publisher = {Springer International Publishing},
address = {Cham},
abstract = {The paper deals with the problem of searching for a neural network architecture. The paper presents a mathematical formulation of the problem of searching a neural network model, optimal from the point of view of a predefined criterion. The analysis of components of this problem is given. Some difficulties encountered by researchers when solving the NAS problem are described. A computational experiment is conducted, which consists in the search of a neural network architecture on the MNIST dataset.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
The paper deals with the problem of searching for a neural network architecture. The paper presents a mathematical formulation of the problem of searching a neural network model, optimal from the point of view of a predefined criterion. The analysis of components of this problem is given. Some difficulties encountered by researchers when solving the NAS problem are described. A computational experiment is conducted, which consists in the search of a neural network architecture on the MNIST dataset.
2931.
Graham-Knight, John Brandon; Bond, Corey; Najjaran, Homayoun; Lucet, Yves; Lasserre, Patricia
Predicting and explaining performance and diversity of neural network architecture for semantic segmentation Journal Article
In: Expert Systems with Applications, vol. 214, pp. 119101, 2023, ISSN: 0957-4174.
@article{GRAHAMKNIGHT2023119101,
title = {Predicting and explaining performance and diversity of neural network architecture for semantic segmentation},
author = {John Brandon Graham-Knight and Corey Bond and Homayoun Najjaran and Yves Lucet and Patricia Lasserre},
url = {https://www.sciencedirect.com/science/article/pii/S0957417422021194},
doi = {https://doi.org/10.1016/j.eswa.2022.119101},
issn = {0957-4174},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {Expert Systems with Applications},
volume = {214},
pages = {119101},
abstract = {This paper proposes searching for network architectures which achieve similar performance while promoting diversity, in order to facilitate ensembling. We explain prediction performance and diversity of various network sizes and activation functions applied to semantic segmentation of the CityScapes dataset. We show that both performance and diversity can be predicted from neural network architecture using explainable boosting machines. A grid search of 144 models is performed, and many of the models exhibit no significant difference in mean performance within a 95% confidence interval. Notably, we find the best performing models have varied network architecture parameters. The explanations for performance largely agree with the accepted wisdom of the machine learning community, which shows that the method is extracting information of value. We find that diversity between models can be achieved by varying network size. Moreover, homogeneous network sizes generally show positive correlation in predictions, and larger models tend to converge to similar solutions. These explanations provide a better understanding of the effects of network parameters to deep learning practitioners; they could also be used in place of naïve search methods or a model pool to inform growing an ensemble.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
This paper proposes searching for network architectures which achieve similar performance while promoting diversity, in order to facilitate ensembling. We explain prediction performance and diversity of various network sizes and activation functions applied to semantic segmentation of the CityScapes dataset. We show that both performance and diversity can be predicted from neural network architecture using explainable boosting machines. A grid search of 144 models is performed, and many of the models exhibit no significant difference in mean performance within a 95% confidence interval. Notably, we find the best performing models have varied network architecture parameters. The explanations for performance largely agree with the accepted wisdom of the machine learning community, which shows that the method is extracting information of value. We find that diversity between models can be achieved by varying network size. Moreover, homogeneous network sizes generally show positive correlation in predictions, and larger models tend to converge to similar solutions. These explanations provide a better understanding of the effects of network parameters to deep learning practitioners; they could also be used in place of naïve search methods or a model pool to inform growing an ensemble.
2930.
Sun, Yanan; Yen, Gary G.; Zhang, Mengjie
End-to-End Performance Predictors Book Chapter
In: Evolutionary Deep Neural Architecture Search: Fundamentals, Methods, and Recent Advances, pp. 237–255, Springer International Publishing, Cham, 2023, ISBN: 978-3-031-16868-0.
@inbook{Sun2023,
title = {End-to-End Performance Predictors},
author = {Yanan Sun and Gary G. Yen and Mengjie Zhang},
url = {https://doi.org/10.1007/978-3-031-16868-0_14},
doi = {10.1007/978-3-031-16868-0_14},
isbn = {978-3-031-16868-0},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
booktitle = {Evolutionary Deep Neural Architecture Search: Fundamentals, Methods, and Recent Advances},
pages = {237--255},
publisher = {Springer International Publishing},
address = {Cham},
abstract = {In fact, common optimization problems in ENAS are computationally expensive and are usually handled using surrogate-assisted EAs(SAEAs) [1], employing inexpensive approximation regression and classification models, such as the Gaussian process model [2], radial basis network (RBN), etc., to replace the costly fitness evaluation [3]. SAEAs have proven to be useful and efficient in a variety of practical optimization applications [1].},
keywords = {},
pubstate = {published},
tppubtype = {inbook}
}
In fact, common optimization problems in ENAS are computationally expensive and are usually handled using surrogate-assisted EAs(SAEAs) [1], employing inexpensive approximation regression and classification models, such as the Gaussian process model [2], radial basis network (RBN), etc., to replace the costly fitness evaluation [3]. SAEAs have proven to be useful and efficient in a variety of practical optimization applications [1].
2929.
Sun, Yanan; Yen, Gary G.; Zhang, Mengjie
Hybrid GA and PSO for Architecture Design Book Chapter
In: Evolutionary Deep Neural Architecture Search: Fundamentals, Methods, and Recent Advances, pp. 171–180, Springer International Publishing, Cham, 2023, ISBN: 978-3-031-16868-0.
@inbook{Sun2023b,
title = {Hybrid GA and PSO for Architecture Design},
author = {Yanan Sun and Gary G. Yen and Mengjie Zhang},
url = {https://doi.org/10.1007/978-3-031-16868-0_9},
doi = {10.1007/978-3-031-16868-0_9},
isbn = {978-3-031-16868-0},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
booktitle = {Evolutionary Deep Neural Architecture Search: Fundamentals, Methods, and Recent Advances},
pages = {171--180},
publisher = {Springer International Publishing},
address = {Cham},
abstract = {In this chapter, a new approach based on EC is introduced for automatically searching for the optimal CNN architecture and determining whether or not to use shortcut connections between one layer and its forward layer. After that, a two-level encoding strategy is applied to a hybrid EC methodology that is composed of a GA and a PSO. This allows for the generation of both the network architecture and the shortcut connections within it. The technique is referred to as DynamicNet because to the fact that during the course of the evolutionary process, both the architecture and the shortcut connections are determined dynamically without any involvement from a human being. On three widely used datasets that have differing degrees of complexity, DynamicNet will be evaluated in comparison with one method that is based on EC and 12 methods that are considered to be state-of-the-art.},
keywords = {},
pubstate = {published},
tppubtype = {inbook}
}
In this chapter, a new approach based on EC is introduced for automatically searching for the optimal CNN architecture and determining whether or not to use shortcut connections between one layer and its forward layer. After that, a two-level encoding strategy is applied to a hybrid EC methodology that is composed of a GA and a PSO. This allows for the generation of both the network architecture and the shortcut connections within it. The technique is referred to as DynamicNet because to the fact that during the course of the evolutionary process, both the architecture and the shortcut connections are determined dynamically without any involvement from a human being. On three widely used datasets that have differing degrees of complexity, DynamicNet will be evaluated in comparison with one method that is based on EC and 12 methods that are considered to be state-of-the-art.
2928.
Sun, Yanan; Yen, Gary G.; Zhang, Mengjie
Architecture Design for Skip-Connection Based CNNs Book Chapter
In: Evolutionary Deep Neural Architecture Search: Fundamentals, Methods, and Recent Advances, pp. 147–170, Springer International Publishing, Cham, 2023, ISBN: 978-3-031-16868-0.
@inbook{Sun2023c,
title = {Architecture Design for Skip-Connection Based CNNs},
author = {Yanan Sun and Gary G. Yen and Mengjie Zhang},
url = {https://doi.org/10.1007/978-3-031-16868-0_8},
doi = {10.1007/978-3-031-16868-0_8},
isbn = {978-3-031-16868-0},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
booktitle = {Evolutionary Deep Neural Architecture Search: Fundamentals, Methods, and Recent Advances},
pages = {147--170},
publisher = {Springer International Publishing},
address = {Cham},
abstract = {In this chapter, an efficient and effective algorithms employing GA is introduced, dubbed CNN-GA, to find the best CNN architectures for specific image classification tasks automatically, such that the found CNN can be directly employed without any need for manual tuning. CNN-GA is an algorithm for automating the architecture design of CNN. Please keep in note that the terms ``automatic'' and ``automatic + manually + tuning'' are discussed from the perspective of end-users, rather than developers. In developing high-performance CNN architecture design algorithms, however, adequate domain expertise should be promoted. This effort is not difficult to comprehend by comparing it to the design of the Windows Operating System by Microsoft scientists: to ensure the users could be able to effectively operate on computers even if they do not have considerable understanding of operating systems, the scientists should put as much of their professional knowledge as they possibly can while building a user-friendly operating system.},
keywords = {},
pubstate = {published},
tppubtype = {inbook}
}
In this chapter, an efficient and effective algorithms employing GA is introduced, dubbed CNN-GA, to find the best CNN architectures for specific image classification tasks automatically, such that the found CNN can be directly employed without any need for manual tuning. CNN-GA is an algorithm for automating the architecture design of CNN. Please keep in note that the terms ``automatic'' and ``automatic + manually + tuning'' are discussed from the perspective of end-users, rather than developers. In developing high-performance CNN architecture design algorithms, however, adequate domain expertise should be promoted. This effort is not difficult to comprehend by comparing it to the design of the Windows Operating System by Microsoft scientists: to ensure the users could be able to effectively operate on computers even if they do not have considerable understanding of operating systems, the scientists should put as much of their professional knowledge as they possibly can while building a user-friendly operating system.
2927.
Sun, Yanan; Yen, Gary G.; Zhang, Mengjie
Differential Evolution for Architecture Design Book Chapter
In: Evolutionary Deep Neural Architecture Search: Fundamentals, Methods, and Recent Advances, pp. 193–202, Springer International Publishing, Cham, 2023, ISBN: 978-3-031-16868-0.
@inbook{Sun2023d,
title = {Differential Evolution for Architecture Design},
author = {Yanan Sun and Gary G. Yen and Mengjie Zhang},
url = {https://doi.org/10.1007/978-3-031-16868-0_11},
doi = {10.1007/978-3-031-16868-0_11},
isbn = {978-3-031-16868-0},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
booktitle = {Evolutionary Deep Neural Architecture Search: Fundamentals, Methods, and Recent Advances},
pages = {193--202},
publisher = {Springer International Publishing},
address = {Cham},
abstract = {The general goal of this chapter is to explore the capacity of DE, named DECNN, to evolve deep CNN architectures and parameters automatically. Designing new crossover and mutation operators of DE, as well as an encoding scheme, and a second crossover operator will help to achieve the goal. DECNN will be evaluated on six datasets of various complexity that are widely used and compared to 12 state-of-the-art methods.},
keywords = {},
pubstate = {published},
tppubtype = {inbook}
}
The general goal of this chapter is to explore the capacity of DE, named DECNN, to evolve deep CNN architectures and parameters automatically. Designing new crossover and mutation operators of DE, as well as an encoding scheme, and a second crossover operator will help to achieve the goal. DECNN will be evaluated on six datasets of various complexity that are widely used and compared to 12 state-of-the-art methods.
2926.
Jing, Kun; Chen, Luoyu; Xu, Jungang
An architecture entropy regularizer for differentiable neural architecture search Journal Article
In: Neural Networks, vol. 158, pp. 111-120, 2023, ISSN: 0893-6080.
@article{JING2023111,
title = {An architecture entropy regularizer for differentiable neural architecture search},
author = {Kun Jing and Luoyu Chen and Jungang Xu},
url = {https://www.sciencedirect.com/science/article/pii/S0893608022004567},
doi = {https://doi.org/10.1016/j.neunet.2022.11.015},
issn = {0893-6080},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {Neural Networks},
volume = {158},
pages = {111-120},
abstract = {Differentiable architecture search (DARTS) is one of the prevailing paradigms of neural architecture search (NAS) due to allowing efficient gradient-based optimization during the search phase. However, its poor stability and generalizability are intolerable. We argue that the crux is the locally optimal architecture parameter caused by a dilemma, which is that the solutions to the Matthew effect and discretization discrepancy are inconsistent. To escape from the dilemma, we propose an architecture entropy to measure the discrepancy of the architecture parameters of different candidate operations and use it as a regularizer to control the learning of architecture parameters. Extensive experiments show that an architecture entropy regularizer with a negative or positive coefficient can effectively solve one side of the contradiction respectively, and the regularizer with a variable coefficient can relieve DARTS from the dilemma. Experimental results demonstrate that our architecture entropy regularizer can significantly improve different differentiable NAS algorithms on different datasets and different search spaces. Furthermore, we also achieve more accurate and more robust results on CIFAR-10 and ImageNet. The code is publicly available at https://github.com/kunjing96/DARTS-AER.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Differentiable architecture search (DARTS) is one of the prevailing paradigms of neural architecture search (NAS) due to allowing efficient gradient-based optimization during the search phase. However, its poor stability and generalizability are intolerable. We argue that the crux is the locally optimal architecture parameter caused by a dilemma, which is that the solutions to the Matthew effect and discretization discrepancy are inconsistent. To escape from the dilemma, we propose an architecture entropy to measure the discrepancy of the architecture parameters of different candidate operations and use it as a regularizer to control the learning of architecture parameters. Extensive experiments show that an architecture entropy regularizer with a negative or positive coefficient can effectively solve one side of the contradiction respectively, and the regularizer with a variable coefficient can relieve DARTS from the dilemma. Experimental results demonstrate that our architecture entropy regularizer can significantly improve different differentiable NAS algorithms on different datasets and different search spaces. Furthermore, we also achieve more accurate and more robust results on CIFAR-10 and ImageNet. The code is publicly available at https://github.com/kunjing96/DARTS-AER.
2925.
Yao, Fengqin; Wang, Shengke; Ding, Laihui; Zhong, Guoqiang; Bullock, Leon Bevan; Xu, Zhiwei; Dong, Junyu
Lightweight network learning with Zero-Shot Neural Architecture Search for UAV images Journal Article
In: Knowledge-Based Systems, vol. 260, pp. 110142, 2023, ISSN: 0950-7051.
@article{YAO2023110142,
title = {Lightweight network learning with Zero-Shot Neural Architecture Search for UAV images},
author = {Fengqin Yao and Shengke Wang and Laihui Ding and Guoqiang Zhong and Leon Bevan Bullock and Zhiwei Xu and Junyu Dong},
url = {https://www.sciencedirect.com/science/article/pii/S0950705122012382},
doi = {https://doi.org/10.1016/j.knosys.2022.110142},
issn = {0950-7051},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {Knowledge-Based Systems},
volume = {260},
pages = {110142},
abstract = {Lightweight Network Architecture is essential for autonomous and intelligent monitoring of Unmanned Aerial Vehicles (UAVs), such as in object detection, image segmentation, and crowd counting applications. The state-of-the-art lightweight network learning based on Neural Architecture Search (NAS) usually costs enormous computation resources. Alternatively, low-performance embedded platforms and high-resolution drone images pose a challenge for lightweight network learning. To alleviate this problem, this paper proposes a new lightweight object detection model, called GhostShuffleNet (GSNet), for UAV images, which is built based on Zero-Shot Neural Architecture Search. This paper also introduces the new components which compose GSNet, namely GhostShuffle units (loosely based on ShuffleNetV2) and the backbone GSmodel-L. Firstly, a lightweight search space is constructed with the GhostShuffle (GS) units to reduce the parameters and floating-point operations (FLOPs). Secondly, the parameters, FLOPs, layers, and memory access cost (MAC) as constraints add to search strategy on a Zero-Shot Neural structure search algorithm, which then searches for an optimal network GSmodel-L. Finally, the optimal GSmodel-L is used as the backbone network and a Ghost-PAN feature fusion module and detection heads are added to complete the design of the lightweight object detection network (GSNet). Extensive experiments are conducted on the VisDrone2019 (14.92%mAP) dataset and the our UAV-OUC-DET (8.38%mAP) dataset demonstrating the efficiency and effectiveness of GSNet. The completed code is available at: https://github.com/yfq-yy/GSNet.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Lightweight Network Architecture is essential for autonomous and intelligent monitoring of Unmanned Aerial Vehicles (UAVs), such as in object detection, image segmentation, and crowd counting applications. The state-of-the-art lightweight network learning based on Neural Architecture Search (NAS) usually costs enormous computation resources. Alternatively, low-performance embedded platforms and high-resolution drone images pose a challenge for lightweight network learning. To alleviate this problem, this paper proposes a new lightweight object detection model, called GhostShuffleNet (GSNet), for UAV images, which is built based on Zero-Shot Neural Architecture Search. This paper also introduces the new components which compose GSNet, namely GhostShuffle units (loosely based on ShuffleNetV2) and the backbone GSmodel-L. Firstly, a lightweight search space is constructed with the GhostShuffle (GS) units to reduce the parameters and floating-point operations (FLOPs). Secondly, the parameters, FLOPs, layers, and memory access cost (MAC) as constraints add to search strategy on a Zero-Shot Neural structure search algorithm, which then searches for an optimal network GSmodel-L. Finally, the optimal GSmodel-L is used as the backbone network and a Ghost-PAN feature fusion module and detection heads are added to complete the design of the lightweight object detection network (GSNet). Extensive experiments are conducted on the VisDrone2019 (14.92%mAP) dataset and the our UAV-OUC-DET (8.38%mAP) dataset demonstrating the efficiency and effectiveness of GSNet. The completed code is available at: https://github.com/yfq-yy/GSNet.
2924.
Wang, Wenna; Zhang, Xiuwei; Cui, Hengfei; Yin, Hanlin; Zhang, Yannnig
FP-DARTS: Fast parallel differentiable neural architecture search for image classification Journal Article
In: Pattern Recognition, vol. 136, pp. 109193, 2023, ISSN: 0031-3203.
@article{WANG2023109193,
title = {FP-DARTS: Fast parallel differentiable neural architecture search for image classification},
author = {Wenna Wang and Xiuwei Zhang and Hengfei Cui and Hanlin Yin and Yannnig Zhang},
url = {https://www.sciencedirect.com/science/article/pii/S0031320322006720},
doi = {https://doi.org/10.1016/j.patcog.2022.109193},
issn = {0031-3203},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {Pattern Recognition},
volume = {136},
pages = {109193},
abstract = {Neural Architecture Search (NAS) has made remarkable progress in automatic machine learning. However, it still suffers massive computing overheads limiting its wide applications. In this paper, we present an efficient search method, Fast Parallel Differential Neural Architecture Search (FP-DARTS). The proposed method is carefully designed from three levels to construct and train the super-network. Firstly, at the operation-level, to reduce the computational burden, different from the standard DARTS search space (8 operations), we decompose the operation set into two non-overlapping operator sub-sets (4 operations for each). Adopting these two reduced search spaces, two over-parameterized sub-networks are constructed. Secondly, at the channel-level, the partially-connected strategy is adopted, where each sub-network only adopts partial channels. Then these two sub-networks construct a two-parallel-path super-network by addition. Thirdly, at the training-level, the binary gate is introduced to control whether a path participates in the super-network training. It may suffer an unfair issue when using softmax to select the best input for intermediate nodes across two operator sub-sets. To tackle this problem, the sigmoid function is introduced, which measures the performance of operations without compression. Extensive experiments demonstrate the effectiveness of the proposed algorithm. Specifically, FP-DARTS achieves 2.50% test error with only 0.08 GPU-days on CIFAR10, and a state-of-the-art top-1 error rate of 23.7% on ImageNet using only 2.44 GPU-days for search.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Neural Architecture Search (NAS) has made remarkable progress in automatic machine learning. However, it still suffers massive computing overheads limiting its wide applications. In this paper, we present an efficient search method, Fast Parallel Differential Neural Architecture Search (FP-DARTS). The proposed method is carefully designed from three levels to construct and train the super-network. Firstly, at the operation-level, to reduce the computational burden, different from the standard DARTS search space (8 operations), we decompose the operation set into two non-overlapping operator sub-sets (4 operations for each). Adopting these two reduced search spaces, two over-parameterized sub-networks are constructed. Secondly, at the channel-level, the partially-connected strategy is adopted, where each sub-network only adopts partial channels. Then these two sub-networks construct a two-parallel-path super-network by addition. Thirdly, at the training-level, the binary gate is introduced to control whether a path participates in the super-network training. It may suffer an unfair issue when using softmax to select the best input for intermediate nodes across two operator sub-sets. To tackle this problem, the sigmoid function is introduced, which measures the performance of operations without compression. Extensive experiments demonstrate the effectiveness of the proposed algorithm. Specifically, FP-DARTS achieves 2.50% test error with only 0.08 GPU-days on CIFAR10, and a state-of-the-art top-1 error rate of 23.7% on ImageNet using only 2.44 GPU-days for search.
2923.
Jin, Cong; Huang, Jinjie; Wei, Tianshu; Chen, Yuanjian
Neural architecture search based on dual attention mechanism for image classification Journal Article
In: Mathematical Biosciences and Engineering, vol. 20, no. 2, pp. 2691-2715, 2023, ISSN: 1551-0018.
@article{nokey,
title = {Neural architecture search based on dual attention mechanism for image classification},
author = {Cong Jin and Jinjie Huang and Tianshu Wei and Yuanjian Chen},
url = {https://www.aimspress.com/article/doi/10.3934/mbe.2023126},
doi = {10.3934/mbe.2023126},
issn = {1551-0018},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {Mathematical Biosciences and Engineering},
volume = {20},
number = {2},
pages = {2691-2715},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2922.
Jin, Yaochu; Zhu, Hangyu; Xu, Jinjin; Chen, Yang
Evolutionary Multi-objective Federated Learning Book Chapter
In: Federated Learning: Fundamentals and Advances, pp. 139–164, Springer Nature Singapore, Singapore, 2023, ISBN: 978-981-19-7083-2.
@inbook{Jin2023,
title = {Evolutionary Multi-objective Federated Learning},
author = {Yaochu Jin and Hangyu Zhu and Jinjin Xu and Yang Chen},
url = {https://link.springer.com/chapter/10.1007/978-981-19-7083-2_3},
doi = {10.1007/978-981-19-7083-2_3},
isbn = {978-981-19-7083-2},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
booktitle = {Federated Learning: Fundamentals and Advances},
pages = {139--164},
publisher = {Springer Nature Singapore},
address = {Singapore},
abstract = {Different from model quantization and partial model uploads presented in the previous chapter, evolutionary federated learning, more specifically, evolutionary federated neural architecture search, aims to optimize the architecture of neural network models, thereby reducing the communication costs caused by frequent model transmissions, generating lightweight neural models that are better suited for mobile and other edge devices, and also enhancing the final global model performance. To achieve this, scalable and efficient encoding methods for deep neural architectures must be designed and evolved using multi-objective evolutionary algorithms. This chapter presents two multi-objective evolutionary algorithms for federated neural architecture search. The first one employs a probabilistic representation of deep neural architectures that describes the connectivity between two neighboring layers and simultaneously maximizing the performance and minimizing the complexity of the neural architectures using a multi-objective evolutionary algorithm. However, this evolutionary framework is not practical for real-time optimization of the neural architectures in a federated environment. To tackle this challenge, a real-time federated evolutionary neural architecture search is then introduced. In addition to adopting a different neural search space, a double sampling strategy, including sampling subnetworks from a pretrained supernet and sampling clients for model update, is proposed so that the performance of the neural architectures becomes more stable, and each client needs to train one local model in one communication round, thereby preventing sudden performance drops during the optimization and avoiding training multiple submodels in one communication round. This way, evolutionary neural architecture search is made practical for real-time real-world applications.},
keywords = {},
pubstate = {published},
tppubtype = {inbook}
}
Different from model quantization and partial model uploads presented in the previous chapter, evolutionary federated learning, more specifically, evolutionary federated neural architecture search, aims to optimize the architecture of neural network models, thereby reducing the communication costs caused by frequent model transmissions, generating lightweight neural models that are better suited for mobile and other edge devices, and also enhancing the final global model performance. To achieve this, scalable and efficient encoding methods for deep neural architectures must be designed and evolved using multi-objective evolutionary algorithms. This chapter presents two multi-objective evolutionary algorithms for federated neural architecture search. The first one employs a probabilistic representation of deep neural architectures that describes the connectivity between two neighboring layers and simultaneously maximizing the performance and minimizing the complexity of the neural architectures using a multi-objective evolutionary algorithm. However, this evolutionary framework is not practical for real-time optimization of the neural architectures in a federated environment. To tackle this challenge, a real-time federated evolutionary neural architecture search is then introduced. In addition to adopting a different neural search space, a double sampling strategy, including sampling subnetworks from a pretrained supernet and sampling clients for model update, is proposed so that the performance of the neural architectures becomes more stable, and each client needs to train one local model in one communication round, thereby preventing sudden performance drops during the optimization and avoiding training multiple submodels in one communication round. This way, evolutionary neural architecture search is made practical for real-time real-world applications.
2921.
Lyu, Bo; Lu, Longfei; Hamdi, Maher; Wen, Shiping; Yang, Yin; Li, Ke
MTLP-JR: Multi-task learning-based prediction for joint ranking in neural architecture search Journal Article
In: Computers and Electrical Engineering, vol. 105, pp. 108474, 2023, ISSN: 0045-7906.
@article{LYU2023108474,
title = {MTLP-JR: Multi-task learning-based prediction for joint ranking in neural architecture search},
author = {Bo Lyu and Longfei Lu and Maher Hamdi and Shiping Wen and Yin Yang and Ke Li},
url = {https://www.sciencedirect.com/science/article/pii/S0045790622006899},
doi = {https://doi.org/10.1016/j.compeleceng.2022.108474},
issn = {0045-7906},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {Computers and Electrical Engineering},
volume = {105},
pages = {108474},
abstract = {At present, great attentions have been paid to multi-objective neural architecture search (NAS) and resource-aware NAS for their comprehensive consideration of the overall evaluation of architectures, including inference latency, precision, and model scale. However NAS also exacerbates the ever-increasing cost (engineering, time complexity, computation resource). Aiming to alleviate this, the reproducible NAS research releases the benchmark, which includes the metrics (e.g. Accuracy, Latency, and Parameters) of representative models from the typical search space on specific tasks. Motivated by the multi-objective NAS, resource-aware NAS, and reproducible NAS, this paper dedicates to binary-relation prediction (Latency, Accuracy), which is a more reasonable and effective way to satisfy the general NAS scenarios with less cost. We conduct a reproducible NAS study on the MobileNet-based search space and release the dataset. Further, we first propose the modeling of common features among prediction tasks (Latency, Accuracy, Parameters, and FLOPs), which will facilitate the prediction of individual tasks, and creatively formulate the architecture ranking prediction with a multi-task learning framework. Eventually, the proposed multi-task learning based binary-relation prediction model reaches the performance of 94.3% on Latency and 85.02% on Top1 Accuracy even with only 100 training points, which outperforms the single-task learning based model.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
At present, great attentions have been paid to multi-objective neural architecture search (NAS) and resource-aware NAS for their comprehensive consideration of the overall evaluation of architectures, including inference latency, precision, and model scale. However NAS also exacerbates the ever-increasing cost (engineering, time complexity, computation resource). Aiming to alleviate this, the reproducible NAS research releases the benchmark, which includes the metrics (e.g. Accuracy, Latency, and Parameters) of representative models from the typical search space on specific tasks. Motivated by the multi-objective NAS, resource-aware NAS, and reproducible NAS, this paper dedicates to binary-relation prediction (Latency, Accuracy), which is a more reasonable and effective way to satisfy the general NAS scenarios with less cost. We conduct a reproducible NAS study on the MobileNet-based search space and release the dataset. Further, we first propose the modeling of common features among prediction tasks (Latency, Accuracy, Parameters, and FLOPs), which will facilitate the prediction of individual tasks, and creatively formulate the architecture ranking prediction with a multi-task learning framework. Eventually, the proposed multi-task learning based binary-relation prediction model reaches the performance of 94.3% on Latency and 85.02% on Top1 Accuracy even with only 100 training points, which outperforms the single-task learning based model.
2920.
Gudzius, Povilas; Kurasova, Olga; Darulis, Vytenis; Filatovas, Ernestas
AutoML-Based Neural Architecture Search for Object Recognition in Satellite Imagery Journal Article
In: Remote Sensing, vol. 15, no. 1, 2023, ISSN: 2072-4292.
@article{rs15010091,
title = {AutoML-Based Neural Architecture Search for Object Recognition in Satellite Imagery},
author = {Povilas Gudzius and Olga Kurasova and Vytenis Darulis and Ernestas Filatovas},
url = {https://www.mdpi.com/2072-4292/15/1/91},
doi = {10.3390/rs15010091},
issn = {2072-4292},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {Remote Sensing},
volume = {15},
number = {1},
abstract = {Advancements in optical satellite hardware and lowered costs for satellite launches raised the high demand for geospatial intelligence. The object recognition problem in multi-spectral satellite imagery carries dataset properties unique to this problem. Perspective distortion, resolution variability, data spectrality, and other features make it difficult for a specific human-invented neural network to perform well on a dispersed type of scenery, ranging data quality, and different objects. UNET, MACU, and other manually designed network architectures deliver high-performance results for accuracy and prediction speed in large objects. However, once trained on different datasets, the performance drops and requires manual recalibration or further configuration testing to adjust the neural network architecture. To solve these issues, AutoML-based techniques can be employed. In this paper, we focus on Neural Architecture Search that is capable of obtaining a well-performing network configuration without human manual intervention. Firstly, we conducted detailed testing on the top four performing neural networks for object recognition in satellite imagery to compare their performance: FastFCN, DeepLabv3, UNET, and MACU. Then we applied and further developed a Neural Architecture Search technique for the best-performing manually designed MACU by optimizing a search space at the artificial neuron cellular level of the network. Several NAS-MACU versions were explored and evaluated. Our developed AutoML process generated a NAS-MACU neural network that produced better performance compared with MACU, especially in a low-information intensity environment. The experimental investigation was performed on our annotated and updated publicly available satellite imagery dataset. We can state that the application of the Neural Architecture Search procedure has the capability to be applied across various datasets and object recognition problems within the remote sensing research field.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Advancements in optical satellite hardware and lowered costs for satellite launches raised the high demand for geospatial intelligence. The object recognition problem in multi-spectral satellite imagery carries dataset properties unique to this problem. Perspective distortion, resolution variability, data spectrality, and other features make it difficult for a specific human-invented neural network to perform well on a dispersed type of scenery, ranging data quality, and different objects. UNET, MACU, and other manually designed network architectures deliver high-performance results for accuracy and prediction speed in large objects. However, once trained on different datasets, the performance drops and requires manual recalibration or further configuration testing to adjust the neural network architecture. To solve these issues, AutoML-based techniques can be employed. In this paper, we focus on Neural Architecture Search that is capable of obtaining a well-performing network configuration without human manual intervention. Firstly, we conducted detailed testing on the top four performing neural networks for object recognition in satellite imagery to compare their performance: FastFCN, DeepLabv3, UNET, and MACU. Then we applied and further developed a Neural Architecture Search technique for the best-performing manually designed MACU by optimizing a search space at the artificial neuron cellular level of the network. Several NAS-MACU versions were explored and evaluated. Our developed AutoML process generated a NAS-MACU neural network that produced better performance compared with MACU, especially in a low-information intensity environment. The experimental investigation was performed on our annotated and updated publicly available satellite imagery dataset. We can state that the application of the Neural Architecture Search procedure has the capability to be applied across various datasets and object recognition problems within the remote sensing research field.
2919.
Wang, Xiaojuan; He, Mingshu; Yang, Liu; Wang, Hui; Zhong, Yun
In: Electronics, vol. 12, no. 1, 2023, ISSN: 2079-9292.
@article{electronics12010050,
title = {Human Activity Recognition Based on an Efficient Neural Architecture Search Framework Using Evolutionary Multi-Objective Surrogate-Assisted Algorithms},
author = {Xiaojuan Wang and Mingshu He and Liu Yang and Hui Wang and Yun Zhong},
url = {https://www.mdpi.com/2079-9292/12/1/50},
doi = {10.3390/electronics12010050},
issn = {2079-9292},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {Electronics},
volume = {12},
number = {1},
abstract = {Human activity recognition (HAR) is a popular and challenging research topic driven by various applications. Deep learning methods have been used to improve HAR models’ accuracy and efficiency. However, this kind of method has a lot of manually adjusted parameters, which cost researchers a lot of time to train and test. So, it is challenging to design a suitable model. In this paper, we propose HARNAS, an efficient approach for automatic architecture search for HAR. Inspired by the popular multi-objective evolutionary algorithm, which has a strong capability in solving problems with multiple conflicting objectives, we set weighted f1-score, flops, and the number of parameters as objects. Furthermore, we use a surrogate model to select models with a high score from the large candidate set. Moreover, the chosen models are added to the training set of the surrogate model, which makes the surrogate model update along the search process. Our method avoids manually designing the network structure, and the experiment results demonstrate that it can reduce 40% training costs on both time and computing resources on the OPPORTUNITY dataset and 75% on the UniMiB-SHAR dataset. Additionally, we also prove the portability of the trained surrogate model and HAR model by transferring them from the training dataset to a new dataset.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Human activity recognition (HAR) is a popular and challenging research topic driven by various applications. Deep learning methods have been used to improve HAR models’ accuracy and efficiency. However, this kind of method has a lot of manually adjusted parameters, which cost researchers a lot of time to train and test. So, it is challenging to design a suitable model. In this paper, we propose HARNAS, an efficient approach for automatic architecture search for HAR. Inspired by the popular multi-objective evolutionary algorithm, which has a strong capability in solving problems with multiple conflicting objectives, we set weighted f1-score, flops, and the number of parameters as objects. Furthermore, we use a surrogate model to select models with a high score from the large candidate set. Moreover, the chosen models are added to the training set of the surrogate model, which makes the surrogate model update along the search process. Our method avoids manually designing the network structure, and the experiment results demonstrate that it can reduce 40% training costs on both time and computing resources on the OPPORTUNITY dataset and 75% on the UniMiB-SHAR dataset. Additionally, we also prove the portability of the trained surrogate model and HAR model by transferring them from the training dataset to a new dataset.
2918.
Verma, Monu; Lubal, Priyanka; Vipparthi, Santosh Kumar; Abdel-Mottaleb, Mohamed
RNAS-MER: A Refined Neural Architecture Search With Hybrid Spatiotemporal Operations for Micro-Expression Recognition Proceedings Article
In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 4770-4779, 2023.
@inproceedings{Verma_2023_WACV,
title = {RNAS-MER: A Refined Neural Architecture Search With Hybrid Spatiotemporal Operations for Micro-Expression Recognition},
author = {Monu Verma and Priyanka Lubal and Santosh Kumar Vipparthi and Mohamed Abdel-Mottaleb},
url = {https://openaccess.thecvf.com/content/WACV2023/html/Verma_RNAS-MER_A_Refined_Neural_Architecture_Search_With_Hybrid_Spatiotemporal_Operations_WACV_2023_paper.html},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
pages = {4770-4779},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
2917.
Nagpure, Vikrant; Okuma, Kenji
Searching Efficient Neural Architecture With Multi-Resolution Fusion Transformer for Appearance-Based Gaze Estimation Proceedings Article
In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 890-899, 2023.
@inproceedings{Nagpure_2023_WACV,
title = {Searching Efficient Neural Architecture With Multi-Resolution Fusion Transformer for Appearance-Based Gaze Estimation},
author = {Vikrant Nagpure and Kenji Okuma},
url = {https://openaccess.thecvf.com/content/WACV2023/html/Nagpure_Searching_Efficient_Neural_Architecture_With_Multi-Resolution_Fusion_Transformer_for_Appearance-Based_WACV_2023_paper.html},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
pages = {890-899},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
2916.
Ahn, Daehyun; Kim, Hyungjun; Kim, Taesu; Park, Eunhyeok; Kim, Jae-Joon
Searching for Robust Binary Neural Networks via Bimodal Parameter Perturbation Proceedings Article
In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 2410-2419, 2023.
@inproceedings{Ahn_2023_WACV,
title = {Searching for Robust Binary Neural Networks via Bimodal Parameter Perturbation},
author = {Daehyun Ahn and Hyungjun Kim and Taesu Kim and Eunhyeok Park and Jae-Joon Kim},
url = {https://openaccess.thecvf.com/content/WACV2023/html/Ahn_Searching_for_Robust_Binary_Neural_Networks_via_Bimodal_Parameter_Perturbation_WACV_2023_paper.html},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
pages = {2410-2419},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
2915.
Li, Yinqiao; Cao, Runzhe; He, Qiaozhi; Xiao, Tong; Zhu, Jingbo
Learning Reliable Neural Networks with Distributed Architecture Representations Journal Article
In: ACM Trans. Asian Low-Resour. Lang. Inf. Process., 2023, ISSN: 2375-4699, (Just Accepted).
@article{10.1145/3578709,
title = {Learning Reliable Neural Networks with Distributed Architecture Representations},
author = {Yinqiao Li and Runzhe Cao and Qiaozhi He and Tong Xiao and Jingbo Zhu},
url = {https://doi.org/10.1145/3578709},
doi = {10.1145/3578709},
issn = {2375-4699},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {ACM Trans. Asian Low-Resour. Lang. Inf. Process.},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
abstract = {Neural architecture search (NAS) has shown the strong performance of learning neural models automatically in recent years. But most NAS systems are unreliable due to the architecture gap brought by discrete representations of atomic architectures. In this paper, we improve the performance and robustness of NAS var narrowing the gap between architecture representations. More specifically, we apply a general contraction mapping to model neural networks with distributed representations (call it ArchDAR). Moreover, for a better search result, we present a joint learning approach to integrating distributed representations with advanced architecture search methods. We implement our ArchDAR in a differentiable architecture search model and test learned architectures on the language modeling task. On the PTB data, it outperforms a strong baseline significantly by 1.8 perplexity scores. Also, the search process with distributed representations is more stable which yields a faster structural convergence when it works with the DARTS model.},
note = {Just Accepted},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Neural architecture search (NAS) has shown the strong performance of learning neural models automatically in recent years. But most NAS systems are unreliable due to the architecture gap brought by discrete representations of atomic architectures. In this paper, we improve the performance and robustness of NAS var narrowing the gap between architecture representations. More specifically, we apply a general contraction mapping to model neural networks with distributed representations (call it ArchDAR). Moreover, for a better search result, we present a joint learning approach to integrating distributed representations with advanced architecture search methods. We implement our ArchDAR in a differentiable architecture search model and test learned architectures on the language modeling task. On the PTB data, it outperforms a strong baseline significantly by 1.8 perplexity scores. Also, the search process with distributed representations is more stable which yields a faster structural convergence when it works with the DARTS model.
2914.
Ahmadian, Arash; Fei, Yue; Liu, Louis S. P.; Plataniotis, Konstantinos N.; Hosseini, Mahdi S.
Pseudo-Inverted Bottleneck Convolution for DARTS Search Space Technical Report
2023.
@techreport{https://doi.org/10.48550/arxiv.2301.01286,
title = {Pseudo-Inverted Bottleneck Convolution for DARTS Search Space},
author = {Arash Ahmadian and Yue Fei and Louis S. P. Liu and Konstantinos N. Plataniotis and Mahdi S. Hosseini},
url = {https://arxiv.org/abs/2301.01286},
doi = {10.48550/ARXIV.2301.01286},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
publisher = {arXiv},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2913.
Ding, Li; Spector, Lee
Multi-Objective Evolutionary Architecture Search for Parameterized Quantum Circuits Journal Article
In: Entropy, vol. 25, no. 1, 2023, ISSN: 1099-4300.
@article{e25010093,
title = {Multi-Objective Evolutionary Architecture Search for Parameterized Quantum Circuits},
author = {Li Ding and Lee Spector},
url = {https://www.mdpi.com/1099-4300/25/1/93},
doi = {10.3390/e25010093},
issn = {1099-4300},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {Entropy},
volume = {25},
number = {1},
abstract = {Recent work on hybrid quantum-classical machine learning systems has demonstrated success in utilizing parameterized quantum circuits (PQCs) to solve the challenging reinforcement learning (RL) tasks, with provable learning advantages over classical systems, e.g., deep neural networks. While existing work demonstrates and exploits the strength of PQC-based models, the design choices of PQC architectures and the interactions between different quantum circuits on learning tasks are generally underexplored. In this work, we introduce a Multi-objective Evolutionary Architecture Search framework for parameterized quantum circuits (MEAS-PQC), which uses a multi-objective genetic algorithm with quantum-specific configurations to perform efficient searching of optimal PQC architectures. Experimental results show that our method can find architectures that have superior learning performance on three benchmark RL tasks, and are also optimized for additional objectives including reductions in quantum noise and model size. Further analysis of patterns and probability distributions of quantum operations helps identify performance-critical design choices of hybrid quantum-classical learning systems.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Recent work on hybrid quantum-classical machine learning systems has demonstrated success in utilizing parameterized quantum circuits (PQCs) to solve the challenging reinforcement learning (RL) tasks, with provable learning advantages over classical systems, e.g., deep neural networks. While existing work demonstrates and exploits the strength of PQC-based models, the design choices of PQC architectures and the interactions between different quantum circuits on learning tasks are generally underexplored. In this work, we introduce a Multi-objective Evolutionary Architecture Search framework for parameterized quantum circuits (MEAS-PQC), which uses a multi-objective genetic algorithm with quantum-specific configurations to perform efficient searching of optimal PQC architectures. Experimental results show that our method can find architectures that have superior learning performance on three benchmark RL tasks, and are also optimized for additional objectives including reductions in quantum noise and model size. Further analysis of patterns and probability distributions of quantum operations helps identify performance-critical design choices of hybrid quantum-classical learning systems.
2912.
Saeed, Fahman; Hussain, Muhammad; Aboalsamh, Hatim A.; Adel, Fadwa Al; Owaifeer, Adi Mohammed Al
Designing the Architecture of a Convolutional Neural Network Automatically for Diabetic Retinopathy Diagnosis Journal Article
In: Mathematics, vol. 11, no. 2, 2023, ISSN: 2227-7390.
@article{math11020307,
title = {Designing the Architecture of a Convolutional Neural Network Automatically for Diabetic Retinopathy Diagnosis},
author = {Fahman Saeed and Muhammad Hussain and Hatim A. Aboalsamh and Fadwa Al Adel and Adi Mohammed Al Owaifeer},
url = {https://www.mdpi.com/2227-7390/11/2/307},
doi = {10.3390/math11020307},
issn = {2227-7390},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {Mathematics},
volume = {11},
number = {2},
abstract = {Diabetic retinopathy (DR) is a leading cause of blindness in middle-aged diabetic patients. Regular screening for DR using fundus imaging aids in detecting complications and delays the progression of the disease. Because manual screening takes time and is subjective, deep learning has been used to help graders. Pre-trained or brute force CNN models are used in existing DR grading CNN-based approaches that are not suited to fundus image complexity. To solve this problem, we present a method for automatically customizing CNN models based on fundus image lesions. It uses k-medoid clustering, principal component analysis (PCA), and inter-class and intra-class variations to determine the CNN model’s depth and width. The designed models are lightweight, adapted to the internal structures of fundus images, and encode the discriminative patterns of DR lesions. The technique is validated on a local dataset from King Saud University Medical City, Saudi Arabia, and two challenging Kaggle datasets: EyePACS and APTOS2019. The auto-designed models outperform well-known pre-trained CNN models such as ResNet152, DenseNet121, and ResNeSt50, as well as Google’s AutoML and Auto-Keras models based on neural architecture search (NAS). The proposed method outperforms current CNN-based DR screening methods. The proposed method can be used in various clinical settings to screen for DR and refer patients to ophthalmologists for further evaluation and treatment.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Diabetic retinopathy (DR) is a leading cause of blindness in middle-aged diabetic patients. Regular screening for DR using fundus imaging aids in detecting complications and delays the progression of the disease. Because manual screening takes time and is subjective, deep learning has been used to help graders. Pre-trained or brute force CNN models are used in existing DR grading CNN-based approaches that are not suited to fundus image complexity. To solve this problem, we present a method for automatically customizing CNN models based on fundus image lesions. It uses k-medoid clustering, principal component analysis (PCA), and inter-class and intra-class variations to determine the CNN model’s depth and width. The designed models are lightweight, adapted to the internal structures of fundus images, and encode the discriminative patterns of DR lesions. The technique is validated on a local dataset from King Saud University Medical City, Saudi Arabia, and two challenging Kaggle datasets: EyePACS and APTOS2019. The auto-designed models outperform well-known pre-trained CNN models such as ResNet152, DenseNet121, and ResNeSt50, as well as Google’s AutoML and Auto-Keras models based on neural architecture search (NAS). The proposed method outperforms current CNN-based DR screening methods. The proposed method can be used in various clinical settings to screen for DR and refer patients to ophthalmologists for further evaluation and treatment.
2911.
Magalhães, Dimmy; Lima, Ricardo H. R.; Pozo, Aurora
Creating deep neural networks for text classification tasks using grammar genetic programming Journal Article
In: Applied Soft Computing, pp. 110009, 2023, ISSN: 1568-4946.
@article{MAGALHAES2023110009,
title = {Creating deep neural networks for text classification tasks using grammar genetic programming},
author = {Dimmy Magalhães and Ricardo H. R. Lima and Aurora Pozo},
url = {https://www.sciencedirect.com/science/article/pii/S1568494623000273},
doi = {https://doi.org/10.1016/j.asoc.2023.110009},
issn = {1568-4946},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {Applied Soft Computing},
pages = {110009},
abstract = {Text classification is one of the Natural Language Processing (NLP) tasks. Its objective is to label textual elements, such as phrases, queries, paragraphs, and documents. In NLP, several approaches have achieved promising results regarding this task. Deep Learning-based approaches have been widely used in this context, with deep neural networks (DNNs) adding the ability to generate a representation for the data and a learning model. The increasing scale and complexity of DNN architectures was expected, creating new challenges to design and configure the models. In this paper, we present a study on the application of a grammar-based evolutionary approach to the design of DNNs, using models based on Convolutional Neural Networks (CNNs), Long Short-Term Memory (LSTM), and Graph Neural Networks (GNNs). We propose different grammars, which were defined to capture the features of each type of network, also proposing some combinations, verifying their impact on the produced designs and performance of the generated models. We create a grammar that is able to generate different networks specialized on text classification, by modification of Grammatical Evolution (GE), and it is composed of three main components: the grammar, mapping, and search engine. Our results offer promising future research directions as they show that the projected architectures have a performance comparable to that of their counterparts but can still be further improved. We were able to improve the results of a manually structured neural network in 8,18% in the best case.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Text classification is one of the Natural Language Processing (NLP) tasks. Its objective is to label textual elements, such as phrases, queries, paragraphs, and documents. In NLP, several approaches have achieved promising results regarding this task. Deep Learning-based approaches have been widely used in this context, with deep neural networks (DNNs) adding the ability to generate a representation for the data and a learning model. The increasing scale and complexity of DNN architectures was expected, creating new challenges to design and configure the models. In this paper, we present a study on the application of a grammar-based evolutionary approach to the design of DNNs, using models based on Convolutional Neural Networks (CNNs), Long Short-Term Memory (LSTM), and Graph Neural Networks (GNNs). We propose different grammars, which were defined to capture the features of each type of network, also proposing some combinations, verifying their impact on the produced designs and performance of the generated models. We create a grammar that is able to generate different networks specialized on text classification, by modification of Grammatical Evolution (GE), and it is composed of three main components: the grammar, mapping, and search engine. Our results offer promising future research directions as they show that the projected architectures have a performance comparable to that of their counterparts but can still be further improved. We were able to improve the results of a manually structured neural network in 8,18% in the best case.
2910.
Li, Xingzhuo; Hou, Sujuan; Zhang, Baisong; Wang, Jing; Jia, Weikuan; Zheng, Yuanjie
Long-Range Dependence Involutional Network for Logo Detection Journal Article
In: Entropy, vol. 25, no. 1, 2023, ISSN: 1099-4300.
@article{e25010174,
title = {Long-Range Dependence Involutional Network for Logo Detection},
author = {Xingzhuo Li and Sujuan Hou and Baisong Zhang and Jing Wang and Weikuan Jia and Yuanjie Zheng},
url = {https://www.mdpi.com/1099-4300/25/1/174},
doi = {10.3390/e25010174},
issn = {1099-4300},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {Entropy},
volume = {25},
number = {1},
abstract = {Logo detection is one of the crucial branches in computer vision due to various real-world applications, such as automatic logo detection and recognition, intelligent transportation, and trademark infringement detection. Compared with traditional handcrafted-feature-based methods, deep learning-based convolutional neural networks (CNNs) can learn both low-level and high-level image features. Recent decades have witnessed the great feature representation capabilities of deep CNNs and their variants, which have been very good at discovering intricate structures in high-dimensional data and are thereby applicable to many domains including logo detection. However, logo detection remains challenging, as existing detection methods cannot solve well the problems of a multiscale and large aspect ratios. In this paper, we tackle these challenges by developing a novel long-range dependence involutional network (LDI-Net). Specifically, we designed a strategy that combines a new operator and a self-attention mechanism via rethinking the intrinsic principle of convolution called long-range dependence involution (LD involution) to alleviate the detection difficulties caused by large aspect ratios. We also introduce a multilevel representation neural architecture search (MRNAS) to detect multiscale logo objects by constructing a novel multipath topology. In addition, we implemented an adaptive RoI pooling module (ARM) to improve detection efficiency by addressing the problem of logo deformation. Comprehensive experiments on four benchmark logo datasets demonstrate the effectiveness and efficiency of the proposed approach.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Logo detection is one of the crucial branches in computer vision due to various real-world applications, such as automatic logo detection and recognition, intelligent transportation, and trademark infringement detection. Compared with traditional handcrafted-feature-based methods, deep learning-based convolutional neural networks (CNNs) can learn both low-level and high-level image features. Recent decades have witnessed the great feature representation capabilities of deep CNNs and their variants, which have been very good at discovering intricate structures in high-dimensional data and are thereby applicable to many domains including logo detection. However, logo detection remains challenging, as existing detection methods cannot solve well the problems of a multiscale and large aspect ratios. In this paper, we tackle these challenges by developing a novel long-range dependence involutional network (LDI-Net). Specifically, we designed a strategy that combines a new operator and a self-attention mechanism via rethinking the intrinsic principle of convolution called long-range dependence involution (LD involution) to alleviate the detection difficulties caused by large aspect ratios. We also introduce a multilevel representation neural architecture search (MRNAS) to detect multiscale logo objects by constructing a novel multipath topology. In addition, we implemented an adaptive RoI pooling module (ARM) to improve detection efficiency by addressing the problem of logo deformation. Comprehensive experiments on four benchmark logo datasets demonstrate the effectiveness and efficiency of the proposed approach.
2909.
Xie, Xiangning; Song, Xiaotian; Lv, Zeqiong; Yen, Gary G.; Ding, Weiping; Sun, Yanan
Efficient Evaluation Methods for Neural Architecture Search: A Survey Technical Report
2023.
@techreport{https://doi.org/10.48550/arxiv.2301.05919,
title = {Efficient Evaluation Methods for Neural Architecture Search: A Survey},
author = {Xiangning Xie and Xiaotian Song and Zeqiong Lv and Gary G. Yen and Weiping Ding and Yanan Sun},
url = {https://arxiv.org/abs/2301.05919},
doi = {10.48550/ARXIV.2301.05919},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
publisher = {arXiv},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2908.
Lee, Seunghyun; Song, Byung Cheol
Fast Filter Pruning via Coarse-to-Fine Neural Architecture Search and Contrastive Knowledge Transfer Journal Article
In: IEEE Transactions on Neural Networks and Learning Systems, pp. 1-12, 2023.
@article{10018843,
title = {Fast Filter Pruning via Coarse-to-Fine Neural Architecture Search and Contrastive Knowledge Transfer},
author = {Seunghyun Lee and Byung Cheol Song},
url = {https://ieeexplore.ieee.org/abstract/document/10018843},
doi = {10.1109/TNNLS.2023.3236336},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {IEEE Transactions on Neural Networks and Learning Systems},
pages = {1-12},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2907.
Chauhan, Anshumaan; Bhattacharyya, Siddhartha; Vadivel, S.
DQNAS: Neural Architecture Search using Reinforcement Learning Technical Report
2023.
@techreport{DBLP:journals/corr/abs-2301-06687,
title = {DQNAS: Neural Architecture Search using Reinforcement Learning},
author = {Anshumaan Chauhan and Siddhartha Bhattacharyya and S. Vadivel},
url = {https://doi.org/10.48550/arXiv.2301.06687},
doi = {10.48550/arXiv.2301.06687},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {CoRR},
volume = {abs/2301.06687},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2906.
Ye, Peng; He, Tong; Li, Baopu; Chen, Tao; Bai, Lei; Ouyang, Wanli
(beta)-DARTS++: Bi-level Regularization for Proxy-robust Differentiable Architecture Search Technical Report
2023.
@techreport{DBLP:journals/corr/abs-2301-06393,
title = {(beta)-DARTS++: Bi-level Regularization for Proxy-robust Differentiable Architecture Search},
author = {Peng Ye and Tong He and Baopu Li and Tao Chen and Lei Bai and Wanli Ouyang},
url = {https://doi.org/10.48550/arXiv.2301.06393},
doi = {10.48550/arXiv.2301.06393},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {CoRR},
volume = {abs/2301.06393},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2905.
Ford, Noah; Winder, John; McClellan, Josh
Adaptive Neural Networks Using Residual Fitting Technical Report
2023.
@techreport{DBLP:journals/corr/abs-2301-05744,
title = {Adaptive Neural Networks Using Residual Fitting},
author = {Noah Ford and John Winder and Josh McClellan},
url = {https://doi.org/10.48550/arXiv.2301.05744},
doi = {10.48550/arXiv.2301.05744},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {CoRR},
volume = {abs/2301.05744},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2904.
Lakhmiri, Dounia; Zolnouri, Mahdi; Nia, Vahid Partovi; Tribes, Christophe; Digabel, Sébastien Le
Scaling Deep Networks with the Mesh Adaptive Direct Search algorithm Technical Report
2023.
@techreport{DBLP:journals/corr/abs-2301-06641,
title = {Scaling Deep Networks with the Mesh Adaptive Direct Search algorithm},
author = {Dounia Lakhmiri and Mahdi Zolnouri and Vahid Partovi Nia and Christophe Tribes and Sébastien Le Digabel},
url = {https://doi.org/10.48550/arXiv.2301.06641},
doi = {10.48550/arXiv.2301.06641},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {CoRR},
volume = {abs/2301.06641},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2903.
White, Colin; Safari, Mahmoud; Sukthanker, Rhea; Ru, Binxin; Elsken, Thomas; Zela, Arber; Dey, Debadeepta; Hutter, Frank
Neural Architecture Search: Insights from 1000 Papers Technical Report
2023.
@techreport{https://doi.org/10.48550/arxiv.2301.08727,
title = {Neural Architecture Search: Insights from 1000 Papers},
author = {Colin White and Mahmoud Safari and Rhea Sukthanker and Binxin Ru and Thomas Elsken and Arber Zela and Debadeepta Dey and Frank Hutter},
url = {https://arxiv.org/abs/2301.08727},
doi = {10.48550/ARXIV.2301.08727},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
publisher = {arXiv},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2902.
Kong, Gangwei; Li, Chang; Peng, Hu; Han, Zhihui; Qiao, Heyuan
EEG-Based Sleep Stage Classification via Neural Architecture Search Journal Article
In: IEEE transactions on neural systems and rehabilitation engineering: a publication of the IEEE Engineering in Medicine and Biology Society, 2023.
@article{articleb,
title = {EEG-Based Sleep Stage Classification via Neural Architecture Search},
author = {Gangwei Kong and Chang Li and Hu Peng and Zhihui Han and Heyuan Qiao},
url = {https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=10024773&tag=1},
doi = {10.1109/TNSRE.2023.3238764},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {IEEE transactions on neural systems and rehabilitation engineering: a publication of the IEEE Engineering in Medicine and Biology Society},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2901.
Shi, Chaokun; Hao, Yuexing; Li, Gongyan; Xu, Shaoyun
EBNAS: Efficient binary network design for image classification via neural architecture search Journal Article
In: Engineering Applications of Artificial Intelligence, vol. 120, pp. 105845, 2023, ISSN: 0952-1976.
@article{SHI2023105845,
title = {EBNAS: Efficient binary network design for image classification via neural architecture search},
author = {Chaokun Shi and Yuexing Hao and Gongyan Li and Shaoyun Xu},
url = {https://www.sciencedirect.com/science/article/pii/S0952197623000295},
doi = {https://doi.org/10.1016/j.engappai.2023.105845},
issn = {0952-1976},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {Engineering Applications of Artificial Intelligence},
volume = {120},
pages = {105845},
abstract = {To deploy Convolutional Neural Networks (CNNs) on resource-limited devices, binary CNNs with 1-bit activations and weights prove to be a promising approach. Meanwhile, Neural Architecture Search (NAS), which can design lightweight networks beyond artificial ones, has achieved optimal performance in various tasks. To design high-performance binary networks, we propose an efficient binary neural architecture search algorithm, namely EBNAS. In this paper, we propose corresponding improvement strategies to deal with the information loss due to binarization, the discrete error between search and evaluation, and the imbalanced operation advantage in the search space. Specifically, we adopt a new search space consisting of operations suitable for the binary domain. An L2 path regularization and a variance-based edge regularization are introduced to guide the search process and drive architecture parameters toward discretization. In addition, we present a search space simplification strategy and adjust the channel sampling proportions to balance the advantages of different operations. We perform extensive experiments on CIFAR10, CIFAR100, and ImageNet datasets. The results demonstrate the effectiveness of our proposed methods. For example, with binary weights and activations, EBNAS achieves a Top-1 accuracy of 95.61% on CIFAR10, 78.10% on CIFAR100, and 67.8% on ImageNet. With a similar number of model parameters, our algorithm outperforms other binary NAS methods in terms of accuracy and efficiency. Compared with manually designed binary networks, our algorithm remains competitive. The code is available at https://github.com/sscckk/EBNAS.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
To deploy Convolutional Neural Networks (CNNs) on resource-limited devices, binary CNNs with 1-bit activations and weights prove to be a promising approach. Meanwhile, Neural Architecture Search (NAS), which can design lightweight networks beyond artificial ones, has achieved optimal performance in various tasks. To design high-performance binary networks, we propose an efficient binary neural architecture search algorithm, namely EBNAS. In this paper, we propose corresponding improvement strategies to deal with the information loss due to binarization, the discrete error between search and evaluation, and the imbalanced operation advantage in the search space. Specifically, we adopt a new search space consisting of operations suitable for the binary domain. An L2 path regularization and a variance-based edge regularization are introduced to guide the search process and drive architecture parameters toward discretization. In addition, we present a search space simplification strategy and adjust the channel sampling proportions to balance the advantages of different operations. We perform extensive experiments on CIFAR10, CIFAR100, and ImageNet datasets. The results demonstrate the effectiveness of our proposed methods. For example, with binary weights and activations, EBNAS achieves a Top-1 accuracy of 95.61% on CIFAR10, 78.10% on CIFAR100, and 67.8% on ImageNet. With a similar number of model parameters, our algorithm outperforms other binary NAS methods in terms of accuracy and efficiency. Compared with manually designed binary networks, our algorithm remains competitive. The code is available at https://github.com/sscckk/EBNAS.
2900.
Nath, Utkarsh; Wang, Yancheng; Yang, Yingzhen
RNAS-CL: Robust Neural Architecture Search by Cross-Layer Knowledge Distillation Technical Report
2023.
@techreport{https://doi.org/10.48550/arxiv.2301.08092,
title = {RNAS-CL: Robust Neural Architecture Search by Cross-Layer Knowledge Distillation},
author = {Utkarsh Nath and Yancheng Wang and Yingzhen Yang},
url = {https://arxiv.org/abs/2301.08092},
doi = {10.48550/ARXIV.2301.08092},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
publisher = {arXiv},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2899.
Jin, Haifeng; Chollet, François; Song, Qingquan; Hu, Xia
AutoKeras: An AutoML Library for Deep Learning Journal Article
In: Journal of Machine Learning Research, vol. 24, no. 6, pp. 1–6, 2023.
@article{JMLR:v24:20-1355,
title = {AutoKeras: An AutoML Library for Deep Learning},
author = {Haifeng Jin and François Chollet and Qingquan Song and Xia Hu},
url = {http://jmlr.org/papers/v24/20-1355.html},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {Journal of Machine Learning Research},
volume = {24},
number = {6},
pages = {1--6},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2898.
Carrasquilla, Juan; Hibat-Allah, Mohamed; Inack, Estelle; Makhzani, Alireza; Neklyudov, Kirill; Taylor, Graham W.; Torlai, Giacomo
Quantum HyperNetworks: Training Binary Neural Networks in Quantum Superposition Technical Report
2023.
@techreport{https://doi.org/10.48550/arxiv.2301.08292,
title = {Quantum HyperNetworks: Training Binary Neural Networks in Quantum Superposition},
author = {Juan Carrasquilla and Mohamed Hibat-Allah and Estelle Inack and Alireza Makhzani and Kirill Neklyudov and Graham W. Taylor and Giacomo Torlai},
url = {https://arxiv.org/abs/2301.08292},
doi = {10.48550/ARXIV.2301.08292},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
publisher = {arXiv},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2897.
Shi, Chaokun; Hao, Yuexing; Li, Gongyan; Xu, Shaoyun
EBNAS: Efficient binary network design for image classification via neural architecture search Journal Article
In: Engineering Applications of Artificial Intelligence, vol. 120, pp. 105845, 2023, ISSN: 0952-1976.
@article{SHI2023105845b,
title = {EBNAS: Efficient binary network design for image classification via neural architecture search},
author = {Chaokun Shi and Yuexing Hao and Gongyan Li and Shaoyun Xu},
url = {https://www.sciencedirect.com/science/article/pii/S0952197623000295},
doi = {https://doi.org/10.1016/j.engappai.2023.105845},
issn = {0952-1976},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {Engineering Applications of Artificial Intelligence},
volume = {120},
pages = {105845},
abstract = {To deploy Convolutional Neural Networks (CNNs) on resource-limited devices, binary CNNs with 1-bit activations and weights prove to be a promising approach. Meanwhile, Neural Architecture Search (NAS), which can design lightweight networks beyond artificial ones, has achieved optimal performance in various tasks. To design high-performance binary networks, we propose an efficient binary neural architecture search algorithm, namely EBNAS. In this paper, we propose corresponding improvement strategies to deal with the information loss due to binarization, the discrete error between search and evaluation, and the imbalanced operation advantage in the search space. Specifically, we adopt a new search space consisting of operations suitable for the binary domain. An L2 path regularization and a variance-based edge regularization are introduced to guide the search process and drive architecture parameters toward discretization. In addition, we present a search space simplification strategy and adjust the channel sampling proportions to balance the advantages of different operations. We perform extensive experiments on CIFAR10, CIFAR100, and ImageNet datasets. The results demonstrate the effectiveness of our proposed methods. For example, with binary weights and activations, EBNAS achieves a Top-1 accuracy of 95.61% on CIFAR10, 78.10% on CIFAR100, and 67.8% on ImageNet. With a similar number of model parameters, our algorithm outperforms other binary NAS methods in terms of accuracy and efficiency. Compared with manually designed binary networks, our algorithm remains competitive. The code is available at https://github.com/sscckk/EBNAS.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
To deploy Convolutional Neural Networks (CNNs) on resource-limited devices, binary CNNs with 1-bit activations and weights prove to be a promising approach. Meanwhile, Neural Architecture Search (NAS), which can design lightweight networks beyond artificial ones, has achieved optimal performance in various tasks. To design high-performance binary networks, we propose an efficient binary neural architecture search algorithm, namely EBNAS. In this paper, we propose corresponding improvement strategies to deal with the information loss due to binarization, the discrete error between search and evaluation, and the imbalanced operation advantage in the search space. Specifically, we adopt a new search space consisting of operations suitable for the binary domain. An L2 path regularization and a variance-based edge regularization are introduced to guide the search process and drive architecture parameters toward discretization. In addition, we present a search space simplification strategy and adjust the channel sampling proportions to balance the advantages of different operations. We perform extensive experiments on CIFAR10, CIFAR100, and ImageNet datasets. The results demonstrate the effectiveness of our proposed methods. For example, with binary weights and activations, EBNAS achieves a Top-1 accuracy of 95.61% on CIFAR10, 78.10% on CIFAR100, and 67.8% on ImageNet. With a similar number of model parameters, our algorithm outperforms other binary NAS methods in terms of accuracy and efficiency. Compared with manually designed binary networks, our algorithm remains competitive. The code is available at https://github.com/sscckk/EBNAS.
2896.
Eminaga, Okyaz; Abbas, Mahmoud; Shen, Jeanne; Laurie, Mark; Brooks, James D.; Liao, Joseph C.; Rubin, Daniel L.
PlexusNet: A neural network architectural concept for medical image classification Journal Article
In: Computers in Biology and Medicine, pp. 106594, 2023, ISSN: 0010-4825.
@article{EMINAGA2023106594,
title = {PlexusNet: A neural network architectural concept for medical image classification},
author = {Okyaz Eminaga and Mahmoud Abbas and Jeanne Shen and Mark Laurie and James D. Brooks and Joseph C. Liao and Daniel L. Rubin},
url = {https://www.sciencedirect.com/science/article/pii/S0010482523000598},
doi = {https://doi.org/10.1016/j.compbiomed.2023.106594},
issn = {0010-4825},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {Computers in Biology and Medicine},
pages = {106594},
abstract = {State-of-the-art (SOTA) convolutional neural network models have been widely adapted in medical imaging and applied to address different clinical problems. However, the complexity and scale of such models may not be justified in medical imaging and subject to the available resource budget. Further increasing the number of representative feature maps for the classification task decreases the model explainability. The current data normalization practice is fixed prior to model development and discounting the specification of the data domain. Acknowledging these issues, the current work proposed a new scalable model family called PlexusNet; the block architecture and model scaling by the network's depth, width, and branch regulate PlexusNet's architecture. The efficient computation costs outlined the dimensions of PlexusNet scaling and design. PlexusNet includes a new learnable data normalization algorithm for better data generalization. We applied a simple yet effective neural architecture search to design PlexusNet tailored to five clinical classification problems that achieve a performance noninferior to the SOTA models ResNet-18 and EfficientNet B0/1. It also does so with lower parameter capacity and representative feature maps in ten-fold ranges than the smallest SOTA models with comparable performance. The visualization of representative features revealed distinguishable clusters associated with categories based on latent features generated by PlexusNet. The package and source code are at https://github.com/oeminaga/PlexusNet.git.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
State-of-the-art (SOTA) convolutional neural network models have been widely adapted in medical imaging and applied to address different clinical problems. However, the complexity and scale of such models may not be justified in medical imaging and subject to the available resource budget. Further increasing the number of representative feature maps for the classification task decreases the model explainability. The current data normalization practice is fixed prior to model development and discounting the specification of the data domain. Acknowledging these issues, the current work proposed a new scalable model family called PlexusNet; the block architecture and model scaling by the network's depth, width, and branch regulate PlexusNet's architecture. The efficient computation costs outlined the dimensions of PlexusNet scaling and design. PlexusNet includes a new learnable data normalization algorithm for better data generalization. We applied a simple yet effective neural architecture search to design PlexusNet tailored to five clinical classification problems that achieve a performance noninferior to the SOTA models ResNet-18 and EfficientNet B0/1. It also does so with lower parameter capacity and representative feature maps in ten-fold ranges than the smallest SOTA models with comparable performance. The visualization of representative features revealed distinguishable clusters associated with categories based on latent features generated by PlexusNet. The package and source code are at https://github.com/oeminaga/PlexusNet.git.
2895.
Dong, Peijie; Niu, Xin; Li, Lujun; Tian, Zhiliang; Wang, Xiaodong; Wei, Zimian; Pan, Hengyue; Li, Dongsheng
RD-NAS: Enhancing One-shot Supernet Ranking Ability via Ranking Distillation from Zero-cost Proxies Technical Report
2023.
@techreport{DBLP:journals/corr/abs-2301-09850,
title = {RD-NAS: Enhancing One-shot Supernet Ranking Ability via Ranking Distillation from Zero-cost Proxies},
author = {Peijie Dong and Xin Niu and Lujun Li and Zhiliang Tian and Xiaodong Wang and Zimian Wei and Hengyue Pan and Dongsheng Li},
url = {https://doi.org/10.48550/arXiv.2301.09850},
doi = {10.48550/arXiv.2301.09850},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {CoRR},
volume = {abs/2301.09850},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2894.
Zolnouri, Mahdi; Lakhmiri, Dounia; Tribes, Christophe; Sari, Eyyüb; Digabel, Sébastien Le
Efficient Training Under Limited Resources Technical Report
2023.
@techreport{DBLP:journals/corr/abs-2301-09264,
title = {Efficient Training Under Limited Resources},
author = {Mahdi Zolnouri and Dounia Lakhmiri and Christophe Tribes and Eyyüb Sari and Sébastien Le Digabel},
url = {https://doi.org/10.48550/arXiv.2301.09264},
doi = {10.48550/arXiv.2301.09264},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {CoRR},
volume = {abs/2301.09264},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2893.
Thomas, Jibin B.; K.V., Shihabudheen
Neural architecture search algorithm to optimize deep Transformer model for fault detection in electrical power distribution systems Journal Article
In: Engineering Applications of Artificial Intelligence, vol. 120, pp. 105890, 2023, ISSN: 0952-1976.
@article{THOMAS2023105890,
title = {Neural architecture search algorithm to optimize deep Transformer model for fault detection in electrical power distribution systems},
author = {Jibin B. Thomas and Shihabudheen K.V.},
url = {https://www.sciencedirect.com/science/article/pii/S095219762300074X},
doi = {https://doi.org/10.1016/j.engappai.2023.105890},
issn = {0952-1976},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {Engineering Applications of Artificial Intelligence},
volume = {120},
pages = {105890},
abstract = {This paper proposes a neural architecture search algorithm for obtaining an optimum Transformer model to detect and localize different power system faults and uncertain conditions, such as symmetrical shunt faults, unsymmetrical shunt faults, high-impedance faults, switching conditions (capacitor switching, load switching, transformer switching, DG switching and feeder switching), insulator leakage and transformer inrush current in a distribution system. The Transformer model was proposed to tackle the high memory consumption of the deep CNN attention models and the long-term dependency problem of the RNN attention models. There exist different types of attention mechanisms and feedforward networks for designing a Transformer architecture. Hand engineering of these layers can be inefficient and time-consuming. Therefore, this paper makes use of the Differential Architecture Search (DARTS) algorithm to automatically generate optimal Transformer architectures with less search time cost. The algorithm achieves this by making the search process differentiable to architecture hyperparameters thus making the network search process an end-to-end problem. The proposed model attempts to automatically detect faults in a bus using current measurements from distant monitoring points. The proposed fault analysis was conducted on the standard IEEE 14 bus distribution system and the VSB power line fault detection database. The proposed model was found to produce better performance on the test database when evaluated using F1-Score (99.4% for fault type classification and 97.7% for fault location classification), Matthews Correlation Coefficient (MCC) (99.3% for fault type classification and 97.6% for fault location classification), accuracy and Area Under the Curve (AUC). The architecture transferability of the proposed method was also studied using real-world power line data for fault detection.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
This paper proposes a neural architecture search algorithm for obtaining an optimum Transformer model to detect and localize different power system faults and uncertain conditions, such as symmetrical shunt faults, unsymmetrical shunt faults, high-impedance faults, switching conditions (capacitor switching, load switching, transformer switching, DG switching and feeder switching), insulator leakage and transformer inrush current in a distribution system. The Transformer model was proposed to tackle the high memory consumption of the deep CNN attention models and the long-term dependency problem of the RNN attention models. There exist different types of attention mechanisms and feedforward networks for designing a Transformer architecture. Hand engineering of these layers can be inefficient and time-consuming. Therefore, this paper makes use of the Differential Architecture Search (DARTS) algorithm to automatically generate optimal Transformer architectures with less search time cost. The algorithm achieves this by making the search process differentiable to architecture hyperparameters thus making the network search process an end-to-end problem. The proposed model attempts to automatically detect faults in a bus using current measurements from distant monitoring points. The proposed fault analysis was conducted on the standard IEEE 14 bus distribution system and the VSB power line fault detection database. The proposed model was found to produce better performance on the test database when evaluated using F1-Score (99.4% for fault type classification and 97.7% for fault location classification), Matthews Correlation Coefficient (MCC) (99.3% for fault type classification and 97.6% for fault location classification), accuracy and Area Under the Curve (AUC). The architecture transferability of the proposed method was also studied using real-world power line data for fault detection.
2892.
An, Yang; Zhang, Changsheng; Zheng, Xuanyu
Knowledge reconstruction assisted evolutionary algorithm for neural network architecture search Journal Article
In: Knowledge-Based Systems, vol. 264, pp. 110341, 2023, ISSN: 0950-7051.
@article{AN2023110341,
title = {Knowledge reconstruction assisted evolutionary algorithm for neural network architecture search},
author = {Yang An and Changsheng Zhang and Xuanyu Zheng},
url = {https://www.sciencedirect.com/science/article/pii/S0950705123000916},
doi = {https://doi.org/10.1016/j.knosys.2023.110341},
issn = {0950-7051},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {Knowledge-Based Systems},
volume = {264},
pages = {110341},
abstract = {Neural architecture search (NAS) aims to provide a manual-free search method for obtaining robust and high-performance neural network structures. However, limited search space, weak empirical reusability, and low search efficiency limit the performance of NAS. This study proposes an evolutionary knowledge-reconstruction-assisted method for neural network architecture searches. First, a search space construction method based on network blocks with a-priori knowledge of the network morphism is proposed. This can reduce the computational burden and the time required for the search process while increasing the diversity of the search space. Next, a hierarchical variable-length coding strategy is designed for application to the complete evolutionary algorithm; this strategy divides the neural network into two layers for coding, satisfies the need for decoding with neural network weights, and achieves coding of neural network structures with different depths. Furthermore, the complete differential evolution algorithm is used as the search strategy, thus providing a new possibility of using the search space based on network morphism for applications related to evolutionary algorithms. In addition, the results of comparison experiments conducted on CIFAR10 and CIFAR100 indicate that the neural networks obtained using this method achieve similar or better classification accuracy compared with other neural network structure search algorithms and manually designed networks, while effectively reducing computational time and resource requirements.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Neural architecture search (NAS) aims to provide a manual-free search method for obtaining robust and high-performance neural network structures. However, limited search space, weak empirical reusability, and low search efficiency limit the performance of NAS. This study proposes an evolutionary knowledge-reconstruction-assisted method for neural network architecture searches. First, a search space construction method based on network blocks with a-priori knowledge of the network morphism is proposed. This can reduce the computational burden and the time required for the search process while increasing the diversity of the search space. Next, a hierarchical variable-length coding strategy is designed for application to the complete evolutionary algorithm; this strategy divides the neural network into two layers for coding, satisfies the need for decoding with neural network weights, and achieves coding of neural network structures with different depths. Furthermore, the complete differential evolution algorithm is used as the search strategy, thus providing a new possibility of using the search space based on network morphism for applications related to evolutionary algorithms. In addition, the results of comparison experiments conducted on CIFAR10 and CIFAR100 indicate that the neural networks obtained using this method achieve similar or better classification accuracy compared with other neural network structure search algorithms and manually designed networks, while effectively reducing computational time and resource requirements.
2891.
Li, Guihong; Yang, Yuedong; Bhardwaj, Kartikeya; Marculescu, Radu
ZiCo: Zero-shot NAS via Inverse Coefficient of Variation on Gradients Proceedings Article
In: ICLR 2023, 2023.
@inproceedings{DBLP:journals/corr/abs-2301-11300,
title = {ZiCo: Zero-shot NAS via Inverse Coefficient of Variation on Gradients},
author = {Guihong Li and Yuedong Yang and Kartikeya Bhardwaj and Radu Marculescu},
url = {https://doi.org/10.48550/arXiv.2301.11300},
doi = {10.48550/arXiv.2301.11300},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
booktitle = {ICLR 2023},
journal = {CoRR},
volume = {abs/2301.11300},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
2890.
Liu, Yukun; Li, Ta; Zhang, Pengyuan; Yan, Yonghong
LWMD: A Comprehensive Compression Platform for End-to-End Automatic Speech Recognition Models Journal Article
In: Applied Sciences, vol. 13, no. 3, 2023, ISSN: 2076-3417.
@article{app13031587,
title = {LWMD: A Comprehensive Compression Platform for End-to-End Automatic Speech Recognition Models},
author = {Yukun Liu and Ta Li and Pengyuan Zhang and Yonghong Yan},
url = {https://www.mdpi.com/2076-3417/13/3/1587},
doi = {10.3390/app13031587},
issn = {2076-3417},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {Applied Sciences},
volume = {13},
number = {3},
abstract = {Recently end-to-end (E2E) automatic speech recognition (ASR) models have achieved promising performance. However, existing models tend to adopt increasing model sizes and suffer from expensive resource consumption for real-world applications. To compress E2E ASR models and obtain smaller model sizes, we propose a comprehensive compression platform named LWMD (light-weight model designing), which consists of two essential parts: a light-weight architecture search (LWAS) framework and a differentiable structured pruning (DSP) algorithm. On the one hand, the LWAS framework adopts the neural architecture search (NAS) technique to automatically search light-weight architectures for E2E ASR models. By integrating different architecture topologies of existing models together, LWAS designs a topology-fused search space. Furthermore, combined with the E2E ASR training criterion, LWAS develops a resource-aware search algorithm to select light-weight architectures from the search space. On the other hand, given the searched architectures, the DSP algorithm performs structured pruning to reduce parameter numbers further. With a Gumbel re-parameter trick, DSP builds a stronger correlation between the pruning criterion and the model performance than conventional pruning methods. And an attention-similarity loss function is further developed for better performance. On two mandarin datasets, Aishell-1 and HKUST, the compression results are well evaluated and analyzed to demonstrate the effectiveness of the LWMD platform.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Recently end-to-end (E2E) automatic speech recognition (ASR) models have achieved promising performance. However, existing models tend to adopt increasing model sizes and suffer from expensive resource consumption for real-world applications. To compress E2E ASR models and obtain smaller model sizes, we propose a comprehensive compression platform named LWMD (light-weight model designing), which consists of two essential parts: a light-weight architecture search (LWAS) framework and a differentiable structured pruning (DSP) algorithm. On the one hand, the LWAS framework adopts the neural architecture search (NAS) technique to automatically search light-weight architectures for E2E ASR models. By integrating different architecture topologies of existing models together, LWAS designs a topology-fused search space. Furthermore, combined with the E2E ASR training criterion, LWAS develops a resource-aware search algorithm to select light-weight architectures from the search space. On the other hand, given the searched architectures, the DSP algorithm performs structured pruning to reduce parameter numbers further. With a Gumbel re-parameter trick, DSP builds a stronger correlation between the pruning criterion and the model performance than conventional pruning methods. And an attention-similarity loss function is further developed for better performance. On two mandarin datasets, Aishell-1 and HKUST, the compression results are well evaluated and analyzed to demonstrate the effectiveness of the LWMD platform.
2889.
Son, David; Putter, Floran; Vogel, Sebastian; Corporaal, Henk
BOMP-NAS: Bayesian Optimization Mixed Precision NAS Technical Report
2023.
@techreport{DBLP:journals/corr/abs-2301-11810,
title = {BOMP-NAS: Bayesian Optimization Mixed Precision NAS},
author = {David Son and Floran Putter and Sebastian Vogel and Henk Corporaal},
url = {https://doi.org/10.48550/arXiv.2301.11810},
doi = {10.48550/arXiv.2301.11810},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {CoRR},
volume = {abs/2301.11810},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2888.
Wang, Yawen; Zhang, Shihua
In: Mathematics, vol. 11, no. 3, 2023, ISSN: 2227-7390.
@article{math11030729,
title = {Prediction of Tumor Lymph Node Metastasis Using Wasserstein Distance-Based Generative Adversarial Networks Combing with Neural Architecture Search for Predicting},
author = {Yawen Wang and Shihua Zhang},
url = {https://www.mdpi.com/2227-7390/11/3/729},
doi = {10.3390/math11030729},
issn = {2227-7390},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
journal = {Mathematics},
volume = {11},
number = {3},
abstract = {Long non-coding RNAs (lncRNAs) play an important role in development and gene expression and can be used as genetic indicators for cancer prediction. Generally, lncRNA expression profiles tend to have small sample sizes with large feature sizes; therefore, insufficient data, especially the imbalance of positive and negative samples, often lead to inaccurate prediction results. In this study, we developed a predictor WGAN-psoNN, constructed with the Wasserstein distance-based generative adversarial network (WGAN) and particle swarm optimization neural network (psoNN) algorithms to predict lymph node metastasis events in tumors by using lncRNA expression profiles. To overcome the complicated manual parameter adjustment process, this is the first time the neural network architecture search (NAS) method has been used to automatically set network parameters and predict lymph node metastasis events via deep learning. In addition, the algorithm makes full use of the advantages of WGAN to generate samples to solve the problem of imbalance between positive and negative samples in the data set. On the other hand, by constructing multiple GAN networks, Wasserstein distance was used to select the optimal sample generation. Comparative experiments were conducted on eight representative cancer-related lncRNA expression profile datasets; the prediction results demonstrate the effectiveness and robustness of the newly proposed method. Thus, the model dramatically reduces the requirement for deep learning for data quantity and the difficulty of architecture selection and has the potential to be applied to other classification problems.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Long non-coding RNAs (lncRNAs) play an important role in development and gene expression and can be used as genetic indicators for cancer prediction. Generally, lncRNA expression profiles tend to have small sample sizes with large feature sizes; therefore, insufficient data, especially the imbalance of positive and negative samples, often lead to inaccurate prediction results. In this study, we developed a predictor WGAN-psoNN, constructed with the Wasserstein distance-based generative adversarial network (WGAN) and particle swarm optimization neural network (psoNN) algorithms to predict lymph node metastasis events in tumors by using lncRNA expression profiles. To overcome the complicated manual parameter adjustment process, this is the first time the neural network architecture search (NAS) method has been used to automatically set network parameters and predict lymph node metastasis events via deep learning. In addition, the algorithm makes full use of the advantages of WGAN to generate samples to solve the problem of imbalance between positive and negative samples in the data set. On the other hand, by constructing multiple GAN networks, Wasserstein distance was used to select the optimal sample generation. Comparative experiments were conducted on eight representative cancer-related lncRNA expression profile datasets; the prediction results demonstrate the effectiveness and robustness of the newly proposed method. Thus, the model dramatically reduces the requirement for deep learning for data quantity and the difficulty of architecture selection and has the potential to be applied to other classification problems.
2887.
Vidnerová, Petra; Kalina, Jan
Multi-objective Bayesian Optimization for Neural Architecture Search Proceedings Article
In: Rutkowski, Leszek; Scherer, Rafał; Korytkowski, Marcin; Pedrycz, Witold; Tadeusiewicz, Ryszard; Zurada, Jacek M. (Ed.): Ärtificial Intelligence and Soft Computing", pp. 144–153, Springer International Publishing, Cham, 2023, ISBN: 978-3-031-23492-7.
@inproceedings{10.1007/978-3-031-23492-7_13,
title = {Multi-objective Bayesian Optimization for Neural Architecture Search},
author = {Petra Vidnerová and Jan Kalina},
editor = {Leszek Rutkowski and Rafał Scherer and Marcin Korytkowski and Witold Pedrycz and Ryszard Tadeusiewicz and Jacek M. Zurada},
url = {https://link.springer.com/chapter/10.1007/978-3-031-23492-7_13},
isbn = {978-3-031-23492-7},
year = {2023},
date = {2023-01-01},
urldate = {2023-01-01},
booktitle = {Ärtificial Intelligence and Soft Computing"},
pages = {144--153},
publisher = {Springer International Publishing},
address = {Cham},
abstract = {Ä novel multi-objective algorithm denoted as MO-BayONet is proposed for the Neural Architecture Search (NAS) in this paper. The method based on Bayesian optimization encodes the candidate architectures directly as lists of layers and constructs an extra feature vector for the corresponding surrogate model. The general method allows to accompany the search for the optimal network by additional criteria besides the network performance. The NAS method is applied to combine classification accuracy with network size on two benchmark datasets here. The results indicate that MO-BayONet is able to outperform an available genetic algorithm based approach."},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Ä novel multi-objective algorithm denoted as MO-BayONet is proposed for the Neural Architecture Search (NAS) in this paper. The method based on Bayesian optimization encodes the candidate architectures directly as lists of layers and constructs an extra feature vector for the corresponding surrogate model. The general method allows to accompany the search for the optimal network by additional criteria besides the network performance. The NAS method is applied to combine classification accuracy with network size on two benchmark datasets here. The results indicate that MO-BayONet is able to outperform an available genetic algorithm based approach."