Maintained by Difan Deng and Marius Lindauer.
The following list considers papers related to neural architecture search. It is by no means complete. If you miss a paper on the list, please let us know.
Please note that although NAS methods steadily improve, the quality of empirical evaluations in this field are still lagging behind compared to other areas in machine learning, AI and optimization. We would therefore like to share some best practices for empirical evaluations of NAS methods, which we believe will facilitate sustained and measurable progress in the field. If you are interested in a teaser, please read our blog post or directly jump to our checklist.
Transformers have gained increasing popularity in different domains. For a comprehensive list of papers focusing on Neural Architecture Search for Transformer-Based spaces, the awesome-transformer-search repo is all you need.
2022
2076.
Chu, Xiaowen; He, Xin
MedPipe: End-to-End Joint Search of Data Augmentation Policy and Neural Architecture for 3D Medical Image Classification Journal Article
In: 2022.
@article{Chu2022,
title = {MedPipe: End-to-End Joint Search of Data Augmentation Policy and Neural Architecture for 3D Medical Image Classification},
author = {Xiaowen Chu and Xin He},
url = {https://www.techrxiv.org/articles/preprint/MedPipe_End-to-End_Joint_Search_of_Data_Augmentation_Policy_and_Neural_Architecture_for_3D_Medical_Image_Classification/19513780},
doi = {10.36227/techrxiv.19513780.v1},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2075.
Qian, Jikuan; Li, Rui; Yang, Xin; Huang, Yuhao; Luo, Mingyuan; Lin, Zehui; Hong, Wenhui; Huang, Ruobing; Fan, Haining; Ni, Dong; Cheng, Jun
HASA: Hybrid Architecture Search with Aggregation Strategy for Echinococcosis Classification and Ovary Segmentation in Ultrasound Images Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2204-06697,
title = {HASA: Hybrid Architecture Search with Aggregation Strategy for Echinococcosis Classification and Ovary Segmentation in Ultrasound Images},
author = {Jikuan Qian and Rui Li and Xin Yang and Yuhao Huang and Mingyuan Luo and Zehui Lin and Wenhui Hong and Ruobing Huang and Haining Fan and Dong Ni and Jun Cheng},
url = {https://doi.org/10.48550/arXiv.2204.06697},
doi = {10.48550/arXiv.2204.06697},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2204.06697},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2074.
Yu, Xinyi; Wang, Xiaowei; Zhang, Mingyang; Rong, Jintao; Ou, Linlin
De-IReps: Searching for improved Re-parameterizing Architecture based on Differentiable Evolution Strategy Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2204-06403,
title = {De-IReps: Searching for improved Re-parameterizing Architecture based on Differentiable Evolution Strategy},
author = {Xinyi Yu and Xiaowei Wang and Mingyang Zhang and Jintao Rong and Linlin Ou},
url = {https://doi.org/10.48550/arXiv.2204.06403},
doi = {10.48550/arXiv.2204.06403},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2204.06403},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2073.
Escovedo, Tatiana; Figueiredo, Karla; Szwarcman, Daniela; Vellasco, Marley
Neuroevolutionary Models Based on Quantum-Inspired Evolutionary Algorithms Book Chapter
In: Smith, Alice E. (Ed.): Women in Computational Intelligence: Key Advances and Perspectives on Emerging Topics, pp. 303–333, Springer International Publishing, Cham, 2022, ISBN: 978-3-030-79092-9.
@inbook{Escovedo2022,
title = {Neuroevolutionary Models Based on Quantum-Inspired Evolutionary Algorithms},
author = {Tatiana Escovedo and Karla Figueiredo and Daniela Szwarcman and Marley Vellasco},
editor = {Alice E. Smith},
url = {https://doi.org/10.1007/978-3-030-79092-9_14},
doi = {10.1007/978-3-030-79092-9_14},
isbn = {978-3-030-79092-9},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
booktitle = {Women in Computational Intelligence: Key Advances and Perspectives on Emerging Topics},
pages = {303--333},
publisher = {Springer International Publishing},
address = {Cham},
abstract = {Ärtificial neural networks are powerful and flexible models that have proved useful in solving a countless number of real-world problems. More recently, deep neural networks have gained the attention of the machine learning community for surpassing human-level performance in a variety of tasks. To reach these excellent results, an expert needs to spend a significant amount of time designing the neural configuration, such as the best architecture, input variables, and training parameters, with long trial-and-error sessions. Neuroevolution is a method that tries to automate this design process by applied evolutionary computation algorithms to optimise the neural network hyperparameters. However, the high computational cost associated with the use of traditional evolutionary algorithms for neural network configuration is still an issue. In this regard, quantum-inspired evolutionary algorithms emerged as an excellent optimisation method for neuroevolution, as they provide faster convergence, with a reduced computational cost. This chapter presents an overview of quantum-inspired evolutionary algorithms and their application to the configuration of different neural network models, such as Multi-Layer Perceptrons, Echo-State Networks and Convolutional Neural Networks. Applications in system identification, image classification and concept drift environments are also detailed, with promising results."},
keywords = {},
pubstate = {published},
tppubtype = {inbook}
}
Ärtificial neural networks are powerful and flexible models that have proved useful in solving a countless number of real-world problems. More recently, deep neural networks have gained the attention of the machine learning community for surpassing human-level performance in a variety of tasks. To reach these excellent results, an expert needs to spend a significant amount of time designing the neural configuration, such as the best architecture, input variables, and training parameters, with long trial-and-error sessions. Neuroevolution is a method that tries to automate this design process by applied evolutionary computation algorithms to optimise the neural network hyperparameters. However, the high computational cost associated with the use of traditional evolutionary algorithms for neural network configuration is still an issue. In this regard, quantum-inspired evolutionary algorithms emerged as an excellent optimisation method for neuroevolution, as they provide faster convergence, with a reduced computational cost. This chapter presents an overview of quantum-inspired evolutionary algorithms and their application to the configuration of different neural network models, such as Multi-Layer Perceptrons, Echo-State Networks and Convolutional Neural Networks. Applications in system identification, image classification and concept drift environments are also detailed, with promising results."
2072.
Shen, Junhong; Khodak, Mikhail; Talwalkar, Ameet
Efficient Architecture Search for Diverse Tasks Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2204-07554,
title = {Efficient Architecture Search for Diverse Tasks},
author = {Junhong Shen and Mikhail Khodak and Ameet Talwalkar},
url = {https://doi.org/10.48550/arXiv.2204.07554},
doi = {10.48550/arXiv.2204.07554},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2204.07554},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2071.
Xue, Fanghui; Yang, Biao; Qi, Yingyong; Xin, Jack
Searching Intrinsic Dimensions of Vision Transformers Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2204-07722,
title = {Searching Intrinsic Dimensions of Vision Transformers},
author = {Fanghui Xue and Biao Yang and Yingyong Qi and Jack Xin},
url = {https://doi.org/10.48550/arXiv.2204.07722},
doi = {10.48550/arXiv.2204.07722},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2204.07722},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2070.
Zhou, Yanqi; Dong, Xuanyi; Meng, Tianjian; Tan, Mingxing; Akin, Berkin; Peng, Daiyi; Yazdanbakhsh, Amir; Huang, Da; Narayanaswami, Ravi; Laudon, James
Towards the Co-design of Neural Networks and Accelerators Technical Report
2022.
@techreport{DBLP:conf/mlsys/ZhouDMTAPYHNL22,
title = {Towards the Co-design of Neural Networks and Accelerators},
author = {Yanqi Zhou and Xuanyi Dong and Tianjian Meng and Mingxing Tan and Berkin Akin and Daiyi Peng and Amir Yazdanbakhsh and Da Huang and Ravi Narayanaswami and James Laudon},
editor = {Diana Marculescu and Yuejie Chi and Carole-Jean Wu},
url = {https://proceedings.mlsys.org/paper/2022/hash/31fefc0e570cb3860f2a6d4b38c6490d-Abstract.html},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
booktitle = {Proceedings of Machine Learning and Systems 2022, MLSys 2022, Santa
Clara, CA, USA, August 29 - September 1, 2022},
publisher = {mlsys.org},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2069.
Huang, Yimin; Li, Yujun; Ye, Hanrong; Li, Zhenguo; Zhang, Zhihua
Improving Model Training with Multi-fidelity Hyperparameter Evaluation Proceedings Article
In: Marculescu, Diana; Chi, Yuejie; Wu, Carole-Jean (Ed.): Proceedings of Machine Learning and Systems 2022, MLSys 2022, Santa Clara, CA, USA, August 29 - September 1, 2022, mlsys.org, 2022.
@inproceedings{DBLP:conf/mlsys/HuangLYLZ22,
title = {Improving Model Training with Multi-fidelity Hyperparameter Evaluation},
author = {Yimin Huang and Yujun Li and Hanrong Ye and Zhenguo Li and Zhihua Zhang},
editor = {Diana Marculescu and Yuejie Chi and Carole-Jean Wu},
url = {https://proceedings.mlsys.org/paper/2022/hash/a3c65c2974270fd093ee8a9bf8ae7d0b-Abstract.html},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
booktitle = {Proceedings of Machine Learning and Systems 2022, MLSys 2022, Santa
Clara, CA, USA, August 29 - September 1, 2022},
publisher = {mlsys.org},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
2068.
Wang, Huanyu; Zhang, Yongshun; Wu, Jianxin
Versatile, full‐spectrum, and swift network sampling for model generation Journal Article
In: Pattern Recognition, vol. 129, pp. 108729, 2022, ISSN: 0031-3203.
@article{WANG2022108729,
title = {Versatile, full‐spectrum, and swift network sampling for model generation},
author = {Huanyu Wang and Yongshun Zhang and Jianxin Wu},
url = {https://www.sciencedirect.com/science/article/pii/S0031320322002102},
doi = {https://doi.org/10.1016/j.patcog.2022.108729},
issn = {0031-3203},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Pattern Recognition},
volume = {129},
pages = {108729},
abstract = {Given one task, it is difficult to generate CNN models for many different hardware platforms with extremely diverse computing power for this task. Repeating network pruning or architecture search for each platform is very time-consuming. In this paper, we propose properties that are required for this model generation problem: versatile (fits diverse applications and network structures), full-spectrum (generates models for devices with tiny to gigantic computing power), and swift (total training time for all platforms is short, and generated models have low latency). We show that existing methods do not satisfy these requirements and propose a VFS method (the V/F/S represents Versatile/Full-spectrum/Swift, respectively). VFS uses importance sampling to sample many submodels with versatile structures and with different input image resolutions. We propose new fine-tuning strategies that only need to fine-tune a best candidate submodel for few epochs for each platform. VFS satisfies all three requirements. It generates versatile models with low latency for diverse applications, is suitable for devices with a wide range of computing power differences, and the models which are generated by VFS achieve state-of-the-art accuracy.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Given one task, it is difficult to generate CNN models for many different hardware platforms with extremely diverse computing power for this task. Repeating network pruning or architecture search for each platform is very time-consuming. In this paper, we propose properties that are required for this model generation problem: versatile (fits diverse applications and network structures), full-spectrum (generates models for devices with tiny to gigantic computing power), and swift (total training time for all platforms is short, and generated models have low latency). We show that existing methods do not satisfy these requirements and propose a VFS method (the V/F/S represents Versatile/Full-spectrum/Swift, respectively). VFS uses importance sampling to sample many submodels with versatile structures and with different input image resolutions. We propose new fine-tuning strategies that only need to fine-tune a best candidate submodel for few epochs for each platform. VFS satisfies all three requirements. It generates versatile models with low latency for diverse applications, is suitable for devices with a wide range of computing power differences, and the models which are generated by VFS achieve state-of-the-art accuracy.
2067.
Künzel, Steven; Meyer-Nieberg, Silja
ÄNN-EMOA: Evolving Neural Networks Efficiently Proceedings Article
In: Laredo, Juan Luis Jiménez; Hidalgo, J. Ignacio; Babaagba, Kehinde Oluwatoyin (Ed.): Äpplications of Evolutionary Computation", pp. 402–417, Springer International Publishing, Cham, 2022, ISBN: 978-3-031-02462-7.
@inproceedings{10.1007/978-3-031-02462-7_26,
title = {ÄNN-EMOA: Evolving Neural Networks Efficiently},
author = {Steven Künzel and Silja Meyer-Nieberg},
editor = {Juan Luis Jiménez Laredo and J. Ignacio Hidalgo and Kehinde Oluwatoyin Babaagba},
isbn = {978-3-031-02462-7},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
booktitle = {Äpplications of Evolutionary Computation"},
pages = {402--417},
publisher = {Springer International Publishing},
address = {Cham},
abstract = {Multi-objective neuroevolution is a research field of growing importance within reinforcement learning. This paper introduces ANN-EMOA, a novel multi-objective neuroevolutionary algorithm that is inspired by nNEAT and aims at high efficiency, usability, and comprehensibility. To that end it applies a simple encoding and efficient variation operators. Diversity plays a key role in evolutionary computation. For this reason, we apply the Riesz s-energy to foster diversity explicitly. This paper also develops a new efficient approach to determine the individual Riesz s-energy contribution of each solution within a set. To assess the performance of the new ANN-EMOA it is compared to nNEAT and NEAT-MODS, two multi-objective variants of NEAT, in the multi-objective Double Pole Balancing problem. While other domains and more complex test cases need to be investigated, these promising first results show that ANN-EMOA does not only converge faster and to higher quality-levels than its competitors, but it also maintains more compact network-genomes and shows convincing performance even with comparably small populations.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Multi-objective neuroevolution is a research field of growing importance within reinforcement learning. This paper introduces ANN-EMOA, a novel multi-objective neuroevolutionary algorithm that is inspired by nNEAT and aims at high efficiency, usability, and comprehensibility. To that end it applies a simple encoding and efficient variation operators. Diversity plays a key role in evolutionary computation. For this reason, we apply the Riesz s-energy to foster diversity explicitly. This paper also develops a new efficient approach to determine the individual Riesz s-energy contribution of each solution within a set. To assess the performance of the new ANN-EMOA it is compared to nNEAT and NEAT-MODS, two multi-objective variants of NEAT, in the multi-objective Double Pole Balancing problem. While other domains and more complex test cases need to be investigated, these promising first results show that ANN-EMOA does not only converge faster and to higher quality-levels than its competitors, but it also maintains more compact network-genomes and shows convincing performance even with comparably small populations.
2066.
Saha, Swapnil Sayan; Sandha, Sandeep; Garcia, Luis; Srivastava, Mani
TinyOdom: Hardware-Aware Efficient Neural Inertial Navigation Journal Article
In: Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies, vol. 6, pp. 32, 2022.
@article{article,
title = {TinyOdom: Hardware-Aware Efficient Neural Inertial Navigation},
author = {Swapnil Sayan Saha and Sandeep Sandha and Luis Garcia and Mani Srivastava},
url = {https://www.researchgate.net/publication/360075622_TinyOdom_Hardware-Aware_Efficient_Neural_Inertial_Navigation},
doi = {10.1145/3534594},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Proceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies},
volume = {6},
pages = {32},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2065.
Wang, Fuyi; Zhang, Leo Yu; Pan, Lei; Hu, Shengshan; Doss, Robin
Towards Privacy-Preserving Neural Architecture Search Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2204-10958,
title = {Towards Privacy-Preserving Neural Architecture Search},
author = {Fuyi Wang and Leo Yu Zhang and Lei Pan and Shengshan Hu and Robin Doss},
url = {https://doi.org/10.48550/arXiv.2204.10958},
doi = {10.48550/arXiv.2204.10958},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2204.10958},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2064.
Li, Chang; Zhang, Zhongzhen; Zhang, Xiaodong; Huang, Guoning; Liu, Yu; Chen, Xun
EEG-based Emotion Recognition via Transformer Neural Architecture Search Journal Article
In: IEEE Transactions on Industrial Informatics, pp. 1-1, 2022.
@article{9763316,
title = {EEG-based Emotion Recognition via Transformer Neural Architecture Search},
author = {Chang Li and Zhongzhen Zhang and Xiaodong Zhang and Guoning Huang and Yu Liu and Xun Chen},
url = {https://ieeexplore.ieee.org/abstract/document/9763316},
doi = {10.1109/TII.2022.3170422},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {IEEE Transactions on Industrial Informatics},
pages = {1-1},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2063.
Zhang, Xixi; Zhao, Haitao; Zhu, Hongbo; Adebisi, Bamidele; Gui, Guan; Gacanin, Haris; Adachi, Fumiyuki
NAS-AMR: Neural Architecture Search Based Automatic Modulation Recognition for Integrated Sensing and Communication Systems Journal Article
In: IEEE Transactions on Cognitive Communications and Networking, pp. 1-1, 2022.
@article{9762373,
title = {NAS-AMR: Neural Architecture Search Based Automatic Modulation Recognition for Integrated Sensing and Communication Systems},
author = {Xixi Zhang and Haitao Zhao and Hongbo Zhu and Bamidele Adebisi and Guan Gui and Haris Gacanin and Fumiyuki Adachi},
url = {https://ieeexplore.ieee.org/abstract/document/9762373},
doi = {10.1109/TCCN.2022.3169740},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {IEEE Transactions on Cognitive Communications and Networking},
pages = {1-1},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2062.
Lu, Zexin; Xia, Wenjun; Huang, Yongqiang; Hou, Mingzheng; Chen, Hu; Shan, Hongming; Zhang, Yi
Low-Dose CT Denoising via Neural Architecture Search Proceedings Article
In: 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI), pp. 1-5, 2022.
@inproceedings{9761513,
title = {Low-Dose CT Denoising via Neural Architecture Search},
author = {Zexin Lu and Wenjun Xia and Yongqiang Huang and Mingzheng Hou and Hu Chen and Hongming Shan and Yi Zhang},
url = {https://ieeexplore.ieee.org/abstract/document/9761513},
doi = {10.1109/ISBI52829.2022.9761513},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
booktitle = {2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI)},
pages = {1-5},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
2061.
Cai, Jie; Wang, Xin; Guan, Chaoyu; Tang, Yateng; Xu, Jin; Zhong, Bin; Zhu, Wenwu
Multimodal Continual Graph Learning with Neural Architecture Search Proceedings Article
In: Laforest, Frédérique; Troncy, Raphaël; Simperl, Elena; Agarwal, Deepak; Gionis, Aristides; Herman, Ivan; Médini, Lionel (Ed.): WWW '22: The ACM Web Conference 2022, Virtual Event, Lyon, France, April 25 - 29, 2022, pp. 1292–1300, ACM, 2022.
@inproceedings{DBLP:conf/www/Cai0GTXZ022,
title = {Multimodal Continual Graph Learning with Neural Architecture Search},
author = {Jie Cai and Xin Wang and Chaoyu Guan and Yateng Tang and Jin Xu and Bin Zhong and Wenwu Zhu},
editor = {Frédérique Laforest and Raphaël Troncy and Elena Simperl and Deepak Agarwal and Aristides Gionis and Ivan Herman and Lionel Médini},
url = {https://doi.org/10.1145/3485447.3512176},
doi = {10.1145/3485447.3512176},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
booktitle = {WWW '22: The ACM Web Conference 2022, Virtual Event, Lyon, France,
April 25 - 29, 2022},
pages = {1292--1300},
publisher = {ACM},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
2060.
Liang, Zixuan; Sun, Yanan
Automating Neural Architecture Design without Search Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2204-11838,
title = {Automating Neural Architecture Design without Search},
author = {Zixuan Liang and Yanan Sun},
url = {https://doi.org/10.48550/arXiv.2204.11838},
doi = {10.48550/arXiv.2204.11838},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2204.11838},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2059.
Cheng, Mingyue; Liu, Zhiding; Liu, Qi; Ge, Shenyang; Chen, Enhong
Towards Automatic Discovering of Deep Hybrid Network Architecture for Sequential Recommendation Proceedings Article
In: Proceedings of the ACM Web Conference 2022, pp. 1923–1932, Association for Computing Machinery, Virtual Event, Lyon, France, 2022, ISBN: 9781450390965.
@inproceedings{10.1145/3485447.3512066,
title = {Towards Automatic Discovering of Deep Hybrid Network Architecture for Sequential Recommendation},
author = {Mingyue Cheng and Zhiding Liu and Qi Liu and Shenyang Ge and Enhong Chen},
url = {https://doi.org/10.1145/3485447.3512066},
doi = {10.1145/3485447.3512066},
isbn = {9781450390965},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
booktitle = {Proceedings of the ACM Web Conference 2022},
pages = {1923–1932},
publisher = {Association for Computing Machinery},
address = {Virtual Event, Lyon, France},
series = {WWW '22},
abstract = {Recent years have witnessed great success in deep learning-based sequential recommendation (SR), which can provide more timely and accurate recommendations. One of the most effective deep SR architectures is to stack high-performance residual blocks, e.g., prevalent self-attentive and convolutional operations, for capturing long- and short-range dependence of sequential behaviors. By carefully revisiting previous models, we observe: 1) simple architecture modification of gating each residual connection can help us train deeper SR models and yield significant improvements; 2) compared with self-attention mechanism, stacking of convolution layers also can cover each item of the whole sequential behaviors and achieve competitive or even superior performance. Guided by these findings, it is meaningful to design a deeper hybrid SR model to ensemble the capacity of both self-attentive and convolutional architectures for SR tasks. In this work, we aim to achieve this goal in the automatic algorithm sense, and propose NASR, an efficient neural architecture search (NAS) method that can automatically select the architecture operation on each layer. Specifically, we firstly design a Table-like search space, involving both self-attentive and convolutional-based SR architectures in a flexible manner. In the search phase, we leverage weight-sharing supernets to encode the entire search space, and further propose to factorize the whole supernet into blocks to ensure the potential candidate SR architectures can be fully trained. Owning to lacking supervisions, we train each block-wise supernet with a self-supervised contrastive optimization scheme, in which the training signals are constructed by conducting data augmentation on original sequential behaviors. The empirical studies show that the discovered deep hybrid network architectures can exhibit substantial improvements over compared baselines, indicating the practicality of searching deep hybrid network architectures on SR tasks. Notably, we show the discovered architecture also enjoys good generalizability and transferability among different datasets.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Recent years have witnessed great success in deep learning-based sequential recommendation (SR), which can provide more timely and accurate recommendations. One of the most effective deep SR architectures is to stack high-performance residual blocks, e.g., prevalent self-attentive and convolutional operations, for capturing long- and short-range dependence of sequential behaviors. By carefully revisiting previous models, we observe: 1) simple architecture modification of gating each residual connection can help us train deeper SR models and yield significant improvements; 2) compared with self-attention mechanism, stacking of convolution layers also can cover each item of the whole sequential behaviors and achieve competitive or even superior performance. Guided by these findings, it is meaningful to design a deeper hybrid SR model to ensemble the capacity of both self-attentive and convolutional architectures for SR tasks. In this work, we aim to achieve this goal in the automatic algorithm sense, and propose NASR, an efficient neural architecture search (NAS) method that can automatically select the architecture operation on each layer. Specifically, we firstly design a Table-like search space, involving both self-attentive and convolutional-based SR architectures in a flexible manner. In the search phase, we leverage weight-sharing supernets to encode the entire search space, and further propose to factorize the whole supernet into blocks to ensure the potential candidate SR architectures can be fully trained. Owning to lacking supervisions, we train each block-wise supernet with a self-supervised contrastive optimization scheme, in which the training signals are constructed by conducting data augmentation on original sequential behaviors. The empirical studies show that the discovered deep hybrid network architectures can exhibit substantial improvements over compared baselines, indicating the practicality of searching deep hybrid network architectures on SR tasks. Notably, we show the discovered architecture also enjoys good generalizability and transferability among different datasets.
2058.
Sun, Qigong; Li, Xiufang; Jiao, Licheng; Ren, Yan; Shang, Fanhua; Liu, Fang
Fast and Effective: A Novel Sequential Single-Path Search for Mixed-Precision-Quantized Networks Journal Article
In: IEEE Transactions on Cybernetics, pp. 1-13, 2022.
@article{9762902,
title = {Fast and Effective: A Novel Sequential Single-Path Search for Mixed-Precision-Quantized Networks},
author = {Qigong Sun and Xiufang Li and Licheng Jiao and Yan Ren and Fanhua Shang and Fang Liu},
url = {https://ieeexplore.ieee.org/abstract/document/9762902},
doi = {10.1109/TCYB.2022.3164285},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {IEEE Transactions on Cybernetics},
pages = {1-13},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2057.
Peng, Yameng; Song, Andy; Ciesielski, Vic; Fayek, Haytham M.; Chang, Xiaojun
PRE-NAS: Predictor-assisted Evolutionary Neural Architecture Search Journal Article
In: GECCO, vol. abs/2204.12726, 2022.
@article{DBLP:journals/corr/abs-2204-12726,
title = {PRE-NAS: Predictor-assisted Evolutionary Neural Architecture Search},
author = {Yameng Peng and Andy Song and Vic Ciesielski and Haytham M. Fayek and Xiaojun Chang},
url = {https://doi.org/10.48550/arXiv.2204.12726},
doi = {10.48550/arXiv.2204.12726},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {GECCO},
volume = {abs/2204.12726},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2056.
Ji, Junkai; Zhao, Jiajun; Lin, Qiuzhen; Tan, Kay Chen
Competitive Decomposition-Based Multiobjective Architecture Search for the Dendritic Neural Model Journal Article
In: IEEE Transactions on Cybernetics, pp. 1-14, 2022.
@article{9764663,
title = {Competitive Decomposition-Based Multiobjective Architecture Search for the Dendritic Neural Model},
author = {Junkai Ji and Jiajun Zhao and Qiuzhen Lin and Kay Chen Tan},
url = {https://ieeexplore.ieee.org/abstract/document/9764663},
doi = {10.1109/TCYB.2022.3165374},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {IEEE Transactions on Cybernetics},
pages = {1-14},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2055.
Tang, Cheng; Ji, Junkai; Lin, Qiuzhen; Zhou, Yan
Evolutionary Neural Architecture Design of Liquid State Machine for Image Classification Proceedings Article
In: ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 91-95, 2022.
@inproceedings{9747040,
title = {Evolutionary Neural Architecture Design of Liquid State Machine for Image Classification},
author = {Cheng Tang and Junkai Ji and Qiuzhen Lin and Yan Zhou},
url = {https://ieeexplore.ieee.org/abstract/document/9747040},
doi = {10.1109/ICASSP43922.2022.9747040},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
booktitle = {ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
pages = {91-95},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
2054.
Nair, Saeejith; Abbasi, Saad; Wong, Alexander; Shafiee, Mohammad Javad
MAPLE-Edge: A Runtime Latency Predictor for Edge Devices Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2204-12950,
title = {MAPLE-Edge: A Runtime Latency Predictor for Edge Devices},
author = {Saeejith Nair and Saad Abbasi and Alexander Wong and Mohammad Javad Shafiee},
url = {https://doi.org/10.48550/arXiv.2204.12950},
doi = {10.48550/arXiv.2204.12950},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2204.12950},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2053.
Wen, Tiancheng; Ding, Zhonggan; Yao, Yongqiang; Wang, Yaxiong; Qian, Xueming
PicassoNet: Searching Adaptive Architecture for Efficient Facial Landmark Localization Journal Article
In: IEEE Transactions on Neural Networks and Learning Systems, pp. 1-12, 2022.
@article{9764821,
title = {PicassoNet: Searching Adaptive Architecture for Efficient Facial Landmark Localization},
author = {Tiancheng Wen and Zhonggan Ding and Yongqiang Yao and Yaxiong Wang and Xueming Qian},
url = {https://ieeexplore.ieee.org/abstract/document/9764821},
doi = {10.1109/TNNLS.2022.3167743},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {IEEE Transactions on Neural Networks and Learning Systems},
pages = {1-12},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2052.
Li, Di; Bai, Yunpeng; Bai, Zongwen; Li, Ying; Shang, Changjing; Shen, Qiang
Decomposed Neural Architecture Search for image denoising Journal Article
In: Applied Soft Computing, vol. 124, pp. 108914, 2022, ISSN: 1568-4946.
@article{LI2022108914,
title = {Decomposed Neural Architecture Search for image denoising},
author = {Di Li and Yunpeng Bai and Zongwen Bai and Ying Li and Changjing Shang and Qiang Shen},
url = {https://www.sciencedirect.com/science/article/pii/S1568494622002769},
doi = {https://doi.org/10.1016/j.asoc.2022.108914},
issn = {1568-4946},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Applied Soft Computing},
volume = {124},
pages = {108914},
abstract = {In practical applications of deep learning, as the demand for the modeling capability increases, the network size may need to be massively enlarged in response. This may form a significant challenge in practice, especially when facing the dilemma of limited computational resources, making model compression indispensable. It can be time-consuming and interminable to obtain an appropriate network architecture through manual compression. In this paper, we propose an automated method for searching decomposed network architectures, named DNAS (standing for Decomposed Neural Architecture Search). It integrates both tasks of neural architecture search and tensor decomposition based model compression within a unified framework. The method is able to efficiently find a compact network with high performance for image denoising, with respect to memory and runtime. Particularly, using one single V100 GPU, it only takes about 1.5 h to obtain a denoising network on the BSD500 dataset. Experimental results demonstrate that compared with models developed using existing methods, DNAS consumes significantly less inference time and employs much fewer trainable parameters, outperforming existing approaches on both synthetic and real-world denoising datasets.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
In practical applications of deep learning, as the demand for the modeling capability increases, the network size may need to be massively enlarged in response. This may form a significant challenge in practice, especially when facing the dilemma of limited computational resources, making model compression indispensable. It can be time-consuming and interminable to obtain an appropriate network architecture through manual compression. In this paper, we propose an automated method for searching decomposed network architectures, named DNAS (standing for Decomposed Neural Architecture Search). It integrates both tasks of neural architecture search and tensor decomposition based model compression within a unified framework. The method is able to efficiently find a compact network with high performance for image denoising, with respect to memory and runtime. Particularly, using one single V100 GPU, it only takes about 1.5 h to obtain a denoising network on the BSD500 dataset. Experimental results demonstrate that compared with models developed using existing methods, DNAS consumes significantly less inference time and employs much fewer trainable parameters, outperforming existing approaches on both synthetic and real-world denoising datasets.
2051.
Mo, Lingfei; Guan, Xuchen
Neural component search for single image super-resolution Journal Article
In: Signal Processing: Image Communication, vol. 106, pp. 116725, 2022, ISSN: 0923-5965.
@article{MO2022116725,
title = {Neural component search for single image super-resolution},
author = {Lingfei Mo and Xuchen Guan},
url = {https://www.sciencedirect.com/science/article/pii/S0923596522000583},
doi = {https://doi.org/10.1016/j.image.2022.116725},
issn = {0923-5965},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Signal Processing: Image Communication},
volume = {106},
pages = {116725},
abstract = {Deep learning has become the mainstream method in the field of single image super-resolution (SISR), and the neural architecture search has been gradually applied to build SISR networks in a non-hand-crafted way. However, the existing methods can only search the structure of models and the searching speed is slow. To solve this problem, a neural component search (NCS) method is proposed. When searching for SISR networks, the color space and the composition of loss functions during training are also parts of the search space. Under a specific computational constraint, the peak signal noise ratio (PSNR) or structural similarity (SSIM) can be used as the reward to search out an optimal super-resolution network. In addition, a super graph is designed with the idea of parameter sharing to sample adaptive residual dense networks (ARDNs), thus the NCS can complete the search of SISR networks at faster speed compared to existing methods. Experimental results indicate that ARDNs searched by the NCS is competitive with the hand-crafted state-of-the-art networks, and ARDNs achieve favorable performance against state-of-the-art methods with similar computational consumption.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Deep learning has become the mainstream method in the field of single image super-resolution (SISR), and the neural architecture search has been gradually applied to build SISR networks in a non-hand-crafted way. However, the existing methods can only search the structure of models and the searching speed is slow. To solve this problem, a neural component search (NCS) method is proposed. When searching for SISR networks, the color space and the composition of loss functions during training are also parts of the search space. Under a specific computational constraint, the peak signal noise ratio (PSNR) or structural similarity (SSIM) can be used as the reward to search out an optimal super-resolution network. In addition, a super graph is designed with the idea of parameter sharing to sample adaptive residual dense networks (ARDNs), thus the NCS can complete the search of SISR networks at faster speed compared to existing methods. Experimental results indicate that ARDNs searched by the NCS is competitive with the hand-crafted state-of-the-art networks, and ARDNs achieve favorable performance against state-of-the-art methods with similar computational consumption.
2050.
Sun, Hanbo; Wang, Chenyu; Zhu, Zhenhua; Ning, Xuefei; Dai, Guohao; Yang, Huazhong; Wang, Yu
Gibbon: Efficient Co-Exploration of NN Model and Processing-In-Memory Architecture Proceedings Article
In: Bolchini, Cristiana; Verbauwhede, Ingrid; Vatajelu, Ioana (Ed.): 2022 Design, Automation & Test in Europe Conference & Exhibition, DATE 2022, Antwerp, Belgium, March 14-23, 2022, pp. 867–872, IEEE, 2022.
@inproceedings{DBLP:conf/date/SunWZNDYW22,
title = {Gibbon: Efficient Co-Exploration of NN Model and Processing-In-Memory Architecture},
author = {Hanbo Sun and Chenyu Wang and Zhenhua Zhu and Xuefei Ning and Guohao Dai and Huazhong Yang and Yu Wang},
editor = {Cristiana Bolchini and Ingrid Verbauwhede and Ioana Vatajelu},
url = {https://doi.org/10.23919/DATE54114.2022.9774605},
doi = {10.23919/DATE54114.2022.9774605},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
booktitle = {2022 Design, Automation & Test in Europe Conference &
Exhibition, DATE 2022, Antwerp, Belgium, March 14-23, 2022},
pages = {867--872},
publisher = {IEEE},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
2049.
Wang, Aili; Xue, Dong; Wu, Haibin; Gu, Yanfeng
Efficient Convolutional Neural Architecture Search for LiDAR DSM Classification Journal Article
In: IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1-17, 2022.
@article{9766045,
title = {Efficient Convolutional Neural Architecture Search for LiDAR DSM Classification},
author = {Aili Wang and Dong Xue and Haibin Wu and Yanfeng Gu},
url = {https://ieeexplore.ieee.org/abstract/document/9766045},
doi = {10.1109/TGRS.2022.3171520},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {IEEE Transactions on Geoscience and Remote Sensing},
volume = {60},
pages = {1-17},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2048.
Mun, Jiwoo; Ha, Seokhyeon; Lee, Jungwoo
DE-DARTS: Neural architecture search with dynamic exploration Journal Article
In: ICT Express, 2022, ISSN: 2405-9595.
@article{MUN2022,
title = {DE-DARTS: Neural architecture search with dynamic exploration},
author = {Jiwoo Mun and Seokhyeon Ha and Jungwoo Lee},
url = {https://www.sciencedirect.com/science/article/pii/S2405959522000613},
doi = {https://doi.org/10.1016/j.icte.2022.04.005},
issn = {2405-9595},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {ICT Express},
abstract = {Neural architecture search (NAS) methods automatically find optimal neural networks without human assistance. Numerous algorithms for NAS have been studied to find architectures with gradient-based search. Differentiable architecture search (DARTS), one of the key papers of gradient-based search, dramatically reduced search cost, and showed outstanding performance through continuous relaxation and meta-learning based approximation. However, one of the issues with DARTS is that the gradient-based search process is biased due to the nested bi-level optimization structure, and the greedy behavior of the gradient descent. As a result, there is a problem that search spaces are limited to a limited set of architectures. To overcome the bias of the gradient-based search in network architecture search (NAS), we used a dynamic search method. This technique allows gradient-based search to have exploration. In this paper, we present a novel approach, namely, Dynamic-Exploration DARTS (DE-DARTS). For effective exploration, we use dynamic attention networks (DANs) in DE-DARTS, which change model architectures based on input data. As our DANs are activated early in the search, more diverse architectures are considered, depending on input data at the beginning of search. Our algorithm is evaluated in multiple image classification datasets including CIFAR-10, and ImageNet, and shows improved performance.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Neural architecture search (NAS) methods automatically find optimal neural networks without human assistance. Numerous algorithms for NAS have been studied to find architectures with gradient-based search. Differentiable architecture search (DARTS), one of the key papers of gradient-based search, dramatically reduced search cost, and showed outstanding performance through continuous relaxation and meta-learning based approximation. However, one of the issues with DARTS is that the gradient-based search process is biased due to the nested bi-level optimization structure, and the greedy behavior of the gradient descent. As a result, there is a problem that search spaces are limited to a limited set of architectures. To overcome the bias of the gradient-based search in network architecture search (NAS), we used a dynamic search method. This technique allows gradient-based search to have exploration. In this paper, we present a novel approach, namely, Dynamic-Exploration DARTS (DE-DARTS). For effective exploration, we use dynamic attention networks (DANs) in DE-DARTS, which change model architectures based on input data. As our DANs are activated early in the search, more diverse architectures are considered, depending on input data at the beginning of search. Our algorithm is evaluated in multiple image classification datasets including CIFAR-10, and ImageNet, and shows improved performance.
2047.
Ingolfsson, Thorir Mar; Vero, Mark; Wang, Xiaying; Lamberti, Lorenzo; Benini, Luca; Spallanzani, Matteo
Reducing neural architecture search spaces with training-free statistics and computational graph clustering Proceedings Article
In: Proceedings of the 19th ACM International Conference on Computing Frontiers, pp. 213–214, 2022.
@inproceedings{ingolfsson2022reducing,
title = {Reducing neural architecture search spaces with training-free statistics and computational graph clustering},
author = {Thorir Mar Ingolfsson and Mark Vero and Xiaying Wang and Lorenzo Lamberti and Luca Benini and Matteo Spallanzani},
url = {https://arxiv.org/pdf/2204.14103.pdf},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
booktitle = {Proceedings of the 19th ACM International Conference on Computing Frontiers},
pages = {213--214},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
2046.
Le, Cat P.; Soltani, Mohammadreza; Dong, Juncheng; Tarokh, Vahid
Fisher Task Distance and its Application in Neural Architecture Search Journal Article
In: IEEE Access, vol. 10, pp. 47235-47249, 2022.
@article{9766163,
title = {Fisher Task Distance and its Application in Neural Architecture Search},
author = {Cat P. Le and Mohammadreza Soltani and Juncheng Dong and Vahid Tarokh},
url = {https://ieeexplore.ieee.org/abstract/document/9766163},
doi = {10.1109/ACCESS.2022.3171741},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {IEEE Access},
volume = {10},
pages = {47235-47249},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2045.
Akin, Berkin; Gupta, Suyog; Long, Yun; Spiridonov, Anton; Wang, Zhuo; White, Marie; Xu, Hao; Zhou, Ping; Zhou, Yanqi
Searching for Efficient Neural Architectures for On-Device ML on Edge TPUs Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2204-14007,
title = {Searching for Efficient Neural Architectures for On-Device ML on Edge TPUs},
author = {Berkin Akin and Suyog Gupta and Yun Long and Anton Spiridonov and Zhuo Wang and Marie White and Hao Xu and Ping Zhou and Yanqi Zhou},
url = {https://doi.org/10.48550/arXiv.2204.14007},
doi = {10.48550/arXiv.2204.14007},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2204.14007},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2044.
Wang, Linnan; Yu, Chenhan; Salian, Satish; Kierat, Slawomir; Migacz, Szymon; Florea, Alex Fit
Searching the Deployable Convolution Neural Networks for GPUs Proceedings Article
In: CVPR 2022, 2022.
@inproceedings{DBLP:journals/corr/abs-2205-00841,
title = {Searching the Deployable Convolution Neural Networks for GPUs},
author = {Linnan Wang and Chenhan Yu and Satish Salian and Slawomir Kierat and Szymon Migacz and Alex Fit Florea},
url = {https://openaccess.thecvf.com/content/CVPR2022/papers/Wang_Searching_the_Deployable_Convolution_Neural_Networks_for_GPUs_CVPR_2022_paper.pdf},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
booktitle = {CVPR 2022},
journal = {CoRR},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
2043.
Kawa, Sajad Ahmad; Wani, M Arif
Designing Convolution Neural Network Architecture by utilizing the Complexity Model of the Dataset Proceedings Article
In: 2022 9th International Conference on Computing for Sustainable Global Development (INDIACom), pp. 221-225, 2022.
@inproceedings{9763256,
title = {Designing Convolution Neural Network Architecture by utilizing the Complexity Model of the Dataset},
author = {Sajad Ahmad Kawa and M Arif Wani},
url = {https://ieeexplore.ieee.org/abstract/document/9763256},
doi = {10.23919/INDIACom54597.2022.9763256},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
booktitle = {2022 9th International Conference on Computing for Sustainable Global Development (INDIACom)},
pages = {221-225},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
2042.
Dong, Zhen; Zhou, Kaicheng; Li, Guohao; Zhou, Qiang; Guo, Mingfei; Ghanem, Bernard; Keutzer, Kurt; Zhang, Shanghang
UnrealNAS: Can We Search Neural Architectures with Unreal Data? Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2205-02162,
title = {UnrealNAS: Can We Search Neural Architectures with Unreal Data?},
author = {Zhen Dong and Kaicheng Zhou and Guohao Li and Qiang Zhou and Mingfei Guo and Bernard Ghanem and Kurt Keutzer and Shanghang Zhang},
url = {https://doi.org/10.48550/arXiv.2205.02162},
doi = {10.48550/arXiv.2205.02162},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2205.02162},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2041.
Pourchot, Aloïs; Bailly, Kévin; Ducarouge, Alexis; Sigaud, Olivier
An extensive appraisal of weight-sharing on the NAS-Bench-101 benchmark Journal Article
In: Neurocomputing, vol. 498, pp. 28-42, 2022, ISSN: 0925-2312.
@article{POURCHOT202228,
title = {An extensive appraisal of weight-sharing on the NAS-Bench-101 benchmark},
author = {Aloïs Pourchot and Kévin Bailly and Alexis Ducarouge and Olivier Sigaud},
url = {https://www.sciencedirect.com/science/article/pii/S092523122200501X},
doi = {https://doi.org/10.1016/j.neucom.2022.04.108},
issn = {0925-2312},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Neurocomputing},
volume = {498},
pages = {28-42},
abstract = {Weight-sharing (WS) has recently emerged as a paradigm to accelerate the automated search for efficient neural architectures, a process dubbed Neural Architecture Search (NAS). By using and training the same set of weights for the whole search space, WS allows for the quick evaluation of millions of architectures, where classical NAS approaches require lengthy individual trainings. Although very appealing, WS is not without drawbacks and several works have started to question its capabilities on small hand-crafted benchmarks. In this paper, we take advantage of the NAS-Bench-101 dataset to challenge the efficiency of a uniform-sampling based WS variant on several representative search spaces. After reviewing previous studies on WS and highlighting several of their shortcomings, we introduce our own experimental setup, from which we extract several good practices that one should keep in mind when evaluating WS. With our experiments we first establish that, given the correct evaluation procedure, WS is able to produce accuracy scores decently correlated with standalone ones. We then provide evidence that on some search spaces, this WS variant is able to rapidly find better than random architectures, whilst it is equivalent or sometimes even worse than a baseline random search on others, as we find that given the same budget, the probability of superiority of an architecture found using WS over an architecture found through random search can vary between 7% and 78% depending on the search space. We present evidence that the search space itself has an intricate effect on the capabilities of WS and can bias weight-sharing towards certain architectural patterns with no clear accuracy advantage. We conclude that the impact of WS is heavily search-space dependent and difficult to anticipate for a given problem.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Weight-sharing (WS) has recently emerged as a paradigm to accelerate the automated search for efficient neural architectures, a process dubbed Neural Architecture Search (NAS). By using and training the same set of weights for the whole search space, WS allows for the quick evaluation of millions of architectures, where classical NAS approaches require lengthy individual trainings. Although very appealing, WS is not without drawbacks and several works have started to question its capabilities on small hand-crafted benchmarks. In this paper, we take advantage of the NAS-Bench-101 dataset to challenge the efficiency of a uniform-sampling based WS variant on several representative search spaces. After reviewing previous studies on WS and highlighting several of their shortcomings, we introduce our own experimental setup, from which we extract several good practices that one should keep in mind when evaluating WS. With our experiments we first establish that, given the correct evaluation procedure, WS is able to produce accuracy scores decently correlated with standalone ones. We then provide evidence that on some search spaces, this WS variant is able to rapidly find better than random architectures, whilst it is equivalent or sometimes even worse than a baseline random search on others, as we find that given the same budget, the probability of superiority of an architecture found using WS over an architecture found through random search can vary between 7% and 78% depending on the search space. We present evidence that the search space itself has an intricate effect on the capabilities of WS and can bias weight-sharing towards certain architectural patterns with no clear accuracy advantage. We conclude that the impact of WS is heavily search-space dependent and difficult to anticipate for a given problem.
2040.
Jin, Charles; Phothilimthana, Phitchaya Mangpo; Roy, Sudip
(alpha)NAS: Neural Architecture Search using Property Guided Synthesis Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2205-03960,
title = {(alpha)NAS: Neural Architecture Search using Property Guided Synthesis},
author = {Charles Jin and Phitchaya Mangpo Phothilimthana and Sudip Roy},
url = {https://doi.org/10.48550/arXiv.2205.03960},
doi = {10.48550/arXiv.2205.03960},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2205.03960},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2039.
Wang, Ting-Ting; Chu, Shu-Chuan; Hu, Chia-Cheng; Jia, Han-Dong; Pan, Jeng-Shyang
Efficient Network Architecture Search Using Hybrid Optimizer Journal Article
In: Entropy, vol. 24, no. 5, 2022, ISSN: 1099-4300.
@article{e24050656,
title = {Efficient Network Architecture Search Using Hybrid Optimizer},
author = {Ting-Ting Wang and Shu-Chuan Chu and Chia-Cheng Hu and Han-Dong Jia and Jeng-Shyang Pan},
url = {https://www.mdpi.com/1099-4300/24/5/656},
doi = {10.3390/e24050656},
issn = {1099-4300},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Entropy},
volume = {24},
number = {5},
abstract = {Manually designing a convolutional neural network (CNN) is an important deep learning method for solving the problem of image classification. However, most of the existing CNN structure designs consume a significant amount of time and computing resources. Over the years, the demand for neural architecture search (NAS) methods has been on the rise. Therefore, we propose a novel deep architecture generation model based on Aquila optimization (AO) and a genetic algorithm (GA). The main contributions of this paper are as follows: Firstly, a new encoding strategy representing the CNN coding structure is proposed, so that the evolutionary computing algorithm can be combined with CNN. Secondly, a new mechanism for updating location is proposed, which incorporates three typical operators from GA cleverly into the model we have designed so that the model can find the optimal solution in the limited search space. Thirdly, the proposed method can deal with the variable-length CNN structure by adding skip connections. Fourthly, combining traditional CNN layers and residual blocks and introducing a grouping strategy provides greater possibilities for searching for the optimal CNN structure. Additionally, we use two notable datasets, consisting of the MNIST and CIFAR-10 datasets for model evaluation. The experimental results show that our proposed model has good results in terms of search accuracy and time.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Manually designing a convolutional neural network (CNN) is an important deep learning method for solving the problem of image classification. However, most of the existing CNN structure designs consume a significant amount of time and computing resources. Over the years, the demand for neural architecture search (NAS) methods has been on the rise. Therefore, we propose a novel deep architecture generation model based on Aquila optimization (AO) and a genetic algorithm (GA). The main contributions of this paper are as follows: Firstly, a new encoding strategy representing the CNN coding structure is proposed, so that the evolutionary computing algorithm can be combined with CNN. Secondly, a new mechanism for updating location is proposed, which incorporates three typical operators from GA cleverly into the model we have designed so that the model can find the optimal solution in the limited search space. Thirdly, the proposed method can deal with the variable-length CNN structure by adding skip connections. Fourthly, combining traditional CNN layers and residual blocks and introducing a grouping strategy provides greater possibilities for searching for the optimal CNN structure. Additionally, we use two notable datasets, consisting of the MNIST and CIFAR-10 datasets for model evaluation. The experimental results show that our proposed model has good results in terms of search accuracy and time.
2038.
Li, Boyang; Lu, Qing; Jiang, Weiwen; Jung, Taeho; Shi, Yiyu
A collaboration strategy in the mining pool for proof-of-neural-architecture consensus Journal Article
In: Blockchain: Research and Applications, pp. 100089, 2022, ISSN: 2096-7209.
@article{LI2022100089,
title = {A collaboration strategy in the mining pool for proof-of-neural-architecture consensus},
author = {Boyang Li and Qing Lu and Weiwen Jiang and Taeho Jung and Yiyu Shi},
url = {https://www.sciencedirect.com/science/article/pii/S2096720922000306},
doi = {https://doi.org/10.1016/j.bcra.2022.100089},
issn = {2096-7209},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Blockchain: Research and Applications},
pages = {100089},
abstract = {In most popular public accessible cryptocurrency systems, the mining pool plays a key role because mining cryptocurrency with the mining pool turns the non-profitable situation into profitable for individual miners. In many recent novel blockchain consensuses, the deep learning training procedure becomes the task for miners to prove their workload, thus the computation power of miners will not purely be spent on the hash puzzle. In this way, the hardware and energy will support the blockchain service and deep learning training simultaneously. While the incentive of miners is to earn tokens, individual miners are motivated to join mining pools to become more competitive. In this paper, we are the first to demonstrate a mining pool solution for novel consensuses based on deep learning. The mining pool manager partitions the full searching space into subspaces and all miners are scheduled to collaborate on the Neural Architecture Search (NAS) tasks in the assigned subspace. Experiments demonstrate that the performance of this type of mining pool is more competitive than an individual miner. Due to the uncertainty of miners’ behaviors, the mining pool manager checks the standard deviation of the performance of high reward miners and prepares backup miners to ensure completion of the tasks of high reward miners.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
In most popular public accessible cryptocurrency systems, the mining pool plays a key role because mining cryptocurrency with the mining pool turns the non-profitable situation into profitable for individual miners. In many recent novel blockchain consensuses, the deep learning training procedure becomes the task for miners to prove their workload, thus the computation power of miners will not purely be spent on the hash puzzle. In this way, the hardware and energy will support the blockchain service and deep learning training simultaneously. While the incentive of miners is to earn tokens, individual miners are motivated to join mining pools to become more competitive. In this paper, we are the first to demonstrate a mining pool solution for novel consensuses based on deep learning. The mining pool manager partitions the full searching space into subspaces and all miners are scheduled to collaborate on the Neural Architecture Search (NAS) tasks in the assigned subspace. Experiments demonstrate that the performance of this type of mining pool is more competitive than an individual miner. Due to the uncertainty of miners’ behaviors, the mining pool manager checks the standard deviation of the performance of high reward miners and prepares backup miners to ensure completion of the tasks of high reward miners.
2037.
Zheng, Chenyu; Wang, Junjue; Ma, Ailong; Zhong, Yanfei
AutoLC: Search Lightweight and Top-Performing Architecture for Remote Sensing Image Land-Cover Classification Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2205-05369,
title = {AutoLC: Search Lightweight and Top-Performing Architecture for Remote Sensing Image Land-Cover Classification},
author = {Chenyu Zheng and Junjue Wang and Ailong Ma and Yanfei Zhong},
url = {https://doi.org/10.48550/arXiv.2205.05369},
doi = {10.48550/arXiv.2205.05369},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2205.05369},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2036.
Basha, S. H. Shabbeer; Tula, Debapriya; Vinakota, Sravan Kumar; Dubey, Shiv Ram
Target Aware Network Architecture Search and Compression for Efficient Knowledge Transfer Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2205-05967,
title = {Target Aware Network Architecture Search and Compression for Efficient Knowledge Transfer},
author = {S. H. Shabbeer Basha and Debapriya Tula and Sravan Kumar Vinakota and Shiv Ram Dubey},
url = {https://doi.org/10.48550/arXiv.2205.05967},
doi = {10.48550/arXiv.2205.05967},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2205.05967},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2035.
Ren, Yankun; Li, Longfei; Yang, Xinxing; Zhou, Jun
AutoTransformer: Automatic Transformer Architecture Design For Time Series Classification Proceedings Article
In: Advances in Knowledge Discovery and Data Mining: 26th Pacific-Asia Conference, PAKDD 2022, Chengdu, China, May 16–19, 2022, Proceedings, Part I, pp. 143–155, Springer-Verlag, Chengdu, China, 2022, ISBN: 978-3-031-05932-2.
@inproceedings{10.1007/978-3-031-05933-9_12,
title = {AutoTransformer: Automatic Transformer Architecture Design For Time Series Classification},
author = {Yankun Ren and Longfei Li and Xinxing Yang and Jun Zhou},
url = {https://doi.org/10.1007/978-3-031-05933-9_12},
doi = {10.1007/978-3-031-05933-9_12},
isbn = {978-3-031-05932-2},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
booktitle = {Advances in Knowledge Discovery and Data Mining: 26th Pacific-Asia Conference, PAKDD 2022, Chengdu, China, May 16–19, 2022, Proceedings, Part I},
pages = {143–155},
publisher = {Springer-Verlag},
address = {Chengdu, China},
abstract = {Time series classification (TSC) aims to assign labels to time series. Deep learning methods, such as InceptionTime and Transformer, achieve promising performances in TSC. Although deep learning methods do not require manually crafted features, they do require careful manual design of the network structure. The design of architectures heavily relies on researchers’ prior knowledge and experience. Due to the limitations of human’s knowledge, the designed architecture may not be optimal on the dataset of interest. To automate and optimize the architecture design, we propose a data-driven TSC network architecture design method called AutoTransformer. AutoTransformer designs the suitable network architecture automatically depending on the target TSC dataset. Inspired by the overall architecture of Transformer, we first propose a novel search space tailored for TSC. The search space includes a variety of substructures that are capable of extracting global and local features from time series. Then, with the help of neural architecture search (NAS) technique, a suitable network architecture for the target TSC dataset can be found from the search space. Experimental results show that AutoTransformer finds proper architectures on different TSC datasets and outperforms state-of-the-art methods on the UCR archive. Ablation studies verify the effectiveness of the proposed search space.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Time series classification (TSC) aims to assign labels to time series. Deep learning methods, such as InceptionTime and Transformer, achieve promising performances in TSC. Although deep learning methods do not require manually crafted features, they do require careful manual design of the network structure. The design of architectures heavily relies on researchers’ prior knowledge and experience. Due to the limitations of human’s knowledge, the designed architecture may not be optimal on the dataset of interest. To automate and optimize the architecture design, we propose a data-driven TSC network architecture design method called AutoTransformer. AutoTransformer designs the suitable network architecture automatically depending on the target TSC dataset. Inspired by the overall architecture of Transformer, we first propose a novel search space tailored for TSC. The search space includes a variety of substructures that are capable of extracting global and local features from time series. Then, with the help of neural architecture search (NAS) technique, a suitable network architecture for the target TSC dataset can be found from the search space. Experimental results show that AutoTransformer finds proper architectures on different TSC datasets and outperforms state-of-the-art methods on the UCR archive. Ablation studies verify the effectiveness of the proposed search space.
2034.
Deng, Difan; Karl, Florian; Hutter, Frank; Bischl, Bernd; Lindauer, Marius
Efficient Automated Deep Learning for Time Series Forecasting Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2205-05511,
title = {Efficient Automated Deep Learning for Time Series Forecasting},
author = {Difan Deng and Florian Karl and Frank Hutter and Bernd Bischl and Marius Lindauer},
url = {https://doi.org/10.48550/arXiv.2205.05511},
doi = {10.48550/arXiv.2205.05511},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2205.05511},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2033.
Du, Mengge; Chen, Yuntian; Zhang, Dongxiao
AutoKE: An automatic knowledge embedding framework for scientific machine learning Journal Article
In: CoRR, vol. abs/2205.05390, 2022.
@article{DBLP:journals/corr/abs-2205-05390,
title = {AutoKE: An automatic knowledge embedding framework for scientific machine learning},
author = {Mengge Du and Yuntian Chen and Dongxiao Zhang},
url = {https://doi.org/10.48550/arXiv.2205.05390},
doi = {10.48550/arXiv.2205.05390},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2205.05390},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2032.
Kim, Do-Guk; Lee, Heung-Chang
Proxyless Neural Architecture Adaptation for Supervised Learning and Self-Supervised Learning Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2205-07168,
title = {Proxyless Neural Architecture Adaptation for Supervised Learning and Self-Supervised Learning},
author = {Do-Guk Kim and Heung-Chang Lee},
url = {https://doi.org/10.48550/arXiv.2205.07168},
doi = {10.48550/arXiv.2205.07168},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2205.07168},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2031.
Sun, Jialiang; Zheng, Xiaohu; Yao, Wen; Zhang, Xiaoya; Zhou, Weien
Heat Source Layout Optimization Using Automatic Deep Learning Surrogate Model and Multimodal Neighborhood Search Algorithm Technical Report
2022.
@techreport{DBLP:journals/corr/abs-2205-07812,
title = {Heat Source Layout Optimization Using Automatic Deep Learning Surrogate Model and Multimodal Neighborhood Search Algorithm},
author = {Jialiang Sun and Xiaohu Zheng and Wen Yao and Xiaoya Zhang and Weien Zhou},
url = {https://doi.org/10.48550/arXiv.2205.07812},
doi = {10.48550/arXiv.2205.07812},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {CoRR},
volume = {abs/2205.07812},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
2030.
Belciug, Smaranda
Learning deep neural networks' architectures using differential evolution. Case study: Medical imaging processing Journal Article
In: Computers in Biology and Medicine, vol. 146, pp. 105623, 2022, ISSN: 0010-4825.
@article{BELCIUG2022105623,
title = {Learning deep neural networks' architectures using differential evolution. Case study: Medical imaging processing},
author = {Smaranda Belciug},
url = {https://www.sciencedirect.com/science/article/pii/S0010482522004152},
doi = {https://doi.org/10.1016/j.compbiomed.2022.105623},
issn = {0010-4825},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Computers in Biology and Medicine},
volume = {146},
pages = {105623},
abstract = {The COVID-19 pandemic has changed the way we practice medicine. Cancer patient and obstetric care landscapes have been distorted. Delaying cancer diagnosis or maternal-fetal monitoring increased the number of preventable deaths or pregnancy complications. One solution is using Artificial Intelligence to help the medical personnel establish the diagnosis in a faster and more accurate manner. Deep learning is the state-of-the-art solution for image classification. Researchers manually design the structure of fix deep learning neural networks structures and afterwards verify their performance. The goal of this paper is to propose a potential method for learning deep network architectures automatically. As the number of networks architectures increases exponentially with the number of convolutional layers in the network, we propose a differential evolution algorithm to traverse the search space. At first, we propose a way to encode the network structure as a candidate solution of fixed-length integer array, followed by the initialization of differential evolution method. A set of random individuals is generated, followed by mutation, recombination, and selection. At each generation the individuals with the poorest loss values are eliminated and replaced with more competitive individuals. The model has been tested on three cancer datasets containing MRI scans and histopathological images and two maternal-fetal screening ultrasound images. The novel proposed method has been compared and statistically benchmarked to four state-of-the-art deep learning networks: VGG16, ResNet50, Inception V3, and DenseNet169. The experimental results showed that the model is competitive to other state-of-the-art models, obtaining accuracies between 78.73% and 99.50% depending on the dataset it had been applied on.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
The COVID-19 pandemic has changed the way we practice medicine. Cancer patient and obstetric care landscapes have been distorted. Delaying cancer diagnosis or maternal-fetal monitoring increased the number of preventable deaths or pregnancy complications. One solution is using Artificial Intelligence to help the medical personnel establish the diagnosis in a faster and more accurate manner. Deep learning is the state-of-the-art solution for image classification. Researchers manually design the structure of fix deep learning neural networks structures and afterwards verify their performance. The goal of this paper is to propose a potential method for learning deep network architectures automatically. As the number of networks architectures increases exponentially with the number of convolutional layers in the network, we propose a differential evolution algorithm to traverse the search space. At first, we propose a way to encode the network structure as a candidate solution of fixed-length integer array, followed by the initialization of differential evolution method. A set of random individuals is generated, followed by mutation, recombination, and selection. At each generation the individuals with the poorest loss values are eliminated and replaced with more competitive individuals. The model has been tested on three cancer datasets containing MRI scans and histopathological images and two maternal-fetal screening ultrasound images. The novel proposed method has been compared and statistically benchmarked to four state-of-the-art deep learning networks: VGG16, ResNet50, Inception V3, and DenseNet169. The experimental results showed that the model is competitive to other state-of-the-art models, obtaining accuracies between 78.73% and 99.50% depending on the dataset it had been applied on.
2029.
Frey, Nathan; Soklaski, Ryan; Axelrod, Simon; Samsi, Siddharth; Gomez-Bombarelli, Rafael; Coley, Connor; Gadepally, Vijay
Neural Scaling of Deep Chemical Models Journal Article
In: ChemRxiv, 2022.
@article{frey_soklaski_axelrod_samsi_gomez-bombarelli_coley_gadepally_2022,
title = {Neural Scaling of Deep Chemical Models},
author = {Nathan Frey and Ryan Soklaski and Simon Axelrod and Siddharth Samsi and Rafael Gomez-Bombarelli and Connor Coley and Vijay Gadepally},
doi = {10.26434/chemrxiv-2022-3s512},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {ChemRxiv},
publisher = {Cambridge Open Engage},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
2028.
Miriyala, Srinivas Soumitri; Pujari, Keerthi NagaSree; Naik, Sakshi; Mitra, Kishalay
Evolutionary neural architecture search for surrogate models to enable optimization of industrial continuous crystallization process Journal Article
In: Powder Technology, vol. 405, pp. 117527, 2022, ISSN: 0032-5910.
@article{MIRIYALA2022117527,
title = {Evolutionary neural architecture search for surrogate models to enable optimization of industrial continuous crystallization process},
author = {Srinivas Soumitri Miriyala and Keerthi NagaSree Pujari and Sakshi Naik and Kishalay Mitra},
url = {https://www.sciencedirect.com/science/article/pii/S0032591022004211},
doi = {https://doi.org/10.1016/j.powtec.2022.117527},
issn = {0032-5910},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {Powder Technology},
volume = {405},
pages = {117527},
abstract = {Optimal performance of the crystallization process is of utmost importance for industries handling bulk commodity chemicals to pharmaceuticals. Such an optimization exercise becomes extremely time expensive as the mathematical models mimicking such complex processes involve the solution of Integro-Differential Population Balance Equations using High Resolution Finite Volume Methods. In order to build a fast and robust data based alternative model, a surrogate assisted approach using Artificial Neural Networks has been proposed here. To overcome the heuristics-based estimation of the hyper-parameters in ANNs, we aim to contribute a novel Neural Architecture Search strategy for the auto-tuning of hyper-parameters integrated with sample size determination techniques. While solving a multi-objective optimization of crystallization process ensuring maximum productivity, the results from surrogates are compared with those of a high-fidelity physics driven model, which reports five order of magnitude speed improvement without sacrificing much on accuracy.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Optimal performance of the crystallization process is of utmost importance for industries handling bulk commodity chemicals to pharmaceuticals. Such an optimization exercise becomes extremely time expensive as the mathematical models mimicking such complex processes involve the solution of Integro-Differential Population Balance Equations using High Resolution Finite Volume Methods. In order to build a fast and robust data based alternative model, a surrogate assisted approach using Artificial Neural Networks has been proposed here. To overcome the heuristics-based estimation of the hyper-parameters in ANNs, we aim to contribute a novel Neural Architecture Search strategy for the auto-tuning of hyper-parameters integrated with sample size determination techniques. While solving a multi-objective optimization of crystallization process ensuring maximum productivity, the results from surrogates are compared with those of a high-fidelity physics driven model, which reports five order of magnitude speed improvement without sacrificing much on accuracy.
2027.
Lee, Joo-Hyun; Chang, Joon-Hyuk; Yang, Jae-Mo; Moon, Han-Gil
NAS-TasNet: Neural Architecture Search for Time-Domain Speech Separation Journal Article
In: IEEE Access, vol. 10, pp. 56031-56043, 2022.
@article{9777717,
title = {NAS-TasNet: Neural Architecture Search for Time-Domain Speech Separation},
author = {Joo-Hyun Lee and Joon-Hyuk Chang and Jae-Mo Yang and Han-Gil Moon},
url = {https://ieeexplore.ieee.org/abstract/document/9777717},
doi = {10.1109/ACCESS.2022.3176003},
year = {2022},
date = {2022-01-01},
urldate = {2022-01-01},
journal = {IEEE Access},
volume = {10},
pages = {56031-56043},
keywords = {},
pubstate = {published},
tppubtype = {article}
}