Maintained by Difan Deng and Marius Lindauer.
The following list considers papers related to neural architecture search. It is by no means complete. If you miss a paper on the list, please let us know.
Please note that although NAS methods steadily improve, the quality of empirical evaluations in this field are still lagging behind compared to other areas in machine learning, AI and optimization. We would therefore like to share some best practices for empirical evaluations of NAS methods, which we believe will facilitate sustained and measurable progress in the field. If you are interested in a teaser, please read our blog post or directly jump to our checklist.
Transformers have gained increasing popularity in different domains. For a comprehensive list of papers focusing on Neural Architecture Search for Transformer-Based spaces, the awesome-transformer-search repo is all you need.
2021
1325.
Barakbayeva, Togzhan; Demirci, Fatih M
Fully automatic CNN design with inception blocks Proceedings Article
In: Jiang, Xudong; Fujita, Hiroshi (Ed.): Thirteenth International Conference on Digital Image Processing (ICDIP 2021), pp. 275 – 280, International Society for Optics and Photonics SPIE, 2021.
@inproceedings{,
title = {Fully automatic CNN design with inception blocks},
author = {Togzhan Barakbayeva and Fatih M Demirci},
editor = {Xudong Jiang and Hiroshi Fujita},
url = {https://doi.org/10.1117/12.2601117},
year = {2021},
date = {2021-01-01},
booktitle = {Thirteenth International Conference on Digital Image Processing (ICDIP 2021)},
volume = {11878},
pages = {275 -- 280},
publisher = {SPIE},
organization = {International Society for Optics and Photonics},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
1324.
Wang, Dong-Sheng; Zhao, Cui-Ping; Wang, Qi; Ofori, Kwame Dwomoh; Han, Bin; Liu, Ga-Qiong; Wang, Shi; Wang, Hua-Yu; Zhang, Jing
ANAS: Sentence Similarity Calculation Based on Automatic Neural Architecture Search Proceedings Article
In: Shi, Zhongzhi; Chakraborty, Mihir; Kar, Samarjit (Ed.): Intelligence Science III, pp. 185-195, 2021.
@inproceedings{10.1007/978-3-030-74826-5_16b,
title = {ANAS: Sentence Similarity Calculation Based on Automatic Neural Architecture Search},
author = {Wang, Dong-Sheng and Zhao, Cui-Ping and Wang, Qi and Ofori, Kwame Dwomoh and Han, Bin and Liu, Ga-Qiong and Wang, Shi and Wang, Hua-Yu and Zhang, Jing},
editor = {Shi, Zhongzhi and Chakraborty, Mihir and Kar, Samarjit},
url = {https://link.springer.com/chapter/10.1007%2F978-3-030-74826-5_16},
year = {2021},
date = {2021-01-01},
booktitle = {Intelligence Science III},
pages = {185-195},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
1323.
Xu, Jingjing; Zhao, Liang; Lin, Junyang; Gao, Rundong; Sun, Xu; Yang, Hongxia
KNAS: Green Neural Architecture Search Proceedings Article
In: Meila, Marina; Zhang, Tong (Ed.): Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, pp. 11613–11625, PMLR, 2021.
@inproceedings{DBLP:conf/icml/XuZLG0Y21,
title = {KNAS: Green Neural Architecture Search},
author = {Jingjing Xu and Liang Zhao and Junyang Lin and Rundong Gao and Xu Sun and Hongxia Yang},
editor = {Marina Meila and Tong Zhang},
url = {http://proceedings.mlr.press/v139/xu21m.html},
year = {2021},
date = {2021-01-01},
booktitle = {Proceedings of the 38th International Conference on Machine Learning,
ICML 2021, 18-24 July 2021, Virtual Event},
volume = {139},
pages = {11613--11625},
publisher = {PMLR},
series = {Proceedings of Machine Learning Research},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
1322.
Papież, Bartłomiej W; Yaqub, Mohammad; Jiao, Jianbo; Namburete, Ana I L; Noble, Alison J (Ed.)
Improving Generalization of ENAS-Based CNN Models for Breast Lesion Classification from Ultrasound Images Proceedings Article
In: Papież, Bartłomiej W; Yaqub, Mohammad; Jiao, Jianbo; Namburete, Ana I L; Noble, Alison J (Ed.): Medical Image Understanding and Analysis, pp. 438–453, Springer International Publishing, Cham, 2021, ISBN: 978-3-030-80432-9.
@inproceedings{10.1007/978-3-030-80432-9_33,
title = {Improving Generalization of ENAS-Based CNN Models for Breast Lesion Classification from Ultrasound Images},
editor = {Bart{ł}omiej W Papie{ż} and Mohammad Yaqub and Jianbo Jiao and Ana I L Namburete and Alison J Noble},
url = {https://link.springer.com/chapter/10.1007/978-3-030-80432-9_33},
isbn = {978-3-030-80432-9},
year = {2021},
date = {2021-01-01},
booktitle = {Medical Image Understanding and Analysis},
pages = {438--453},
publisher = {Springer International Publishing},
address = {Cham},
abstract = {Neural Architecture Search (NAS) is one of the most recent developments in automating the design process for deep convolutional neural network (DCNN) architectures. NAS and later Efficient NAS (ENAS) based models have been adopted successfully for various applications including ultrasound image classification for breast lesions. Such a data driven approach leads to creation of DCNN models that are more applicable to the data set at hand but with a risk for model overfitting. In this paper, we first investigate the extent of the ENAS model generalization error problem by using different test data sets of ultrasound images of breast lesions. We have observed a significant reduction of overall average accuracy by nearly 10% and even more severe reduction of specificity rate by more than 20%, indicating that model generalization error is a serious issue with ENAS models for breast lesion classification in ultrasound images. To overcome the generalization error, we examined the effectiveness of a range of techniques including reducing model complexity, use of data augmentation, and use of unbalanced training sets. Experimental results show that different methods for the tuned ENAS models achieved different levels of accuracy when they are tested on internal and two external test data sets. The paper demonstrates that ENAS models trained on an unbalanced dataset with more benign cases tend to generalize well on unseen images achieving average accuracies of 85.8%, 82.7%, and 88.1% respectively for the internal and the two external test data sets not only on specificity alone, but also sensitivity. In particular, the generalization of the refined models across internal and external test data is maintained.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Neural Architecture Search (NAS) is one of the most recent developments in automating the design process for deep convolutional neural network (DCNN) architectures. NAS and later Efficient NAS (ENAS) based models have been adopted successfully for various applications including ultrasound image classification for breast lesions. Such a data driven approach leads to creation of DCNN models that are more applicable to the data set at hand but with a risk for model overfitting. In this paper, we first investigate the extent of the ENAS model generalization error problem by using different test data sets of ultrasound images of breast lesions. We have observed a significant reduction of overall average accuracy by nearly 10% and even more severe reduction of specificity rate by more than 20%, indicating that model generalization error is a serious issue with ENAS models for breast lesion classification in ultrasound images. To overcome the generalization error, we examined the effectiveness of a range of techniques including reducing model complexity, use of data augmentation, and use of unbalanced training sets. Experimental results show that different methods for the tuned ENAS models achieved different levels of accuracy when they are tested on internal and two external test data sets. The paper demonstrates that ENAS models trained on an unbalanced dataset with more benign cases tend to generalize well on unseen images achieving average accuracies of 85.8%, 82.7%, and 88.1% respectively for the internal and the two external test data sets not only on specificity alone, but also sensitivity. In particular, the generalization of the refined models across internal and external test data is maintained.
1321.
Chen, Boyu; Li, Peixia; Li, Chuming; Li, Baopu; Bai, Lei; Lin, Chen; Sun, Ming; Yan, Junjie; Ouyang, Wanli
GLiT: Neural Architecture Search for Global and Local Image Transformer Proceedings Article
In: 2021 {IEEE/CVF} International Conference on Computer Vision, {ICCV} 2021, Montreal, QC, Canada, October 10-17, 2021, 2021.
@inproceedings{DBLP:journals/corr/abs-2107-02960,
title = {GLiT: Neural Architecture Search for Global and Local Image Transformer},
author = {Boyu Chen and Peixia Li and Chuming Li and Baopu Li and Lei Bai and Chen Lin and Ming Sun and Junjie Yan and Wanli Ouyang},
url = {https://arxiv.org/abs/2107.02960},
year = {2021},
date = {2021-01-01},
urldate = {2021-01-01},
booktitle = {2021 {IEEE/CVF} International Conference on Computer Vision, {ICCV}
2021, Montreal, QC, Canada, October 10-17, 2021},
journal = {CoRR},
volume = {abs/2107.02960},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
1320.
Elsken, Thomas; Staffler, Benedikt; Zela, Arber; Metzen, Jan Hendrik; Hutter, Frank
Bag of Tricks for Neural Architecture Search Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2107-03719,
title = {Bag of Tricks for Neural Architecture Search},
author = {Thomas Elsken and Benedikt Staffler and Arber Zela and Jan Hendrik Metzen and Frank Hutter},
url = {https://arxiv.org/abs/2107.03719},
year = {2021},
date = {2021-01-01},
journal = {CoRR},
volume = {abs/2107.03719},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1319.
Lomurno, Eugenio; Samele, Stefano; Matteucci, Matteo; Ardagna, Danilo
Pareto-Optimal Progressive Neural Architecture Search Proceedings Article
In: Proceedings of the Genetic and Evolutionary Computation Conference Companion, pp. 1726–1734, Association for Computing Machinery, Lille, France, 2021, ISBN: 9781450383516.
@inproceedings{10.1145/3449726.3463146,
title = {Pareto-Optimal Progressive Neural Architecture Search},
author = {Eugenio Lomurno and Stefano Samele and Matteo Matteucci and Danilo Ardagna},
url = {https://doi.org/10.1145/3449726.3463146},
doi = {10.1145/3449726.3463146},
isbn = {9781450383516},
year = {2021},
date = {2021-01-01},
booktitle = {Proceedings of the Genetic and Evolutionary Computation Conference Companion},
pages = {1726–1734},
publisher = {Association for Computing Machinery},
address = {Lille, France},
series = {GECCO '21},
abstract = {Neural Architecture Search (NAS) is the process of automating architecture engineering,
searching for the best deep learning configuration. One of the main NAS approaches
proposed in the literature, Progressive Neural Architecture Search (PNAS), seeks for
the architectures with a sequential model-based optimization strategy: it defines
a common recursive structure to generate the networks, whose number of building blocks
rises through iterations. However, NAS algorithms are generally designed for an ideal
setting without considering the needs and the technical constraints imposed by practical
applications. In this paper, we propose a new architecture search named Pareto-Optimal
Progressive Neural Architecture Search (POPNAS) that combines the benefits of PNAS
to a time-accuracy Pareto optimization problem. POPNAS adds a new time predictor to
the existing approach to carry out a joint prediction of time and accuracy for each
candidate neural network, searching through the Pareto front. This allows us to reach
a trade-off between accuracy and training time, identifying neural network architectures
with competitive accuracy in the face of a drastically reduced training time.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Neural Architecture Search (NAS) is the process of automating architecture engineering,
searching for the best deep learning configuration. One of the main NAS approaches
proposed in the literature, Progressive Neural Architecture Search (PNAS), seeks for
the architectures with a sequential model-based optimization strategy: it defines
a common recursive structure to generate the networks, whose number of building blocks
rises through iterations. However, NAS algorithms are generally designed for an ideal
setting without considering the needs and the technical constraints imposed by practical
applications. In this paper, we propose a new architecture search named Pareto-Optimal
Progressive Neural Architecture Search (POPNAS) that combines the benefits of PNAS
to a time-accuracy Pareto optimization problem. POPNAS adds a new time predictor to
the existing approach to carry out a joint prediction of time and accuracy for each
candidate neural network, searching through the Pareto front. This allows us to reach
a trade-off between accuracy and training time, identifying neural network architectures
with competitive accuracy in the face of a drastically reduced training time.
searching for the best deep learning configuration. One of the main NAS approaches
proposed in the literature, Progressive Neural Architecture Search (PNAS), seeks for
the architectures with a sequential model-based optimization strategy: it defines
a common recursive structure to generate the networks, whose number of building blocks
rises through iterations. However, NAS algorithms are generally designed for an ideal
setting without considering the needs and the technical constraints imposed by practical
applications. In this paper, we propose a new architecture search named Pareto-Optimal
Progressive Neural Architecture Search (POPNAS) that combines the benefits of PNAS
to a time-accuracy Pareto optimization problem. POPNAS adds a new time predictor to
the existing approach to carry out a joint prediction of time and accuracy for each
candidate neural network, searching through the Pareto front. This allows us to reach
a trade-off between accuracy and training time, identifying neural network architectures
with competitive accuracy in the face of a drastically reduced training time.
1318.
-, Seyed Mojtaba Marvasti; Khaghani, Javad; Cheng, Li; -, Hossein Ghanei; Kasaei, Shohreh
CHASE: Robust Visual Tracking via Cell-Level Differentiable Neural Architecture Search Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2107-03463,
title = {CHASE: Robust Visual Tracking via Cell-Level Differentiable Neural Architecture Search},
author = {Seyed Mojtaba Marvasti - and Javad Khaghani and Li Cheng and Hossein Ghanei - and Shohreh Kasaei},
url = {https://arxiv.org/abs/2107.03463},
year = {2021},
date = {2021-01-01},
journal = {CoRR},
volume = {abs/2107.03463},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1317.
Li, Jianquan; Liu, Xiaokang; Zhang, Sheng; Yang, Min; Xu, Ruifeng; Qin, Fengqing
Accelerating Neural Architecture Search for Natural Language Processing with Knowledge Distillation and Earth Mover's Distance Proceedings Article
In: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2091–2095, Association for Computing Machinery, Virtual Event, Canada, 2021, ISBN: 9781450380379.
@inproceedings{10.1145/3404835.3463017,
title = {Accelerating Neural Architecture Search for Natural Language Processing with Knowledge Distillation and Earth Mover's Distance},
author = {Jianquan Li and Xiaokang Liu and Sheng Zhang and Min Yang and Ruifeng Xu and Fengqing Qin},
url = {https://doi.org/10.1145/3404835.3463017},
doi = {10.1145/3404835.3463017},
isbn = {9781450380379},
year = {2021},
date = {2021-01-01},
booktitle = {Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval},
pages = {2091–2095},
publisher = {Association for Computing Machinery},
address = {Virtual Event, Canada},
series = {SIGIR '21},
abstract = {Recent AI research has witnessed increasing interests in automatically designing the
architecture of deep neural networks, which is coined as neural architecture search
(NAS). The automatically searched network architectures via NAS methods have outperformed
manually designed architectures on some NLP tasks. However, training a large number
of model configurations for efficient NAS is computationally expensive, creating a
substantial barrier for applying NAS methods in real-life applications. In this paper,
we propose to accelerate neural architecture search for natural language processing
based on knowledge distillation (called KD-NAS). Specifically, instead of searching
the optimal network architecture on the validation set conditioned on the optimal
network weights on the training set, we learn the optimal network by minimizing the
knowledge loss transferred from a pre-trained teacher network to the searching network
based on Earth Mover's Distance (EMD). Experiments on five datasets show that our
method achieves promising performance compared to strong competitors on both accuracy
and searching speed. For reproducibility, we submit the code at: https://github.com/lxk00/KD-NAS-EMD.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Recent AI research has witnessed increasing interests in automatically designing the
architecture of deep neural networks, which is coined as neural architecture search
(NAS). The automatically searched network architectures via NAS methods have outperformed
manually designed architectures on some NLP tasks. However, training a large number
of model configurations for efficient NAS is computationally expensive, creating a
substantial barrier for applying NAS methods in real-life applications. In this paper,
we propose to accelerate neural architecture search for natural language processing
based on knowledge distillation (called KD-NAS). Specifically, instead of searching
the optimal network architecture on the validation set conditioned on the optimal
network weights on the training set, we learn the optimal network by minimizing the
knowledge loss transferred from a pre-trained teacher network to the searching network
based on Earth Mover's Distance (EMD). Experiments on five datasets show that our
method achieves promising performance compared to strong competitors on both accuracy
and searching speed. For reproducibility, we submit the code at: https://github.com/lxk00/KD-NAS-EMD.
architecture of deep neural networks, which is coined as neural architecture search
(NAS). The automatically searched network architectures via NAS methods have outperformed
manually designed architectures on some NLP tasks. However, training a large number
of model configurations for efficient NAS is computationally expensive, creating a
substantial barrier for applying NAS methods in real-life applications. In this paper,
we propose to accelerate neural architecture search for natural language processing
based on knowledge distillation (called KD-NAS). Specifically, instead of searching
the optimal network architecture on the validation set conditioned on the optimal
network weights on the training set, we learn the optimal network by minimizing the
knowledge loss transferred from a pre-trained teacher network to the searching network
based on Earth Mover's Distance (EMD). Experiments on five datasets show that our
method achieves promising performance compared to strong competitors on both accuracy
and searching speed. For reproducibility, we submit the code at: https://github.com/lxk00/KD-NAS-EMD.
1316.
Chen, Jiamin; Gao, Jianliang; Chen, Yibo; Oloulade, Moctard Babatounde; Lyu, Tengfei; Li, Zhao
GraphPAS: Parallel Architecture Search for Graph Neural Networks Proceedings Article
In: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2182–2186, Association for Computing Machinery, Virtual Event, Canada, 2021, ISBN: 9781450380379.
@inproceedings{10.1145/3404835.3463007,
title = {GraphPAS: Parallel Architecture Search for Graph Neural Networks},
author = {Jiamin Chen and Jianliang Gao and Yibo Chen and Moctard Babatounde Oloulade and Tengfei Lyu and Zhao Li},
url = {https://doi.org/10.1145/3404835.3463007},
doi = {10.1145/3404835.3463007},
isbn = {9781450380379},
year = {2021},
date = {2021-01-01},
booktitle = {Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval},
pages = {2182–2186},
publisher = {Association for Computing Machinery},
address = {Virtual Event, Canada},
series = {SIGIR '21},
abstract = {Graph neural architecture search has received a lot of attention as Graph Neural Networks
(GNNs) has been successfully applied on the non-Euclidean data recently. However,
exploring all possible GNNs architectures in the huge search space is too time-consuming
or impossible for big graph data. In this paper, we propose a parallel graph architecture
search (GraphPAS) framework for graph neural networks. In GraphPAS, we explore the
search space in parallel by designing a sharing-based evolution learning, which can
improve the search efficiency without losing the accuracy. Additionally, architecture
information entropy is adopted dynamically for mutation selection probability, which
can reduce space exploration. The experimental result shows that GraphPAS outperforms
state-of-art models with efficiency and accuracy simultaneously.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Graph neural architecture search has received a lot of attention as Graph Neural Networks
(GNNs) has been successfully applied on the non-Euclidean data recently. However,
exploring all possible GNNs architectures in the huge search space is too time-consuming
or impossible for big graph data. In this paper, we propose a parallel graph architecture
search (GraphPAS) framework for graph neural networks. In GraphPAS, we explore the
search space in parallel by designing a sharing-based evolution learning, which can
improve the search efficiency without losing the accuracy. Additionally, architecture
information entropy is adopted dynamically for mutation selection probability, which
can reduce space exploration. The experimental result shows that GraphPAS outperforms
state-of-art models with efficiency and accuracy simultaneously.
(GNNs) has been successfully applied on the non-Euclidean data recently. However,
exploring all possible GNNs architectures in the huge search space is too time-consuming
or impossible for big graph data. In this paper, we propose a parallel graph architecture
search (GraphPAS) framework for graph neural networks. In GraphPAS, we explore the
search space in parallel by designing a sharing-based evolution learning, which can
improve the search efficiency without losing the accuracy. Additionally, architecture
information entropy is adopted dynamically for mutation selection probability, which
can reduce space exploration. The experimental result shows that GraphPAS outperforms
state-of-art models with efficiency and accuracy simultaneously.
1315.
Tan, Haoxian; Guo, Sheng; Zhong, Yujie; Huang, Weilin
Mutually-aware Sub-Graphs Differentiable Architecture Search Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2107-04324,
title = {Mutually-aware Sub-Graphs Differentiable Architecture Search},
author = {Haoxian Tan and Sheng Guo and Yujie Zhong and Weilin Huang},
url = {https://arxiv.org/abs/2107.04324},
year = {2021},
date = {2021-01-01},
journal = {CoRR},
volume = {abs/2107.04324},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1314.
Narayanan, Ashwin Raaghav; Zela, Arber; Saikia, Tonmoy; Brox, Thomas; Hutter, Frank
Multi-headed Neural Ensemble Search Proceedings Article
In: -, Hsuan (Ed.): ICML 2021 Workshop on Uncertainty and Robust-ness in Deep Learning, 2021.
@inproceedings{DBLP:journals/corr/abs-2107-04369,
title = {Multi-headed Neural Ensemble Search},
author = {Ashwin Raaghav Narayanan and Arber Zela and Tonmoy Saikia and Thomas Brox and Frank Hutter},
editor = {Hsuan -},
url = {https://arxiv.org/abs/2107.04369},
year = {2021},
date = {2021-01-01},
booktitle = { ICML 2021 Workshop on Uncertainty and Robust-ness in Deep Learning},
journal = {CoRR},
volume = {abs/2107.04369},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
1313.
Li, Victoria R; Zhang, Zijun; Troyanskaya, Olga G
CROTON: an automated and variant-aware deep learning framework for predicting CRISPR/Cas9 editing outcomes Journal Article
In: Bioinformatics, vol. 37, no. Supplement_1, pp. i342-i348, 2021, ISSN: 1367-4803.
@article{10.1093/bioinformatics/btab268,
title = {CROTON: an automated and variant-aware deep learning framework for predicting CRISPR/Cas9 editing outcomes},
author = {Victoria R Li and Zijun Zhang and Olga G Troyanskaya},
url = {https://doi.org/10.1093/bioinformatics/btab268},
doi = {10.1093/bioinformatics/btab268},
issn = {1367-4803},
year = {2021},
date = {2021-01-01},
journal = {Bioinformatics},
volume = {37},
number = {Supplement_1},
pages = {i342-i348},
abstract = {CRISPR/Cas9 is a revolutionary gene-editing technology that has been widely utilized in biology, biotechnology and medicine. CRISPR/Cas9 editing outcomes depend on local DNA sequences at the target site and are thus predictable. However, existing prediction methods are dependent on both feature and model engineering, which restricts their performance to existing knowledge about CRISPR/Cas9 editing.Herein, deep multi-task convolutional neural networks (CNNs) and neural architecture search (NAS) were used to automate both feature and model engineering and create an end-to-end deep-learning framework, CROTON (CRISPR Outcomes Through cONvolutional neural networks). The CROTON model architecture was tuned automatically with NAS on a synthetic large-scale construct-based dataset and then tested on an independent primary T cell genomic editing dataset. CROTON outperformed existing expert-designed models and non-NAS CNNs in predicting 1 base pair insertion and deletion probability as well as deletion and frameshift frequency. Interpretation of CROTON revealed local sequence determinants for diverse editing outcomes. Finally, CROTON was utilized to assess how single nucleotide variants (SNVs) affect the genome editing outcomes of four clinically relevant target genes: the viral receptors ACE2 and CCR5 and the immune checkpoint inhibitors CTLA4 and PDCD1. Large SNV-induced differences in CROTON predictions in these target genes suggest that SNVs should be taken into consideration when designing widely applicable gRNAs.https://github.com/vli31/CROTON.Supplementary data are available at Bioinformatics online.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
CRISPR/Cas9 is a revolutionary gene-editing technology that has been widely utilized in biology, biotechnology and medicine. CRISPR/Cas9 editing outcomes depend on local DNA sequences at the target site and are thus predictable. However, existing prediction methods are dependent on both feature and model engineering, which restricts their performance to existing knowledge about CRISPR/Cas9 editing.Herein, deep multi-task convolutional neural networks (CNNs) and neural architecture search (NAS) were used to automate both feature and model engineering and create an end-to-end deep-learning framework, CROTON (CRISPR Outcomes Through cONvolutional neural networks). The CROTON model architecture was tuned automatically with NAS on a synthetic large-scale construct-based dataset and then tested on an independent primary T cell genomic editing dataset. CROTON outperformed existing expert-designed models and non-NAS CNNs in predicting 1 base pair insertion and deletion probability as well as deletion and frameshift frequency. Interpretation of CROTON revealed local sequence determinants for diverse editing outcomes. Finally, CROTON was utilized to assess how single nucleotide variants (SNVs) affect the genome editing outcomes of four clinically relevant target genes: the viral receptors ACE2 and CCR5 and the immune checkpoint inhibitors CTLA4 and PDCD1. Large SNV-induced differences in CROTON predictions in these target genes suggest that SNVs should be taken into consideration when designing widely applicable gRNAs.https://github.com/vli31/CROTON.Supplementary data are available at Bioinformatics online.
1312.
Wen, Yu-Wei; Peng, Sheng-Hsuan; Ting, Chuan-Kang
Two-Stage Evolutionary Neural Architecture Search for Transfer Learning Journal Article
In: IEEE Transactions on Evolutionary Computation, pp. 1-1, 2021.
@article{9488292,
title = {Two-Stage Evolutionary Neural Architecture Search for Transfer Learning},
author = {Yu-Wei Wen and Sheng-Hsuan Peng and Chuan-Kang Ting},
url = {https://ieeexplore.ieee.org/abstract/document/9488292},
doi = {10.1109/TEVC.2021.3097937},
year = {2021},
date = {2021-01-01},
journal = {IEEE Transactions on Evolutionary Computation},
pages = {1-1},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
1311.
-hun Shim, Jae; Kong, Kyeongbo; Kang, Suk -Ju
Core-set Sampling for Efficient Neural Architecture Search Proceedings Article
In: ICML 2021 Workshop on Subset Selection in ML, 2021.
@inproceedings{DBLP:journals/corr/abs-2107-06869,
title = {Core-set Sampling for Efficient Neural Architecture Search},
author = {Jae -hun Shim and Kyeongbo Kong and Suk -Ju Kang},
url = {https://arxiv.org/abs/2107.06869},
year = {2021},
date = {2021-01-01},
booktitle = { ICML 2021 Workshop on Subset Selection in ML},
journal = {CoRR},
volume = {abs/2107.06869},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
1310.
Sinha, Nilotpal; -, Kuan
Neural Architecture Search using Covariance Matrix Adaptation Evolution Strategy Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2107-07266,
title = {Neural Architecture Search using Covariance Matrix Adaptation Evolution Strategy},
author = {Nilotpal Sinha and Kuan -},
url = {https://arxiv.org/abs/2107.07266},
year = {2021},
date = {2021-01-01},
journal = {CoRR},
volume = {abs/2107.07266},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1309.
Chen, Lei; Yuan, Fajie; Yang, Jiaxi; Yang, Min; Li, Chengming
Scene-adaptive Knowledge Distillation for Sequential Recommendation via Differentiable Architecture Search Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2107-07173,
title = {Scene-adaptive Knowledge Distillation for Sequential Recommendation via Differentiable Architecture Search},
author = {Lei Chen and Fajie Yuan and Jiaxi Yang and Min Yang and Chengming Li},
url = {https://arxiv.org/abs/2107.07173},
year = {2021},
date = {2021-01-01},
journal = {CoRR},
volume = {abs/2107.07173},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1308.
Liu, Zhu; Ma, Long; Liu, Risheng; Fan, Xin
Learning to Discover a Unified Architecture for Low-Level Vision Journal Article
In: IEEE Signal Processing Letters, vol. 28, pp. 1470-1474, 2021.
@article{9483692,
title = {Learning to Discover a Unified Architecture for Low-Level Vision},
author = {Zhu Liu and Long Ma and Risheng Liu and Xin Fan},
url = {https://ieeexplore.ieee.org/abstract/document/9483692},
doi = {10.1109/LSP.2021.3096456},
year = {2021},
date = {2021-01-01},
journal = {IEEE Signal Processing Letters},
volume = {28},
pages = {1470-1474},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
1307.
Gao, Jiahui; Xu, Hang; Shi, Han; Ren, Xiaozhe; Yu, Philip L H; Liang, Xiaodan; Jiang, Xin; Li, Zhenguo
AutoBERT-Zero: Evolving BERT Backbone from Scratch Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2107-07445,
title = {AutoBERT-Zero: Evolving BERT Backbone from Scratch},
author = {Jiahui Gao and Hang Xu and Han Shi and Xiaozhe Ren and Philip L H Yu and Xiaodan Liang and Xin Jiang and Zhenguo Li},
url = {https://arxiv.org/abs/2107.07445},
year = {2021},
date = {2021-01-01},
journal = {CoRR},
volume = {abs/2107.07445},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1306.
Chrostoforidis, Aristeidis; Kyriakides, George; Margaritis, Konstantinos G
A Novel Evolutionary Algorithm for Hierarchical Neural Architecture Search Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2107-08484,
title = {A Novel Evolutionary Algorithm for Hierarchical Neural Architecture Search},
author = {Aristeidis Chrostoforidis and George Kyriakides and Konstantinos G Margaritis},
url = {https://arxiv.org/abs/2107.08484},
year = {2021},
date = {2021-01-01},
journal = {CoRR},
volume = {abs/2107.08484},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1305.
Odema, Mohanad; Rashid, Nafiul; Demirel, Berken Utku; Faruque, Mohammad Abdullah Al
LENS: Layer Distribution Enabled Neural Architecture Search in Edge-Cloud Hierarchies Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2107-09309,
title = {LENS: Layer Distribution Enabled Neural Architecture Search in Edge-Cloud Hierarchies},
author = {Mohanad Odema and Nafiul Rashid and Berken Utku Demirel and Mohammad Abdullah Al Faruque},
url = {https://arxiv.org/abs/2107.09309},
year = {2021},
date = {2021-01-01},
journal = {CoRR},
volume = {abs/2107.09309},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1304.
Roth, Holger R; Yang, Dong; Li, Wenqi; Myronenko, Andriy; Zhu, Wentao; Xu, Ziyue; Wang, Xiaosong; Xu, Daguang
Federated Whole Prostate Segmentation in MRI with Personalized Neural Architectures Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2107-08111,
title = {Federated Whole Prostate Segmentation in MRI with Personalized Neural Architectures},
author = {Holger R Roth and Dong Yang and Wenqi Li and Andriy Myronenko and Wentao Zhu and Ziyue Xu and Xiaosong Wang and Daguang Xu},
url = {https://arxiv.org/abs/2107.08111},
year = {2021},
date = {2021-01-01},
journal = {CoRR},
volume = {abs/2107.08111},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1303.
Louati, Hassen; Bechikh, Slim; Louati, Ali; Aldaej, Abdulaziz; Said, Lamjed Ben
Evolutionary Optimization of Convolutional Neural Network Architecture Design for Thoracic X-Ray Image Classification Proceedings Article
In: Fujita, Hamido; Selamat, Ali; -, Jerry Chun; Ali, Moonis (Ed.): Advances and Trends in Artificial Intelligence. Artificial Intelligence Practices - 34th International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems, IEA/AIE 2021, Kuala Lumpur, Malaysia, July 26-29, 2021, Proceedings, Part I, pp. 121–132, Springer, 2021.
@inproceedings{DBLP:conf/ieaaie/LouatiBLAS21,
title = {Evolutionary Optimization of Convolutional Neural Network Architecture Design for Thoracic X-Ray Image Classification},
author = {Hassen Louati and Slim Bechikh and Ali Louati and Abdulaziz Aldaej and Lamjed Ben Said},
editor = {Hamido Fujita and Ali Selamat and Jerry Chun - and Moonis Ali},
url = {https://doi.org/10.1007/978-3-030-79457-6_11},
doi = {10.1007/978-3-030-79457-6_11},
year = {2021},
date = {2021-01-01},
booktitle = {Advances and Trends in Artificial Intelligence. Artificial Intelligence
Practices - 34th International Conference on Industrial, Engineering
and Other Applications of Applied Intelligent Systems, IEA/AIE 2021,
Kuala Lumpur, Malaysia, July 26-29, 2021, Proceedings, Part I},
volume = {12798},
pages = {121--132},
publisher = {Springer},
series = {Lecture Notes in Computer Science},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
1302.
Thuy, Hang Duong Thi; Minh, Tuan Nguyen; Van, Phi Nguyen; Quoc, Long Tran
Fully Automated Machine Learning Pipeline for Echocardiogram Segmentation Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2107-08440,
title = {Fully Automated Machine Learning Pipeline for Echocardiogram Segmentation},
author = {Hang Duong Thi Thuy and Tuan Nguyen Minh and Phi Nguyen Van and Long Tran Quoc},
url = {https://arxiv.org/abs/2107.08440},
year = {2021},
date = {2021-01-01},
journal = {CoRR},
volume = {abs/2107.08440},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1301.
Zhang, Peng; Tang, Jinsong; Zhong, Heping; Ning, Mingqiang; Liu, Dandan; Wu, Ke
Self-Trained Target Detection of Radar and Sonar Images Using Automatic Deep Learning Journal Article
In: IEEE Transactions on Geoscience and Remote Sensing, pp. 1-14, 2021.
@article{9489360,
title = {Self-Trained Target Detection of Radar and Sonar Images Using Automatic Deep Learning},
author = {Peng Zhang and Jinsong Tang and Heping Zhong and Mingqiang Ning and Dandan Liu and Ke Wu},
url = {https://ieeexplore.ieee.org/abstract/document/9489360},
doi = {10.1109/TGRS.2021.3096011},
year = {2021},
date = {2021-01-01},
journal = {IEEE Transactions on Geoscience and Remote Sensing},
pages = {1-14},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
1300.
Xue, Yu; Wang, Yankang; Liang, Jiayu; Slowik, Adam
A Self-Adaptive Mutation Neural Architecture Search Algorithm Based on Blocks Journal Article
In: IEEE Computational Intelligence Magazine, vol. 16, no. 3, pp. 67-78, 2021.
@article{9492170,
title = {A Self-Adaptive Mutation Neural Architecture Search Algorithm Based on Blocks},
author = {Yu Xue and Yankang Wang and Jiayu Liang and Adam Slowik},
url = {https://ieeexplore.ieee.org/abstract/document/9492170},
doi = {10.1109/MCI.2021.3084435},
year = {2021},
date = {2021-01-01},
journal = {IEEE Computational Intelligence Magazine},
volume = {16},
number = {3},
pages = {67-78},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
1299.
Phan, Quan Minh; Luong, Ngoc Hoang
Enhancing Multi-objective Evolutionary Neural Architecture Search with Surrogate Models and Potential Point-Guided Local Searches Proceedings Article
In: Fujita, Hamido; Selamat, Ali; Lin, Jerry Chun-Wei; Ali, Moonis (Ed.): pp. 460–472, Springer International Publishing, Cham, 2021, ISBN: 978-3-030-79457-6.
@inproceedings{10.1007/978-3-030-79457-6_39,
title = {Enhancing Multi-objective Evolutionary Neural Architecture Search with Surrogate Models and Potential Point-Guided Local Searches},
author = {Quan Minh Phan and Ngoc Hoang Luong},
editor = {Hamido Fujita and Ali Selamat and Jerry Chun-Wei Lin and Moonis Ali},
url = {https://link.springer.com/chapter/10.1007/978-3-030-79457-6_39},
isbn = {978-3-030-79457-6},
year = {2021},
date = {2021-01-01},
pages = {460--472},
publisher = {Springer International Publishing},
address = {Cham},
abstract = {In this paper, we investigate two methods to enhance the efficiency of multi-objective evolutionary algorithms (MOEAs) when solving Neural Architecture Search (NAS) problems. The first method is to use a surrogate model to predict the accuracy of candidate architectures. Only promising architectures with high predicted accuracy values would then be truly trained and evaluated while the ones with low predicted accuracy would be discarded. The second method is to perform local search for potential solutions on the non-dominated front after each MOEA generation. To demonstrate the effectiveness of the proposed methods, we conduct experiments on benchmark datasets of both macro-level (MacroNAS) and micro-level (NAS-Bench-101, NAS-Bench-201) NAS problems. Experimental results exhibit that the proposed methods achieve improvements on the convergence speed of MOEAs toward Pareto-optimal fronts, especially for macro-level NAS problems.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
In this paper, we investigate two methods to enhance the efficiency of multi-objective evolutionary algorithms (MOEAs) when solving Neural Architecture Search (NAS) problems. The first method is to use a surrogate model to predict the accuracy of candidate architectures. Only promising architectures with high predicted accuracy values would then be truly trained and evaluated while the ones with low predicted accuracy would be discarded. The second method is to perform local search for potential solutions on the non-dominated front after each MOEA generation. To demonstrate the effectiveness of the proposed methods, we conduct experiments on benchmark datasets of both macro-level (MacroNAS) and micro-level (NAS-Bench-101, NAS-Bench-201) NAS problems. Experimental results exhibit that the proposed methods achieve improvements on the convergence speed of MOEAs toward Pareto-optimal fronts, especially for macro-level NAS problems.
1298.
Xue, Song; Chen, Hanlin; Xie, Chunyu; Zhang, Baochang; Gong, Xuan; Doermann, David
Fast and Unsupervised Neural Architecture Evolution for Visual Representation Learning Journal Article
In: IEEE Computational Intelligence Magazine, vol. 16, no. 3, pp. 22-32, 2021.
@article{9492168,
title = {Fast and Unsupervised Neural Architecture Evolution for Visual Representation Learning},
author = {Song Xue and Hanlin Chen and Chunyu Xie and Baochang Zhang and Xuan Gong and David Doermann},
url = {https://ieeexplore.ieee.org/abstract/document/9492168},
doi = {10.1109/MCI.2021.3084394},
year = {2021},
date = {2021-01-01},
journal = {IEEE Computational Intelligence Magazine},
volume = {16},
number = {3},
pages = {22-32},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
1297.
Do, Tu; Luong, Ngoc Hoang
Insightful and Practical Multi-objective Convolutional Neural Network Architecture Search with Evolutionary Algorithms Proceedings Article
In: Fujita, Hamido; Selamat, Ali; Lin, Jerry Chun-Wei; Ali, Moonis (Ed.): pp. 473–479, Springer International Publishing, Cham, 2021, ISBN: 978-3-030-79457-6.
@inproceedings{10.1007/978-3-030-79457-6_40,
title = {Insightful and Practical Multi-objective Convolutional Neural Network Architecture Search with Evolutionary Algorithms},
author = {Tu Do and Ngoc Hoang Luong},
editor = {Hamido Fujita and Ali Selamat and Jerry Chun-Wei Lin and Moonis Ali},
url = {https://link.springer.com/chapter/10.1007/978-3-030-79457-6_40},
isbn = {978-3-030-79457-6},
year = {2021},
date = {2021-01-01},
pages = {473--479},
publisher = {Springer International Publishing},
address = {Cham},
abstract = {This paper investigates a comprehensive convolutional neural network (CNN) representation that encodes both layer connections, and computational block attributes for neural architecture search (NAS). We formulate NAS as a bi-objective optimization problem, where two competing objectives, i.e., the validation accuracy and the model complexity, need to be considered simultaneously. We employ the well-known multi-objective evolutionary algorithm (MOEA) nondominated sorting genetic algorithm II (NSGA-II) to perform multi-objective NAS experiments on the CIFAR-10 dataset. Our NAS runs obtain trade-off fronts of architectures of much wider ranges and better quality compared to NAS runs with less comprehensive representations. We also transfer promising architectures to other datasets, i.e., CIFAR-100, Street View House Numbers, and Intel Image Classification, to verify their applicability. Experimental results indicate that the architectures on the trade-off front obtained at the end of our NAS runs can be straightforwardly employed out of the box without any further modification.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
This paper investigates a comprehensive convolutional neural network (CNN) representation that encodes both layer connections, and computational block attributes for neural architecture search (NAS). We formulate NAS as a bi-objective optimization problem, where two competing objectives, i.e., the validation accuracy and the model complexity, need to be considered simultaneously. We employ the well-known multi-objective evolutionary algorithm (MOEA) nondominated sorting genetic algorithm II (NSGA-II) to perform multi-objective NAS experiments on the CIFAR-10 dataset. Our NAS runs obtain trade-off fronts of architectures of much wider ranges and better quality compared to NAS runs with less comprehensive representations. We also transfer promising architectures to other datasets, i.e., CIFAR-100, Street View House Numbers, and Intel Image Classification, to verify their applicability. Experimental results indicate that the architectures on the trade-off front obtained at the end of our NAS runs can be straightforwardly employed out of the box without any further modification.
1296.
Dey, Debadeepta; Shah, Shital; Bubeck, Sebastien
Ranking Architectures by Feature Extraction Capabilities Proceedings Article
In: 8th ICML Workshop on Automated Machine Learning (AutoML), 2021.
@inproceedings{<LineBreak>dey2021ranking,
title = {Ranking Architectures by Feature Extraction Capabilities},
author = {Debadeepta Dey and Shital Shah and Sebastien Bubeck},
url = {https://openreview.net/forum?id=z0IHb2AUhE},
year = {2021},
date = {2021-01-01},
booktitle = {8th ICML Workshop on Automated Machine Learning (AutoML)},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
1295.
Ma, Jingchen; He, Ni; Yoon, Jin H; Ha, Richard; Li, Jiao; Ma, Weimei; Meng, Tiebao; Lu, Lin; Schwartz, Lawrence H; Wu, Yaopan; Ye, Zhaoxiang; Wu, Peihong; Zhao, Binsheng; Xie, Chuanmiao
Distinguishing benign and malignant lesions on contrast-enhanced breast cone-beam CT with deep learning neural architecture search Journal Article
In: European Journal of Radiology, vol. 142, pp. 109878, 2021, ISSN: 0720-048X.
@article{MA2021109878,
title = {Distinguishing benign and malignant lesions on contrast-enhanced breast cone-beam CT with deep learning neural architecture search},
author = {Jingchen Ma and Ni He and Jin H Yoon and Richard Ha and Jiao Li and Weimei Ma and Tiebao Meng and Lin Lu and Lawrence H Schwartz and Yaopan Wu and Zhaoxiang Ye and Peihong Wu and Binsheng Zhao and Chuanmiao Xie},
url = {https://www.sciencedirect.com/science/article/pii/S0720048X21003594},
doi = {https://doi.org/10.1016/j.ejrad.2021.109878},
issn = {0720-048X},
year = {2021},
date = {2021-01-01},
journal = {European Journal of Radiology},
volume = {142},
pages = {109878},
abstract = {Purpose
To utilize a neural architecture search (NAS) approach to develop a convolutional neural network (CNN) method for distinguishing benign and malignant lesions on breast cone-beam CT (BCBCT).
Method
165 patients with 114 malignant and 86 benign lesions were collected by two institutions from May 2012 to August 2014. The NAS method autonomously generated a CNN model using one institution’s dataset for training (patients/lesions: 71/91) and validation (patients/lesions: 20/23). The model was externally tested on another institution’s dataset (patients/lesions: 74/87), and its performance was compared with fine-tuned ResNet-50 models and two breast radiologists who independently read the lesions in the testing dataset without knowing lesion diagnosis.
Results
The lesion diameters (mean ± SD) were 18.8 ± 12.9 mm, 22.7 ± 10.5 mm, and 20.0 ± 11.8 mm in the training, validation, and external testing set, respectively. Compared to the best ResNet-50 model, the NAS-generated CNN model performed three times faster and, in the external testing set, achieved a higher (though not statistically different) AUC, with sensitivity (95% CI) and specificity (95% CI) of 0.727, 80% (66–90%), and 60% (42–75%), respectively. Meanwhile, the performances of the NAS-generated CNN and the two radiologists’ visual ratings were not statistically different.
Conclusions
Our preliminary results demonstrated that a CNN autonomously generated by NAS performed comparably to pre-trained ResNet models and radiologists in predicting malignant breast lesions on contrast-enhanced BCBCT. In comparison to ResNet, which must be designed by an expert, the NAS approach may be used to automatically generate a deep learning architecture for medical image analysis.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Purpose
To utilize a neural architecture search (NAS) approach to develop a convolutional neural network (CNN) method for distinguishing benign and malignant lesions on breast cone-beam CT (BCBCT).
Method
165 patients with 114 malignant and 86 benign lesions were collected by two institutions from May 2012 to August 2014. The NAS method autonomously generated a CNN model using one institution’s dataset for training (patients/lesions: 71/91) and validation (patients/lesions: 20/23). The model was externally tested on another institution’s dataset (patients/lesions: 74/87), and its performance was compared with fine-tuned ResNet-50 models and two breast radiologists who independently read the lesions in the testing dataset without knowing lesion diagnosis.
Results
The lesion diameters (mean ± SD) were 18.8 ± 12.9 mm, 22.7 ± 10.5 mm, and 20.0 ± 11.8 mm in the training, validation, and external testing set, respectively. Compared to the best ResNet-50 model, the NAS-generated CNN model performed three times faster and, in the external testing set, achieved a higher (though not statistically different) AUC, with sensitivity (95% CI) and specificity (95% CI) of 0.727, 80% (66–90%), and 60% (42–75%), respectively. Meanwhile, the performances of the NAS-generated CNN and the two radiologists’ visual ratings were not statistically different.
Conclusions
Our preliminary results demonstrated that a CNN autonomously generated by NAS performed comparably to pre-trained ResNet models and radiologists in predicting malignant breast lesions on contrast-enhanced BCBCT. In comparison to ResNet, which must be designed by an expert, the NAS approach may be used to automatically generate a deep learning architecture for medical image analysis.
To utilize a neural architecture search (NAS) approach to develop a convolutional neural network (CNN) method for distinguishing benign and malignant lesions on breast cone-beam CT (BCBCT).
Method
165 patients with 114 malignant and 86 benign lesions were collected by two institutions from May 2012 to August 2014. The NAS method autonomously generated a CNN model using one institution’s dataset for training (patients/lesions: 71/91) and validation (patients/lesions: 20/23). The model was externally tested on another institution’s dataset (patients/lesions: 74/87), and its performance was compared with fine-tuned ResNet-50 models and two breast radiologists who independently read the lesions in the testing dataset without knowing lesion diagnosis.
Results
The lesion diameters (mean ± SD) were 18.8 ± 12.9 mm, 22.7 ± 10.5 mm, and 20.0 ± 11.8 mm in the training, validation, and external testing set, respectively. Compared to the best ResNet-50 model, the NAS-generated CNN model performed three times faster and, in the external testing set, achieved a higher (though not statistically different) AUC, with sensitivity (95% CI) and specificity (95% CI) of 0.727, 80% (66–90%), and 60% (42–75%), respectively. Meanwhile, the performances of the NAS-generated CNN and the two radiologists’ visual ratings were not statistically different.
Conclusions
Our preliminary results demonstrated that a CNN autonomously generated by NAS performed comparably to pre-trained ResNet models and radiologists in predicting malignant breast lesions on contrast-enhanced BCBCT. In comparison to ResNet, which must be designed by an expert, the NAS approach may be used to automatically generate a deep learning architecture for medical image analysis.
1294.
Luo, Xiangzhong; Liu, Di; Huai, Shuo; Kong, Hao; Chen, Hui; Liu, Weichen
Designing Efficient DNNs via Hardware-Aware Neural Architecture Search and Beyond Journal Article
In: IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, pp. 1-1, 2021.
@article{9496596,
title = {Designing Efficient DNNs via Hardware-Aware Neural Architecture Search and Beyond},
author = {Xiangzhong Luo and Di Liu and Shuo Huai and Hao Kong and Hui Chen and Weichen Liu},
url = {https://ieeexplore.ieee.org/abstract/document/9496596},
doi = {10.1109/TCAD.2021.3100249},
year = {2021},
date = {2021-01-01},
journal = {IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems},
pages = {1-1},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
1293.
Gracheva, Ekaterina
Trainless model performance estimation based on random weights initialisations for neural architecture search Journal Article
In: Array, vol. 12, pp. 100082, 2021, ISSN: 2590-0056.
@article{GRACHEVA2021100082,
title = {Trainless model performance estimation based on random weights initialisations for neural architecture search},
author = {Ekaterina Gracheva},
url = {https://www.sciencedirect.com/science/article/pii/S2590005621000308},
doi = {https://doi.org/10.1016/j.array.2021.100082},
issn = {2590-0056},
year = {2021},
date = {2021-01-01},
journal = {Array},
volume = {12},
pages = {100082},
abstract = {Neural architecture search has become an indispensable part of the deep learning field. Modern methods allow to find one of the best performing architectures, or to build one from scratch, but they typically make decisions based on the trained accuracy information. In the present article we explore instead how the architectural component of a neural network affects its prediction power. We focus on relationships between the trained accuracy of an architecture and its accuracy prior to training, by considering statistics over multiple initialisations. We observe that minimising the coefficient of variation of the untrained accuracy, CVU, consistently leads to better performing architectures. We test the CVU as a neural architecture search scoring metric using the NAS-Bench-201 database of trained neural architectures. The architectures with the lowest CVU value have on average an accuracy of 91.90±2.27, 64.08±5.63 and 38.76±6.62 for CIFAR-10, CIFAR-100 and a downscaled version of ImageNet, respectively. Since these values are statistically above the random baseline, we make a conclusion that a good architecture should be stable against weights initialisations. It takes about 190 s for CIFAR-10 and CIFAR-100 and 133.9 s for ImageNet16-120 to process 100 architectures, on a batch of 256 images, with 100 initialisations.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Neural architecture search has become an indispensable part of the deep learning field. Modern methods allow to find one of the best performing architectures, or to build one from scratch, but they typically make decisions based on the trained accuracy information. In the present article we explore instead how the architectural component of a neural network affects its prediction power. We focus on relationships between the trained accuracy of an architecture and its accuracy prior to training, by considering statistics over multiple initialisations. We observe that minimising the coefficient of variation of the untrained accuracy, CVU, consistently leads to better performing architectures. We test the CVU as a neural architecture search scoring metric using the NAS-Bench-201 database of trained neural architectures. The architectures with the lowest CVU value have on average an accuracy of 91.90±2.27, 64.08±5.63 and 38.76±6.62 for CIFAR-10, CIFAR-100 and a downscaled version of ImageNet, respectively. Since these values are statistically above the random baseline, we make a conclusion that a good architecture should be stable against weights initialisations. It takes about 190 s for CIFAR-10 and CIFAR-100 and 133.9 s for ImageNet16-120 to process 100 architectures, on a batch of 256 images, with 100 initialisations.
1292.
Chakraborty, Biswadeep; Mukhopadhyay, Saibal
textdollar(mu)textdollarDARTS: Model Uncertainty-Aware Differentiable Architecture Search Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2107-11500,
title = {textdollar(mu)textdollarDARTS: Model Uncertainty-Aware Differentiable Architecture Search},
author = {Biswadeep Chakraborty and Saibal Mukhopadhyay},
url = {https://arxiv.org/abs/2107.11500},
year = {2021},
date = {2021-01-01},
journal = {CoRR},
volume = {abs/2107.11500},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1291.
Yin, Yichun; Chen, Cheng; Shang, Lifeng; Jiang, Xin; Chen, Xiao; Liu, Qun
AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models Proceedings Article
In: Zong, Chengqing; Xia, Fei; Li, Wenjie; Navigli, Roberto (Ed.): Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pp. 5146–5157, Association for Computational Linguistics, 2021.
@inproceedings{DBLP:conf/acl/YinCSJCL20,
title = {AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models},
author = {Yichun Yin and Cheng Chen and Lifeng Shang and Xin Jiang and Xiao Chen and Qun Liu},
editor = {Chengqing Zong and Fei Xia and Wenjie Li and Roberto Navigli},
url = {https://doi.org/10.18653/v1/2021.acl-long.400},
doi = {10.18653/v1/2021.acl-long.400},
year = {2021},
date = {2021-01-01},
booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational
Linguistics and the 11th International Joint Conference on Natural
Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual
Event, August 1-6, 2021},
pages = {5146--5157},
publisher = {Association for Computational Linguistics},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
1290.
Termritthikun, Chakkrit; Jamtsho, Yeshi; Ieamsaard, Jirarat; Muneesawang, Paisarn; Lee, Ivan
EEEA-Net: An Early Exit Evolutionary Neural Architecture Search Journal Article
In: Engineering Applications of Artificial Intelligence, vol. 104, pp. 104397, 2021, ISSN: 0952-1976.
@article{TERMRITTHIKUN2021104397,
title = {EEEA-Net: An Early Exit Evolutionary Neural Architecture Search},
author = {Chakkrit Termritthikun and Yeshi Jamtsho and Jirarat Ieamsaard and Paisarn Muneesawang and Ivan Lee},
url = {https://www.sciencedirect.com/science/article/pii/S0952197621002451},
doi = {https://doi.org/10.1016/j.engappai.2021.104397},
issn = {0952-1976},
year = {2021},
date = {2021-01-01},
journal = {Engineering Applications of Artificial Intelligence},
volume = {104},
pages = {104397},
abstract = {The goals of this research were to search for Convolutional Neural Network (CNN) architectures, suitable for an on-device processor with limited computing resources, performing at substantially lower Network Architecture Search (NAS) costs. A new algorithm entitled an Early Exit Population Initialisation (EE-PI) for Evolutionary Algorithm (EA) was developed to achieve both goals. The EE-PI reduces the total number of parameters in the search process by filtering the models with fewer parameters than the maximum threshold. It will look for a new model to replace those models with parameters more than the threshold. Thereby, reducing the number of parameters, memory usage for model storage and processing time while maintaining the same performance or accuracy. The search time was reduced to 0.52 GPU day. This is a huge and significant achievement compared to the NAS of 4 GPU days achieved using NSGA-Net, 3,150 GPU days by the AmoebaNet model, and the 2,000 GPU days by the NASNet model. As well, Early Exit Evolutionary Algorithm networks (EEEA-Nets) yield network architectures with minimal error and computational cost suitable for a given dataset as a class of network algorithms. Using EEEA-Net on CIFAR-10, CIFAR-100, and ImageNet datasets, our experiments showed that EEEA-Net achieved the lowest error rate among state-of-the-art NAS models, with 2.46% for CIFAR-10, 15.02% for CIFAR-100, and 23.8% for ImageNet dataset. Further, we implemented this image recognition architecture for other tasks, such as object detection, semantic segmentation, and keypoint detection tasks, and, in our experiments, EEEA-Net-C2 outperformed MobileNet-V3 on all of these various tasks. (The algorithm code is available at https://github.com/chakkritte/EEEA-Net).},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
The goals of this research were to search for Convolutional Neural Network (CNN) architectures, suitable for an on-device processor with limited computing resources, performing at substantially lower Network Architecture Search (NAS) costs. A new algorithm entitled an Early Exit Population Initialisation (EE-PI) for Evolutionary Algorithm (EA) was developed to achieve both goals. The EE-PI reduces the total number of parameters in the search process by filtering the models with fewer parameters than the maximum threshold. It will look for a new model to replace those models with parameters more than the threshold. Thereby, reducing the number of parameters, memory usage for model storage and processing time while maintaining the same performance or accuracy. The search time was reduced to 0.52 GPU day. This is a huge and significant achievement compared to the NAS of 4 GPU days achieved using NSGA-Net, 3,150 GPU days by the AmoebaNet model, and the 2,000 GPU days by the NASNet model. As well, Early Exit Evolutionary Algorithm networks (EEEA-Nets) yield network architectures with minimal error and computational cost suitable for a given dataset as a class of network algorithms. Using EEEA-Net on CIFAR-10, CIFAR-100, and ImageNet datasets, our experiments showed that EEEA-Net achieved the lowest error rate among state-of-the-art NAS models, with 2.46% for CIFAR-10, 15.02% for CIFAR-100, and 23.8% for ImageNet dataset. Further, we implemented this image recognition architecture for other tasks, such as object detection, semantic segmentation, and keypoint detection tasks, and, in our experiments, EEEA-Net-C2 outperformed MobileNet-V3 on all of these various tasks. (The algorithm code is available at https://github.com/chakkritte/EEEA-Net).
1289.
Du, Quan; Xu, Nuo; Li, Yinqiao; Xiao, Tong; Zhu, Jingbo
Topology-Sensitive Neural Architecture Search for Language Modeling Journal Article
In: IEEE Access, vol. 9, pp. 107416-107423, 2021.
@article{9502097,
title = {Topology-Sensitive Neural Architecture Search for Language Modeling},
author = {Quan Du and Nuo Xu and Yinqiao Li and Tong Xiao and Jingbo Zhu},
url = {https://ieeexplore.ieee.org/abstract/document/9502097},
doi = {10.1109/ACCESS.2021.3101255},
year = {2021},
date = {2021-01-01},
journal = {IEEE Access},
volume = {9},
pages = {107416-107423},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
1288.
Liu, Yuqiao; Tang, Yehui; Sun, Yanan
Homogeneous Architecture Augmentation for Neural Predictor Journal Article
In: CoRR, vol. abs/2107.13153, 2021.
@article{DBLP:journals/corr/abs-2107-13153,
title = {Homogeneous Architecture Augmentation for Neural Predictor},
author = {Yuqiao Liu and Yehui Tang and Yanan Sun},
url = {https://arxiv.org/abs/2107.13153},
year = {2021},
date = {2021-01-01},
journal = {CoRR},
volume = {abs/2107.13153},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
1287.
Broni-Bediako, Clifford; Murata, Yuki; Mormille, Luiz H B; Atsumi, Masayasu
Searching for CNN Architectures for Remote Sensing Scene Classification Journal Article
In: IEEE Transactions on Geoscience and Remote Sensing, pp. 1-13, 2021.
@article{9497513,
title = {Searching for CNN Architectures for Remote Sensing Scene Classification},
author = {Clifford Broni-Bediako and Yuki Murata and Luiz H B Mormille and Masayasu Atsumi},
doi = {10.1109/TGRS.2021.3097938},
year = {2021},
date = {2021-01-01},
journal = {IEEE Transactions on Geoscience and Remote Sensing},
pages = {1-13},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
1286.
Li, Guihong; Mandal, Sumit K; Ü,; Marculescu, Radu
FLASH: Fast Neural Architecture Search with Hardware Optimization Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2108-00568,
title = {FLASH: Fast Neural Architecture Search with Hardware Optimization},
author = {Guihong Li and Sumit K Mandal and Ü and Radu Marculescu},
url = {https://arxiv.org/abs/2108.00568},
year = {2021},
date = {2021-01-01},
journal = {CoRR},
volume = {abs/2108.00568},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1285.
Mills, Keith G; Salameh, Mohammad; Niu, Di; Han, Fred X; Rezaei, Seyed Saeed Changiz; Yao, Hengshuai; Lu, Wei; Lian, Shuo; Jui, Shangling
Exploring Neural Architecture Search Space via Deep Deterministic Sampling Journal Article
In: IEEE Access, vol. 9, pp. 110962-110974, 2021.
@article{9503404,
title = {Exploring Neural Architecture Search Space via Deep Deterministic Sampling},
author = {Keith G Mills and Mohammad Salameh and Di Niu and Fred X Han and Seyed Saeed Changiz Rezaei and Hengshuai Yao and Wei Lu and Shuo Lian and Shangling Jui},
url = {https://ieeexplore.ieee.org/abstract/document/9503404},
doi = {10.1109/ACCESS.2021.3101975},
year = {2021},
date = {2021-01-01},
journal = {IEEE Access},
volume = {9},
pages = {110962-110974},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
1284.
Xie, Zhenyu; Zhang, Xujie; Zhao, Fuwei; Dong, Haoye; Kampffmeyer, Michael C; Yan, Haonan; Liang, Xiaodan
WAS-VTON: Warping Architecture Search for Virtual Try-on Network Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2108-00386,
title = {WAS-VTON: Warping Architecture Search for Virtual Try-on Network},
author = {Zhenyu Xie and Xujie Zhang and Fuwei Zhao and Haoye Dong and Michael C Kampffmeyer and Haonan Yan and Xiaodan Liang},
url = {https://arxiv.org/abs/2108.00386},
year = {2021},
date = {2021-01-01},
journal = {CoRR},
volume = {abs/2108.00386},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1283.
Liu, Jing; Zhuang, Bohan; Tan, Mingkui; Liu, Xu; Phung, Dinh; Li, Yuanqing; Cai, Jianfei
Elastic Architecture Search for Diverse Tasks with Different Resources Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2108-01224,
title = {Elastic Architecture Search for Diverse Tasks with Different Resources},
author = {Jing Liu and Bohan Zhuang and Mingkui Tan and Xu Liu and Dinh Phung and Yuanqing Li and Jianfei Cai},
url = {https://arxiv.org/abs/2108.01224},
year = {2021},
date = {2021-01-01},
journal = {CoRR},
volume = {abs/2108.01224},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1282.
Mok, Jisoo; Na, Byunggook; Choe, Hyeokjun; Yoon, Sungroh
AdvRush: Searching for Adversarially Robust Neural Architectures Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2108-01289,
title = {AdvRush: Searching for Adversarially Robust Neural Architectures},
author = {Jisoo Mok and Byunggook Na and Hyeokjun Choe and Sungroh Yoon},
url = {https://arxiv.org/abs/2108.01289},
year = {2021},
date = {2021-01-01},
journal = {CoRR},
volume = {abs/2108.01289},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1281.
He, Xiaoyu; Wang, Yong; Wang, Xiaojing; Huang, Weihong; Zhao, Shuang; Chen, Xiang
Simple-Encoded Evolving Convolutional Neural Networks and Its Application to Skin Disease Image Classification Journal Article
In: Swarm and Evolutionary Computation, pp. 100955, 2021, ISSN: 2210-6502.
@article{HE2021100955,
title = {Simple-Encoded Evolving Convolutional Neural Networks and Its Application to Skin Disease Image Classification},
author = {Xiaoyu He and Yong Wang and Xiaojing Wang and Weihong Huang and Shuang Zhao and Xiang Chen},
url = {https://www.sciencedirect.com/science/article/pii/S2210650221001176},
doi = {https://doi.org/10.1016/j.swevo.2021.100955},
issn = {2210-6502},
year = {2021},
date = {2021-01-01},
journal = {Swarm and Evolutionary Computation},
pages = {100955},
abstract = {Automatically searched architectures by neural architecture search (NAS) methods have shown promising performance in various visual recognition tasks. Among NAS methods, evolutionary neural architecture methods are popular because of their potential to find the global optimal convolutional neural networks (CNNs). These methods usually use an individual to represent a CNN, while often facing two challenges: 1) since a CNN has numerous encoded parameters and weights, the length of each individual is long which causes a large search space, and 2) due to the unknown optimal depth of a CNN, it is necessary to deal with a variable-length optimization problem, which leads to the search in a messy way since search spaces with different dimensions may have different optimal solutions. In this paper, we propose a genetic algorithm with a simple encoding scheme (SEECNN) for evolving CNNs to address image classification problems. In our encoding scheme, the parameters and weights of each layer are encoded into an individual and the whole population represents an entire CNN. Over the course of evolution, three offspring subpopulations are separately produced by genetic operators on three subpopulations. In each subpopulation, the lengths of individuals are short and equal. Afterward, we design a stable search strategy to update the population based on the performance improvement, where we only insert, replace, and remove one individual to generate candidate populations. SEECNN is compared with 24 well-known algorithms on nine benchmark datasets and five state-of-the-art methods on a real-world skin disease image classification case. The results demonstrate its effectiveness.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Automatically searched architectures by neural architecture search (NAS) methods have shown promising performance in various visual recognition tasks. Among NAS methods, evolutionary neural architecture methods are popular because of their potential to find the global optimal convolutional neural networks (CNNs). These methods usually use an individual to represent a CNN, while often facing two challenges: 1) since a CNN has numerous encoded parameters and weights, the length of each individual is long which causes a large search space, and 2) due to the unknown optimal depth of a CNN, it is necessary to deal with a variable-length optimization problem, which leads to the search in a messy way since search spaces with different dimensions may have different optimal solutions. In this paper, we propose a genetic algorithm with a simple encoding scheme (SEECNN) for evolving CNNs to address image classification problems. In our encoding scheme, the parameters and weights of each layer are encoded into an individual and the whole population represents an entire CNN. Over the course of evolution, three offspring subpopulations are separately produced by genetic operators on three subpopulations. In each subpopulation, the lengths of individuals are short and equal. Afterward, we design a stable search strategy to update the population based on the performance improvement, where we only insert, replace, and remove one individual to generate candidate populations. SEECNN is compared with 24 well-known algorithms on nine benchmark datasets and five state-of-the-art methods on a real-world skin disease image classification case. The results demonstrate its effectiveness.
1280.
Li, Qing; Zhang, Wei; Zhao, Lin; Wu, Xia; Liu, Tianming
Evolutional Neural Architecture Search for Optimization of Spatiotemporal Brain Network Decomposition Journal Article
In: IEEE Transactions on Biomedical Engineering, pp. 1-1, 2021.
@article{9508863,
title = {Evolutional Neural Architecture Search for Optimization of Spatiotemporal Brain Network Decomposition},
author = {Qing Li and Wei Zhang and Lin Zhao and Xia Wu and Tianming Liu},
url = {https://ieeexplore.ieee.org/abstract/document/9508863},
doi = {10.1109/TBME.2021.3102466},
year = {2021},
date = {2021-01-01},
journal = {IEEE Transactions on Biomedical Engineering},
pages = {1-1},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
1279.
Odema, Mohanad; Rashid, Nafiul; Faruque, Mohammad Abdullah Al
EExNAS: Early-Exit Neural Architecture Search Solutions for Low-Power Wearable Devices Proceedings Article
In: 2021 IEEE/ACM International Symposium on Low Power Electronics and Design (ISLPED), pp. 1-6, 2021.
@inproceedings{9502503,
title = {EExNAS: Early-Exit Neural Architecture Search Solutions for Low-Power Wearable Devices},
author = {Mohanad Odema and Nafiul Rashid and Mohammad Abdullah Al Faruque},
url = {https://ieeexplore.ieee.org/abstract/document/9502503},
doi = {10.1109/ISLPED52811.2021.9502503},
year = {2021},
date = {2021-01-01},
booktitle = {2021 IEEE/ACM International Symposium on Low Power Electronics and Design (ISLPED)},
pages = {1-6},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
1278.
Li, Yuhong; Hao, Cong; Li, Pan; Xiong, Jinjun; Chen, Deming
Generic Neural Architecture Search via Regression Technical Report
2021.
@techreport{DBLP:journals/corr/abs-2108-01899,
title = {Generic Neural Architecture Search via Regression},
author = {Yuhong Li and Cong Hao and Pan Li and Jinjun Xiong and Deming Chen},
url = {https://arxiv.org/abs/2108.01899},
year = {2021},
date = {2021-01-01},
journal = {CoRR},
volume = {abs/2108.01899},
keywords = {},
pubstate = {published},
tppubtype = {techreport}
}
1277.
Zhang, Yongan; Fu, Yonggan; Jiang, Weiwen; Li, Chaojian; You, Haoran; Li, Meng; Chandra, Vikas; Lin, Yingyan
DIAN: Differentiable Accelerator-Network Co-Search Towards Maximal DNN Efficiency Proceedings Article
In: 2021 IEEE/ACM International Symposium on Low Power Electronics and Design (ISLPED), pp. 1-6, 2021.
@inproceedings{9502478,
title = {DIAN: Differentiable Accelerator-Network Co-Search Towards Maximal DNN Efficiency},
author = {Yongan Zhang and Yonggan Fu and Weiwen Jiang and Chaojian Li and Haoran You and Meng Li and Vikas Chandra and Yingyan Lin},
url = {https://ieeexplore.ieee.org/abstract/document/9502478},
doi = {10.1109/ISLPED52811.2021.9502478},
year = {2021},
date = {2021-01-01},
booktitle = {2021 IEEE/ACM International Symposium on Low Power Electronics and Design (ISLPED)},
pages = {1-6},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
1276.
Repin, Denis; Petrov, Tatjana
Automated Deep Abstractions for Stochastic Chemical Reaction Networks Journal Article
In: Information and Computation, pp. 104788, 2021, ISSN: 0890-5401.
@article{REPIN2021104788,
title = {Automated Deep Abstractions for Stochastic Chemical Reaction Networks},
author = {Denis Repin and Tatjana Petrov},
url = {https://www.sciencedirect.com/science/article/pii/S0890540121001048},
doi = {https://doi.org/10.1016/j.ic.2021.104788},
issn = {0890-5401},
year = {2021},
date = {2021-01-01},
journal = {Information and Computation},
pages = {104788},
abstract = {Predicting stochastic cellular dynamics as emerging from the mechanistic models of molecular interactions is a long-standing challenge in systems biology: low-level chemical reaction network (CRN) models give rise to a highly-dimensional continuous-time Markov chain (CTMC) which is computationally demanding and often prohibitive to analyse in practice. A recently proposed abstraction method uses deep learning to replace this CTMC with a discrete-time continuous-space process, by training a mixture density deep neural network with traces sampled at regular time intervals (which can be obtained either by simulating a given CRN or as time-series data from experiment). The major advantage of such abstraction is that it produces a computational model that is dramatically cheaper to execute, while it preserves the statistical features of the training data. In general, the abstraction accuracy improves with the amount of training data. However, depending on the CRN, the overall quality of the method – the efficiency gain and abstraction accuracy – will also depend on the choice of neural network architecture given by hyper-parameters such as the layer types and connections between them. As a consequence, in practice, the modeller has to take care of finding the suitable architecture manually, for each given CRN, through a tedious and time-consuming trial-and-error cycle. In this paper, we propose to further automatise deep abstractions for stochastic CRNs, through learning the neural network architecture along with learning the transition kernel of the abstract process. Automated search of the architecture makes the method applicable directly to any given CRN, which is time-saving for deep learning experts and crucial for non-specialists. We implement the method and demonstrate its performance on a number of representative CRNs with multi-modal emergent phenotypes. Moreover, we showcase that deep abstractions can be used for efficient multi-scale simulations, which are otherwise computationally intractable. To this end, we define a scenario where multiple CRN instances interact across a spatial grid via shared species. Finally, we discuss the limitations and challenges arising when using deep abstractions.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Predicting stochastic cellular dynamics as emerging from the mechanistic models of molecular interactions is a long-standing challenge in systems biology: low-level chemical reaction network (CRN) models give rise to a highly-dimensional continuous-time Markov chain (CTMC) which is computationally demanding and often prohibitive to analyse in practice. A recently proposed abstraction method uses deep learning to replace this CTMC with a discrete-time continuous-space process, by training a mixture density deep neural network with traces sampled at regular time intervals (which can be obtained either by simulating a given CRN or as time-series data from experiment). The major advantage of such abstraction is that it produces a computational model that is dramatically cheaper to execute, while it preserves the statistical features of the training data. In general, the abstraction accuracy improves with the amount of training data. However, depending on the CRN, the overall quality of the method – the efficiency gain and abstraction accuracy – will also depend on the choice of neural network architecture given by hyper-parameters such as the layer types and connections between them. As a consequence, in practice, the modeller has to take care of finding the suitable architecture manually, for each given CRN, through a tedious and time-consuming trial-and-error cycle. In this paper, we propose to further automatise deep abstractions for stochastic CRNs, through learning the neural network architecture along with learning the transition kernel of the abstract process. Automated search of the architecture makes the method applicable directly to any given CRN, which is time-saving for deep learning experts and crucial for non-specialists. We implement the method and demonstrate its performance on a number of representative CRNs with multi-modal emergent phenotypes. Moreover, we showcase that deep abstractions can be used for efficient multi-scale simulations, which are otherwise computationally intractable. To this end, we define a scenario where multiple CRN instances interact across a spatial grid via shared species. Finally, we discuss the limitations and challenges arising when using deep abstractions.