Automatic decoding of brain signals via machine learning can help detect pathologies and enable brain-computer interfaces. As such, it may reduce the workload of doctors, improve medical diagnosis and improve the quality of life of persons with functional impairments.
Movement-related Decoding for Brain-Computer Interfaces
In collaboration with the Neuromedical AI Lab (https://www.tnt.uni-freiburg.de/), our group has pioneered the use of modern deep learning techniques to decode electroencephalographic (EEG) brain signals. In a highly impactful publication, we made the first comprehensive study comparing modern deep learning architectures and training pipelines to more traditional feature-based brain-signal decoding algorithms for decoding movement-related information from EEG (https://onlinelibrary.wiley.com/doi/full/10.1002/hbm.23730), a decoding task highly relevant for brain-computer interfaces. The results showed our deep learning architectures to perform at least as good as well-established feature-based baselines. Subsequent work showed our deep network architectures generalize to a wide variety of brain-signal-decoding tasks, including automatic pathology diagnosis.
Filter Bank Common Spatial Patterns (FBCSP) versus ConvNet decoding accuracies on movement-related decoding. Each small marker represents the accuracy of one subject, the large square markers represent average accuracies across all subjects of both datasets. Markers above the dashed line indicate experiments where ConvNets performed better than FBCSP and opposite for markers below the dashed line. Stars indicate statistically significant differences between FBCSP and ConvNets (Wilcoxon signed-rank test, P < 0.05: *, P < 0.01: **, P < 0.001=***). Bottom left of every plot: linear correlation coefficient between FBCSP and ConvNet decoding accuracies. Mean accuracies were very similar for ConvNets and FBCSP, the (small) statistically significant differences were in direction of the ConvNets. Figure from https://onlinelibrary.wiley.com/doi/full/10.1002/hbm.23730.
Automated Diagnosis
For automatic diagnosis, we performed two studies applying deep learning methods as well as self-developed feature-based approaches to decoding pathology from EEG (https://ieeexplore.ieee.org/document/8257015, https://www.sciencedirect.com/science/article/pii/S1053811920305073). These approaches reached similar pathology decoding accuracy between 81-86%, already close to the typical interrater agreement between neurological doctors. We hope such automated decoding approaches may have clinical utility especially where clinical EEG experts are rare. Our results also showed that as little as 1 minute of training data per recording can already yield good accuracy and the use of contextual information (like the age of the patient or whether the patient was asleep or awake) may improve accuracy further. Further, we showed that automated architecture search methods can find surprising and simple architectures with reasonable pathology decoding accuracy.
Decoding accuracies of all models on pathology decoding during cross-validation and in final evaluation. TCN implemented in Braindecode (BD-TCN) shows the decoding performance. Decoding based on RG achieved accuracy similar to BD-TCN. BD-Deep4 and BD-Shallow ConvNets, RF, and ASC were on the same level, whereas BD-EEGNet achieved marginally lower decoding accuracy. SVM showed the worst performance. Figure from https://www.sciencedirect.com/science/article/pii/S1053811920305073#fig6.
Interpretability
Another focus of our work has been analyzing the trained models to understand what information they extract from the EEG brain signals. We could show that the convolutional networks we trained to decode EEG extract both well-known physiological features as well as more complex features than traditional feature-based algorithms use.
Input-perturbation network-prediction correlation maps for the deep ConvNet for movement-related decoding. Correlation of class predictions and amplitude changes. Averaged over all subjects of the High-Gamma Dataset. Colormaps are scaled per scalp plot. Plausible scalp maps for all frequency bands, for example, contralateral positive correlations for the hand classes in the gamma band Figure from https://onlinelibrary.wiley.com/doi/full/10.1002/hbm.23730.
Braindecode
Finally, our work has also led to the creation of the deep-learning EEG-decoding toolbox braindecode, which is now used and developed by multiple research groups around the world.
