综述 深度学习在神经成像领域的前景与挑战
目录
- 目录
- 词汇与新坑
- introduction
- DL for neuroimaging classification and regression
- Multilayer perceptron models
- Convolutional neural networks and graph convolutional networks
- Recurrent neural network
- GANs
- Attention Modules
- Promises and challenges
- DL for analysis of dynamic activity and connectivity
- modeling spatiotemporal dynamics using DL models
- The combination of DL with conventional neuroimaging tools
- Promises and challeges
- DL for multimodal fusion
- DL frameworks for multimodal fusion
- Multimodal fusion applications in neuroimaging
- Promises and challenges
- Visualization and subytype discovery
- Network visualization for biomarker discovery
- Spectrum and subtype discovery using DL framework
词汇与新坑
- FNC：functional network connectivity
- Deep neural network with weight sparsity control and pre-training extracts hierarchical features and enhances classification performance: Evidence from whole-brain resting-state functional connectivity patterns of schizophrenia
- ICA: independent component analysis
- A novel 5D brain parcellation approach based on spatio-temporal encoding or resting fMRI data from deep residual learning 2021
- Spatio-temporal dynamics of intrinsic networks in functional magnetic imaging data using recurrent neural netowrks
- dFNC:dynamic FNC
- Deep chronnectome learning via full bidirectional long short-term memory networks fro MCI diagnosis 2018
introduction
深度学习（DL）在应用于自然图像分析时非常成功。相比之下，分析神经成像数据提出了一些独特的挑战，包括：
- 更高的维度higher dimensionality
- 更小的样本量smaller sample sizes
- 多种异质模式 multiple heterogeneous modalities
- 有限的标签,limited ground truth。这篇文章中，我们将在神经成像领域的四个不同任务中讨论DL方法：
- 分类/预测classification/prediction
- 动态活动/连接dynamic activity/ connectivity
- 多模态融合 multimodal fusion
- 和解释/可视化 interpretation/visualization。我们强调了这些类别的最新进展，讨论了结合数据特征和模型架构的好处，并推导出了在神经成像数据中使用DL的指导方针。对于每个类别，我们还评估了有前途的应用程序和需要克服的主要挑战。最后，我们讨论了神经成像DL在临床应用中的未来发展方向，这是一个非常有趣的话题。
神经成像是一个强大的工具，正在被用来为健康和紊乱的人类大脑提供重要的见解。它也有潜力将发现和技术进步转化为大脑疾病.
?In contrast to natural images, which are collected under natural light, neuroimaging data consist mostly of raiological images. Because of this, thie noise distribution of neuroimaging varies depending on the acquisition used.
与在自然光下收集的自然图像相比，神经影像学数据主要由图像组成。正因为如此，神经影像学的噪声分布因所使用的采集而异。
?Rician noise in MRI, quantum noise in computed tomography (CT)
MRI中的Rician噪声，计算机断层扫描（CT）中的量子噪声等等。
?As showns in Table 1, neuroimaging data come with many other additional unique aspects, including the number of modalities, high dimensionality, low signal-to-noise ratio, and small sample sizes compared to natural image data.
如表1所示，与自然图像数据相比，神经影像数据具有许多其他独特的方面，包括模态数量、高维数、低信噪比和小样本量。
image.png
MRI作为a noninvasive technique with high spatiotemporal resolution, 是当前研究最广泛的neuroimaging modality.
image.png
?Advanced neuroimaging analysis approaches are essential for linking brain function and structure to network and behavior.
先进的神经影像学分析方法对于将大脑功能和结构与网络和行为联系起来至关重要。
?Linear models and, in particular, flexible matrix decomposition approaches have contributed a lot to our current understanding. For instance, group independent component analysis (ICA), as a purely data-driven algorithm that reveals large-scale networks by making group inferences from funcional MRI (fMRI), is particularly useful for data fusion of multiple modalities, such as genome-wide single-nucleotide polymorphism (SNP) data or event-related potentials.
线性模型，特别是灵活的矩阵分解方法，对我们目前的理解做出了很大贡献。例如，组独立成分分析（ICA）作为一种纯粹的数据驱动算法，通过从功能MRI（fMRI）进行组推断来揭示大规模网络，对于多种模式的数据融合特别有用，例如全基因组单核苷酸多态性（SNP）数据或事件相关的风险。
此外，还有standard machine learning （SML）标准机器学习的方法来分析。SML方法通常需要相当多的领域专业知识来设计特征提取器。这些特征提取器将原始数据转换为合适的内部表示或特征向量。
然后就是深度学习DL的方法。但是复杂的模型容易受到黑盒的影响。
在这篇review综述中，four interrelated topics are covered:
- classification/regression tasks, which are often studied in the context of brain-based biomarker studies, and key DL models.
- DL-based dynamic analysis methods, which are useful for leveraging functional information in neuroimaging data.
- multimodal fusion methods, which are needed to leverage complementary information amoung the modalities
- visualization and subtype discovery, which is crucial for moving to clinical applications and providing clues regarding the underlying biological mechanisms.可视化和亚型发现，这对于转向临床应用和提供有关潜在生物学机制的线索至关重要。
DL for neuroimaging classification and regression
分类和回归是两个被广泛研究的监督学习任务。广义上说，两个任务的目标都是把x（神经成像数据映射到y（诊断，治疗反应和行为）。尽管神经成像数据高度多样化，还是可以分成两大类：structural imaging and functional imaging.
structural neuroimaging data 结构成像，例如structural MRI（sMRI）和diffusion MRI（dMRI弥散MRI），reflect voxel tissue density/volume or structural connectivity.反应了体素组织密度、体积或结构连通性
结构研究的主要目的是为了揭示anatomical relationships解剖关系 in the brain,这也可以用于预测。
functional neuroimaging data关注于大脑活动或连通性的动态变化。由于MRI等身成像的高纬度、低信噪比、高效的特征处理对于减少建模前的冗余是非常重要的。例如，functional MRI的时间过程通常采用atlas-based or data-driven approaches来降低维度，例如ICA。然后将得到的时间签名用于研究时间依赖性，如functional network connectivity （FNC）或者dynamic FNC。在这里我们总结了流行的DL模型的基本机制，并就其相应的神经成像提出了建议
Multilayer perceptron models
通过简单的梯度下降训练的多层感知器MLP是第一个提出可训练的多层特征呢工程的解决方案【3】.
image.png
MLP最适合低纬和较少冗余的输入，如FNC向量。
Convolutional neural networks and graph convolutional networks
?Desprite the great success of CNNs, the non-Euclidean characteristic of graph features such as those obtained from FNC makes the general convolutionand no as well defined as on natural images.
?Similarly, a graph convolutional network GCN is a type of neural network architecture that can capture the graph structure and aggregate node information from the neighborhoods in a convolutional fashion with fewer learnable parameters. GCNs are useful in medical or biochemical applications with graph data such as FNC。
Recurrent neural network
?Compared to classical linear machine learning models, such as a hidden Markov model, an RNN models the long-term nonlinear mechanisms of the sequential data.
GANs
就是生成模型。
Attention Modules
?The use of an attention module was proposed to increase the representation power and improve interpretability by focusing on important brain regions and suppressing unnecessary ones, which is often combined with other DL models for interpretation, allowing the model to dynamically emphasize certain parts of input.
简单的说，就是可解释性。
Promises and challenges
?DL models designed for 3D and 4D neuroimaging data often consist of millions of parameters that require many samples for optimization.
DL for analysis of dynamic activity and connectivity
?The characterization of brain activity and connectivity dynamics (e.g., the chronnectome) is crucial for out understanding of brain function. However, uncovering relevant transient patterns in brain function is challenging because of the lack of computational tools that can effectively capture nonlinear dynamics from high-dimensional data.
大脑活动和连接动力学（例如，connectome）的特征对于理解大脑功能至关重要。然而，揭示大脑功能中的相关瞬态模式具有挑战性，因为缺乏能够有效地从高维数据中捕获非线性动力学的计算工具。
?Recent studies show that DL models, especially RNN-based networks, have the potential to capture whole-brain dynamic information and utilize the time-varing functional connectivity state profiles to expand our understanding of brain function and disorder.
最近的研究表明，深度学习模型，特别是基于RNN的网络，有可能捕获全脑动态信息，并利用时间变化的功能连接状态图来扩展我们对大脑功能和紊乱的理解。
image.png
modeling spatiotemporal dynamics using DL models
传统的神经成像分类方法以functional networks connectivity or spatial maps作为输入特征，忽略了时间动态信息。DL模型具有良好的特征表示学习能力，为直接捕捉时空信息提供了一个潜在的工具。
特别是RNN在序列建模方面取得了巨大的成功，目前广泛应用于brain disorder diagnosis, brain decoding and temporally dynamic functional state translation detection. dFNC是一种从功能核磁共振成像数据中识别time-varing patterns的方法.
后面是对rnn，biLSTM啥的模型，相关文献：
image.png
The combination of DL with conventional neuroimaging tools
为了方便发现神经丞相数据中的动态信息，DL可以与研究充分的数据驱动的机器学习方法混合，如ICA。这也可以提高结果的可解释性。有一篇论文提出了一种新的大脑分割时空网络，该网络将3D的CNN和ICA相结合，使得该框架可以探索5D的大脑动力学。
image.png
image.png
此外RNN-ICA提出了结合RNN和ICA使用方案，which can explicitly optimize linear generative models to model temporal dynamics and infer intrinsic networks from time-series observations (the network structure and identified spatial maps are shown in RNN leverages ICA in figure 3)
image.png
这几篇文章，都可以复现！必须要弄清楚这个ICA和FNC这两个东西，是怎么计算的。构建一个自己的处理图像特征的库。考虑到这个库是提出医学图像特征的，主要针对连接性的，又是医学图像的。叫做connectome感觉不错，如果这个名字已经有了，叫做MidConnectome
Promises and challeges
基于RNN建模挺好的，但是现有工作中，动态特征通常是基于窗口的相关性的，因此窗口大小是一个影响dFNC特征的超参数，具有较短窗口的不能捕捉长时间的相关性，而较长的窗口降低了对快速变化的敏感性。
什么是窗口，在学习dFNC的时候可以了解到。
DL for multimodal fusion
神经成像通常包括多种模式，比如sMRI，fMRI，dMRI，他们为观察和分析大脑提供了多种视图。为了利用不同模式的互补表征，因此使用多模态融合。
DL frameworks for multimodal fusion
?Despite the variety of available models, most multimodal fusion strategies fall into the following two categories: prefusion and postfusion.
- prefusion:concatenates raw features from multiple modalities before sending them to DLs; 这种很简单实现，但是当一个模态的特征维度远比其他的多的时候，或者由于数据结构的heterogeneity异构性，就会不可行。
- postfusion: use DLs for learning feature representation of each modality and then concatenated thaem for subsequent tasks.更灵活，但是寻找最佳结构的时候也更费力。
除了基于concatenation-based postfusion，还有更先进的方法，考虑交叉模态的关系。Multimodal reconstruction，deep canonical correlation analysis （DCCA） and knowledge-transfer-based fusion are three popular multimodal fusion methods.
更先进的3中多模态融合的方法：
- Multimodal reconstraction
- deep canonical correlation analysis DCCA
- knowledge-transfer-based fusion
image.png
- Multimodal reconstraction：是AE在多模态数据当中的一种方法。
- deep canonical correlation analysis DCCA：捕捉跨模态的相关性或者互信息。
这个的参考文献可以学一下：
image.png
- knowledge-transfer-based fusion
Multimodal fusion applications in neuroimaging
Promises and challenges
Visualization and subytype discovery
Network visualization for biomarker discovery
流行的可视化方法可以分成四类：
- interpretable local surrogates可解释性的局部替代物。两个经典方法是：
- Local interpretable model-agnostic explanation LIME
- Shapley additive explanations SHAP 相关文献：
- occlusion analysis 遮挡分析 相关文献：
5. gradient-based methods 基于梯度的方法 6. layer-wise relevance propagation 分层相关性传播
相关文献：
image.png
Spectrum and subtype discovery using DL framework
通过tsne和聚类来可视化，看看能不能发现新的亚型